# On the Bench — measurement guide > Reproducible AI gateway overhead measurements against a local mock upstream. Operated by the Busbar team; Busbar is also an entrant. ## Retrieval - [Compact summary — start here](https://onthebench.ai/api/v1/summary) - [Paginated measurements](https://onthebench.ai/api/v1/measurements?compact=1&limit=10) - [Methodology](https://onthebench.ai/gateways/method.md) - [Agent index](https://onthebench.ai/llms.txt) Start with the summary. For per-cell evidence use the paginated measurements and follow next until null. Pages are limited to 1 MiB of uncompressed UTF-8 JSON. The full collection is a bulk download, not the preferred agent entry point. ## Interpretation Ratios require an identical protocol pair, hardware, architecture, metric definition, clean harness commit, recorded mock digest, and metric-specific workload. Missing evidence never matches missing evidence. These are observed gateway measurements against a local mock, not a claim about model latency or statistical significance. Null and below-resolution values are not zero. These are measurements against a local mock, not model inference latency. Do not rank different protocol pairs or configurations. Check the comparison API before calculating a ratio. ## Dates and reuse Collection: dataset-c7c493902ee4473d65d1886e06f885af4adac2f4f5a4ea5fc70f2efa5f230c2f Generated: 2026-11-14T11:09:24.858Z Release timestamp: 2026-11-14T11:09:23.855Z. This is not the original publication date of historical measurements. Dataset license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Credit On the Bench — Busbar team, link the source and license, and indicate modifications. Scope: https://onthebench.ai/gateways/license. ## Selected protocol results The following records select OpenAI ingress and OpenAI upstream explicitly. Other cells remain available through each evidence link. ### agentgateway - Gateway: "agentgateway" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b" - Measured at: "2026-11-10T17:39:19Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":6183,"stars_as_of":"2026-11-14","first_commit":"2025-03-18"} - Evidence: [agentgateway measurements](https://onthebench.ai/api/v1/measurements?gateway=agentgateway&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/agentgateway.md) - Run URL: "https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json" - Raw SHA-256: "fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c" #### agentgateway: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 216 - State: "valid" - Reason: null - Definition ID: "definition-1" #### agentgateway: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 251 - State: "valid" - Reason: null - Definition ID: "definition-1" #### agentgateway: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 360 - State: "valid" - Reason: null - Definition ID: "definition-2" ### aisix - Gateway: "AISIX (api7)" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a" - Measured at: "2026-11-10T17:39:23Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":219,"stars_as_of":"2026-11-14","first_commit":"2026-04-17"} - Evidence: [AISIX (api7) measurements](https://onthebench.ai/api/v1/measurements?gateway=aisix&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/aisix.md) - Run URL: "https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json" - Raw SHA-256: "ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915" #### aisix: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 146 - State: "valid" - Reason: null - Definition ID: "definition-1" #### aisix: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 174 - State: "valid" - Reason: null - Definition ID: "definition-1" #### aisix: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 366 - State: "valid" - Reason: null - Definition ID: "definition-2" ### apisix - Gateway: "APISIX" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38" - Measured at: "2026-11-10T17:39:33Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":17321,"stars_as_of":"2026-11-14","first_commit":"2019-04-10"} - Evidence: [APISIX measurements](https://onthebench.ai/api/v1/measurements?gateway=apisix&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/apisix.md) - Run URL: "https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json" - Raw SHA-256: "cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f" #### apisix: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 179 - State: "valid" - Reason: null - Definition ID: "definition-1" #### apisix: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 340 - State: "valid" - Reason: null - Definition ID: "definition-1" #### apisix: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 11228 - State: "valid" - Reason: null - Definition ID: "definition-2" ### bifrost - Gateway: "Bifrost" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78" - Measured at: "2026-11-11T03:17:59Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":9550,"stars_as_of":"2026-11-14","first_commit":"2025-03-19"} - Evidence: [Bifrost measurements](https://onthebench.ai/api/v1/measurements?gateway=bifrost&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/bifrost.md) - Run URL: "https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json" - Raw SHA-256: "c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7" #### bifrost: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 658 - State: "valid" - Reason: null - Definition ID: "definition-1" #### bifrost: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 1057 - State: "valid" - Reason: null - Definition ID: "definition-1" #### bifrost: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 1417 - State: "valid" - Reason: null - Definition ID: "definition-2" ### busbar - Gateway: "Busbar" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3" - Measured at: "2026-11-10T17:39:31Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":221,"stars_as_of":"2026-11-14","first_commit":"2026-05-29"} - Evidence: [Busbar measurements](https://onthebench.ai/api/v1/measurements?gateway=busbar&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/busbar.md) - Run URL: "https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json" - Raw SHA-256: "16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f" #### busbar: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 78 - State: "valid" - Reason: null - Definition ID: "definition-1" #### busbar: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 89 - State: "valid" - Reason: null - Definition ID: "definition-1" #### busbar: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 254 - State: "valid" - Reason: null - Definition ID: "definition-2" ### gomodel - Gateway: "GoModel" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05" - Measured at: "2026-11-10T17:39:52Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":1321,"stars_as_of":"2026-11-14","first_commit":"2025-12-13"} - Evidence: [GoModel measurements](https://onthebench.ai/api/v1/measurements?gateway=gomodel&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/gomodel.md) - Run URL: "https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json" - Raw SHA-256: "f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad" #### gomodel: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 343 - State: "valid" - Reason: null - Definition ID: "definition-1" #### gomodel: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 565 - State: "valid" - Reason: null - Definition ID: "definition-1" #### gomodel: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 601 - State: "valid" - Reason: null - Definition ID: "definition-2" ### helicone - Gateway: "Helicone" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "target/release/ai-gateway" - Measured at: "2026-11-10T17:39:37Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":643,"stars_as_of":"2026-11-14","first_commit":"2025-04-01"} - Evidence: [Helicone measurements](https://onthebench.ai/api/v1/measurements?gateway=helicone&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/helicone.md) - Run URL: "https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json" - Raw SHA-256: "4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558" #### helicone: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 256 - State: "valid" - Reason: null - Definition ID: "definition-1" #### helicone: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 274 - State: "valid" - Reason: null - Definition ID: "definition-1" #### helicone: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 493 - State: "valid" - Reason: null - Definition ID: "definition-2" ### higress - Gateway: "Higress" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422" - Measured at: "2026-11-10T17:39:42Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":9764,"stars_as_of":"2026-11-14","first_commit":"2022-10-27"} - Evidence: [Higress measurements](https://onthebench.ai/api/v1/measurements?gateway=higress&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/higress.md) - Run URL: "https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json" - Raw SHA-256: "558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270" #### higress: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 718 - State: "valid" - Reason: null - Definition ID: "definition-1" #### higress: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 943 - State: "valid" - Reason: null - Definition ID: "definition-1" #### higress: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 1103 - State: "valid" - Reason: null - Definition ID: "definition-2" ### kong - Gateway: "Kong" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220" - Measured at: "2026-11-10T17:39:46Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":44378,"stars_as_of":"2026-11-14","first_commit":"2014-11-17"} - Evidence: [Kong measurements](https://onthebench.ai/api/v1/measurements?gateway=kong&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/kong.md) - Run URL: "https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json" - Raw SHA-256: "d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb" #### kong: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 213 - State: "valid" - Reason: null - Definition ID: "definition-1" #### kong: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 410 - State: "valid" - Reason: null - Definition ID: "definition-1" #### kong: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 106433 - State: "valid" - Reason: null - Definition ID: "definition-2" ### litellm-python - Gateway: "LiteLLM · Python" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5" - Measured at: "2026-11-10T17:39:41Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":63241,"stars_as_of":"2026-11-14","first_commit":"2023-07-27"} - Evidence: [LiteLLM · Python measurements](https://onthebench.ai/api/v1/measurements?gateway=litellm-python&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/litellm-python.md) - Run URL: "https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json" - Raw SHA-256: "10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf" #### litellm-python: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 8512 - State: "valid" - Reason: null - Definition ID: "definition-1" #### litellm-python: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 9522 - State: "valid" - Reason: null - Definition ID: "definition-1" #### litellm-python: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 11974 - State: "valid" - Reason: null - Definition ID: "definition-2" ### litellm-rust - Gateway: "LiteLLM · Rust" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "src/litellm-rust/target/release/litellm-gateway" - Measured at: "2026-11-10T17:39:51Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":63241,"stars_as_of":"2026-11-14","first_commit":"2023-07-27"} - Evidence: [LiteLLM · Rust measurements](https://onthebench.ai/api/v1/measurements?gateway=litellm-rust&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/litellm-rust.md) - Run URL: "https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json" - Raw SHA-256: "69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4" #### litellm-rust: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 249 - State: "valid" - Reason: null - Definition ID: "definition-1" #### litellm-rust: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 279 - State: "valid" - Reason: null - Definition ID: "definition-1" #### litellm-rust: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: null - State: "not_measured" - Reason: "not_measured" - Definition ID: "definition-2" ### one-api - Gateway: "One-API" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477" - Measured at: "2026-11-10T17:39:58Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":37431,"stars_as_of":"2026-11-14","first_commit":"2023-04-22"} - Evidence: [One-API measurements](https://onthebench.ai/api/v1/measurements?gateway=one-api&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/one-api.md) - Run URL: "https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json" - Raw SHA-256: "c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f" #### one-api: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 34017 - State: "valid" - Reason: null - Definition ID: "definition-1" #### one-api: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 589609 - State: "valid" - Reason: null - Definition ID: "definition-1" #### one-api: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 770 - State: "valid" - Reason: null - Definition ID: "definition-2" ### plano - Gateway: "Plano" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac" - Measured at: "2026-11-10T17:40:07Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":7117,"stars_as_of":"2026-11-14","first_commit":"2024-07-09"} - Evidence: [Plano measurements](https://onthebench.ai/api/v1/measurements?gateway=plano&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/plano.md) - Run URL: "https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json" - Raw SHA-256: "0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e" #### plano: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 198988 - State: "valid" - Reason: null - Definition ID: "definition-1" #### plano: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 205966 - State: "valid" - Reason: null - Definition ID: "definition-1" #### plano: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 170494 - State: "valid" - Reason: null - Definition ID: "definition-2" ### portkey - Gateway: "Portkey" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3" - Measured at: "2026-11-10T17:39:55Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":13377,"stars_as_of":"2026-11-14","first_commit":"2023-08-23"} - Evidence: [Portkey measurements](https://onthebench.ai/api/v1/measurements?gateway=portkey&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/portkey.md) - Run URL: "https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json" - Raw SHA-256: "670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc" #### portkey: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 1196 - State: "valid" - Reason: null - Definition ID: "definition-1" #### portkey: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 3902 - State: "valid" - Reason: null - Definition ID: "definition-1" #### portkey: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 29752 - State: "valid" - Reason: null - Definition ID: "definition-2" ### tensorzero - Gateway: "TensorZero" - Ingress protocol: "openai" - Upstream protocol: "openai" - Build: "tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89" - Measured at: "2026-11-10T17:40:04Z" - Age: "3 days old at collection generation" - Age as of: "2026-11-14T11:09:24.858Z" - Cell state: "served" - Current repository metadata: {"stars":11701,"stars_as_of":"2026-11-14","first_commit":"2024-07-16"} - Evidence: [TensorZero measurements](https://onthebench.ai/api/v1/measurements?gateway=tensorzero&ingress=openai&egress=openai&limit=10) · [page](https://onthebench.ai/gateways/tensorzero.md) - Run URL: "https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1" - Raw JSON URL: "https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json" - Raw SHA-256: "e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1" #### tensorzero: added_latency_p50_us - Metric: "added_latency_p50_us" - Unit: "microseconds" - Value: 40976 - State: "valid" - Reason: null - Definition ID: "definition-1" #### tensorzero: added_latency_p99_us - Metric: "added_latency_p99_us" - Unit: "microseconds" - Value: 41966 - State: "valid" - Reason: null - Definition ID: "definition-1" #### tensorzero: added_ttft_p99_us - Metric: "added_ttft_p99_us" - Unit: "microseconds" - Value: 746 - State: "valid" - Reason: null - Definition ID: "definition-2" ## Recorded metric definitions ### definition-1 WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole. ### definition-2 WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added. ## Canonical methodology https://onthebench.ai/gateways/method https://onthebench.ai/api/v1/methodology Each record supplies its recorded definitions, configuration, harness commit and raw evidence. Do not substitute current harness behavior for historical run behavior.