Benchmark data and API

Operated by the Busbar team; Busbar is also an entrant. Gateway overhead against a local mock is not end-to-end model latency. Methodology and limitations.

Published collection: dataset-c7c493902ee4473d65d1886e06f885af4adac2f4f5a4ea5fc70f2efa5f230c2f. Generated ; measurements through . Metadata refresh is separate from measurement time. This is a collection of independent runs, not a claim of one complete simultaneous field run.

API discovery · Latest JSON · Markdown · Compact CSV · OpenAPI · JSON schemas · Historical runs

Agent discovery (llms.txt) · Full agent guide · Version 1 measurement schema

Per-gateway exports

Compact results and freshness

Start with the compact summary (one protocol pair, core metrics and provenance). Filter with gateway, ingress and egress. Full collections may exceed retrieval-tool limits; follow paginated measurement links for details. Each compact result and comparison measurement includes measured_at, published_at, generated_at, age, age_days and age_as_of. Age is measured at collection generation, not at request time. published_at is recorded for new releases when publishing begins; it remains null for local builds and older releases without that record.

current_metadata contains repository stars and their stars_as_of date. Legacy gateway.stars fields refer to that same metadata snapshot, not the benchmark date. Stars at measurement time were not recorded and are not inferred.

Context-efficient retrieval

Use /api/v1/measurements?compact=1&limit=10 to receive definitions once, keyed by definition_id. Compact summary uses the same dictionary pattern. Prefer each gateway’s Markdown and /gateways/data.csv?gateway=ID&compact=1 links; the CSV references the definition catalog. Full CSV and full JSON are bulk exports. The paginated run catalog supports exact gateway IDs and families such as family=busbar.

Retrieval

All resources are public and read-only. HTML contains measurements without JavaScript. Explicit Markdown and JSON links avoid content negotiation. Query endpoints return 400 for invalid filters and 404 for unknown entities.

GET /api/v1/summary?ingress=openai&egress=openai
GET /api/v1/gateways
GET /api/v1/gateways/busbar
GET /api/v1/measurements?gateway=busbar&ingress=openai&egress=openai&limit=50
GET /api/v1/compare?a=busbar&b=bifrost&ingress=openai&egress=openai&metric=added_latency_p99_us

Measurements support gateway, ingress, egress, metric, limit (1–100), and offset (0 or greater). Responses are capped at 1 MiB of uncompressed UTF-8 JSON, so a page may contain fewer than limit rows. Follow the returned next URL until null; do not infer completion from row count. An individual oversized record returns 413 with narrowing guidance. Comparisons require two distinct published gateway IDs and a recorded metric. The default protocol pair is OpenAI → OpenAI and the default metric is added_latency_p99_us. Available gateway IDs and metrics are listed by the API.

Compare and cite using HTTP only

  1. Discover gateway IDs with the catalog and choose an explicit ingress/upstream pair. The API calls the upstream protocol egress.
  2. Retrieve OpenAI → Anthropic measurements; follow next until it is null. Unavailable values are not zero.
  3. Request Busbar vs LiteLLM on that pair and inspect comparability.status. Only comparable supports a ratio; incomparable and insufficient_evidence include reasons. Matching protocol names alone is not enough.
  4. Cite each record’s provenance.run_url, tested gateway_build, measured_at, protocol pair and metric definition. provenance.raw_result_url and raw_sha256 identify the exact source bytes. Include the methodology and operator disclosure.

The default dashboard is not a like-for-like ranking. Its representative protocol cell can differ between products. The API returns per-cell evidence and conservative pairwise comparisons; it does not publish an unconditional fastest-gateway ranking. These are independent gateway runs against a local mock, not end-to-end model latency.

Interpretation

Ratios require an identical protocol pair, hardware, architecture, metric definition, clean harness commit, recorded mock digest, and metric-specific workload. Missing evidence never matches missing evidence. These are observed gateway measurements against a local mock, not a claim about model latency or statistical significance.

Every metric carries its original definition, state, units in its field name, and provenance when recorded. Missing evidence remains null. A measured zero is retained; a below-resolution reading has a null value and an explicit state. Throughput readings carry their tail-latency bound, operating concurrency, and lower-bound status. Compare conditions before ranking.

Immutable evidence

Run IDs include the SHA-256 of their exact raw JSON bytes. Raw files contain the complete samples; dashboard drawing series are thinned. Corrected raw content receives a different ID. The latest collection points to individual runs and retains their actual dates. A collection ID identifies its generated evidence representation; it is distinct from a per-gateway run ID.

Dates and licensing

published_at is the release timestamp recorded when publishing begins, not the original publication date of historical measurements; generated_at is collection build time; measured_at belongs to the instrument; stars_as_of belongs to metadata. An original first-publication timestamp is not inferred. Original measurements and dataset compilation are licensed under CC BY 4.0. Credit On the Bench — Busbar team, link the source and license, and indicate modifications. Dataset license and scope; software and third-party materials retain their own terms.