LiteLLM · Rust vs TensorZero

Operated by the Busbar team; Busbar is also an entrant. Gateway overhead against a local mock is not end-to-end model latency. Methodology and limitations.

Requested cell: OpenAI → OpenAI. Metric: Added latency p99 (µs).

Comparability: Comparable. Ratios require an identical protocol pair, hardware, architecture, metric definition, clean harness commit, recorded mock digest, and metric-specific workload. Missing evidence never matches missing evidence. These are observed gateway measurements against a local mock, not a claim about model latency or statistical significance.

Ratio (TensorZero ÷ LiteLLM · Rust): 150.4. No statistical significance is asserted.

Comparison JSON · All LiteLLM · Rust evidence · All TensorZero evidence

LiteLLM · Rust

Build src/litellm-rust/target/release/litellm-gateway. Measured . Run evidence

Ingress → upstreamMetricValueState / reasonp99 bound (ms)
OpenAI → OpenAIAdded latency p50 (µs)249Valid · n=22,021 gateway / 267,021 direct—
OpenAI → OpenAIAdded latency p99 (µs)279Valid · n=22,021 gateway / 267,021 direct—
OpenAI → OpenAIGateway p99 at concurrency 1 (µs)306Valid · n=22,021—
OpenAI → OpenAIDirect-to-mock p99 at concurrency 1 (µs)27Valid · n=267,021—
OpenAI → OpenAICPU per request (µs)272.6Valid—
OpenAI → OpenAIRequests per CPU-second3,668.4Valid—
OpenAI → OpenAICost window concurrency8Valid—
OpenAI → OpenAICost window core utilisation0.679Valid—
OpenAI → OpenAICost window successful requests59,941Valid—
OpenAI → OpenAICost window throughput (req/s)9,989Valid—
OpenAI → OpenAICost window threads5Valid—
OpenAI → OpenAINon-voluntary context switches per request0Valid—
OpenAI → OpenAICost window major page faults0Valid—
OpenAI → OpenAIAdded time to first event p50 (µs)—Not measured—
OpenAI → OpenAIAdded time to first event p99 (µs)—Not measured—
OpenAI → OpenAIAdded inter-frame gap p50 (µs)—Not measured—
OpenAI → OpenAIAdded inter-frame gap p99 (µs)—Not measured—
OpenAI → OpenAIStreams sustained (frames/s)0Valid — measured failure: the gateway was offered stream load and sustained none of it—
OpenAI → OpenAIStreams sustained0Valid — measured failure: the gateway was offered stream load and sustained none of it—
OpenAI → OpenAIIdle RSS (MiB)12.74Valid—
OpenAI → OpenAISteady-state RSS (MiB)18.99Valid—
OpenAI → OpenAIRecovered RSS (MiB)19Valid—
OpenAI → OpenAIPeak RSS (MiB)19.88Valid—
OpenAI → OpenAIPeak RSS high-water mark (MiB)19.48Valid—
OpenAI → OpenAITime to plateau (s)—Not measured—
OpenAI → OpenAIMemory growth (MiB/min)-0.179Valid—
OpenAI → OpenAIMemory load duration (s)300Valid—
OpenAI → OpenAIMemory shape under load0Valid—
OpenAI → OpenAIIdle memory shape—Not measured—
OpenAI → OpenAIIdle memory static1Valid—
OpenAI → OpenAIIdle memory growth (MiB/min)0Valid—
OpenAI → OpenAIThroughput (req/s)8,404concurrency 4; n=50,4351 ms
OpenAI → OpenAIThroughput (req/s)10,049concurrency 16; n=60,3065 ms
OpenAI → OpenAIThroughput (req/s)10,049concurrency 16; n=60,30610 ms
OpenAI → OpenAIThroughput (req/s)10,049concurrency 16; n=60,30650 ms
OpenAI → OpenAIThroughput (req/s)10,049concurrency 16; n=60,306100 ms
OpenAI → OpenAIThroughput (req/s)10,049concurrency 16; n=60,306Unbounded

TensorZero

Build tensorzero/gateway:2026.6.0. Measured . Run evidence

Ingress → upstreamMetricValueState / reasonp99 bound (ms)
OpenAI → OpenAIAdded latency p50 (µs)40,976Valid · n=150 gateway / 253,662 direct—
OpenAI → OpenAIAdded latency p99 (µs)41,966Valid · n=150 gateway / 253,662 direct—
OpenAI → OpenAIGateway p99 at concurrency 1 (µs)41,996Valid · n=150—
OpenAI → OpenAIDirect-to-mock p99 at concurrency 1 (µs)30Valid · n=253,662—
OpenAI → OpenAICPU per request (µs)459.1Valid—
OpenAI → OpenAIRequests per CPU-second2,178.2Valid—
OpenAI → OpenAICost window concurrency8Valid—
OpenAI → OpenAICost window core utilisation0.023Valid—
OpenAI → OpenAICost window successful requests1,198Valid—
OpenAI → OpenAICost window throughput (req/s)199Valid—
OpenAI → OpenAICost window threads6Valid—
OpenAI → OpenAINon-voluntary context switches per request0Valid—
OpenAI → OpenAICost window major page faults0Valid—
OpenAI → OpenAIAdded time to first event p50 (µs)707Valid · n=100 gateway / 100 direct—
OpenAI → OpenAIAdded time to first event p99 (µs)746Valid · n=100 gateway / 100 direct—
OpenAI → OpenAIAdded inter-frame gap p50 (µs)—Below resolution · n=63 gateway / 63 direct—
OpenAI → OpenAIAdded inter-frame gap p99 (µs)—Below resolution · n=63 gateway / 63 direct—
OpenAI → OpenAIStreams sustained (frames/s)8,048.6Valid—
OpenAI → OpenAIStreams sustained1,991Valid—
OpenAI → OpenAIIdle RSS (MiB)48.75Valid—
OpenAI → OpenAISteady-state RSS (MiB)67.62Valid—
OpenAI → OpenAIRecovered RSS (MiB)67.62Valid—
OpenAI → OpenAIPeak RSS (MiB)67.62Valid—
OpenAI → OpenAIPeak RSS high-water mark (MiB)67.62Valid—
OpenAI → OpenAITime to plateau (s)60Valid—
OpenAI → OpenAIMemory growth (MiB/min)0Valid—
OpenAI → OpenAIMemory load duration (s)302Valid—
OpenAI → OpenAIMemory shape under load—Not measured—
OpenAI → OpenAIIdle memory shape—Not measured—
OpenAI → OpenAIIdle memory static1Valid—
OpenAI → OpenAIIdle memory growth (MiB/min)0.025Valid—
OpenAI → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
OpenAI → OpenAIThroughput (req/s)6,275concurrency 256; n=37,65850 ms
OpenAI → OpenAIThroughput (req/s)11,879concurrency 512; n=71,288100 ms
OpenAI → OpenAIThroughput (req/s)12,324concurrency 1,024; n=73,956Unbounded