LiteLLM · Python benchmark measurements

Operated by the Busbar team; Busbar is also an entrant. Gateway overhead against a local mock is not end-to-end model latency. Methodology and limitations.

Python · recorded build: ghcr.io/berriai/litellm:v1.108.3. Measured: .

JSON · Markdown · CSV · Project source · Interactive dashboard

Each row below names its protocol pair. A representative dashboard cell may differ between gateways; compare an identical pair before interpreting a ranking. Unavailable metrics retain their reason; below-resolution readings are not exact zeroes.

Published measurements

Ingress → upstreamMetricValueState / reasonp99 bound (ms)
Cohere → CohereAdded latency p50 (µs)11,850Valid · n=503 gateway / 264,219 direct—
Cohere → CohereAdded latency p99 (µs)12,623Valid · n=503 gateway / 264,219 direct—
Cohere → CohereGateway p99 at concurrency 1 (µs)12,651Valid · n=503—
Cohere → CohereDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=264,219—
Cohere → CohereCPU per request (µs)12,063.5Valid—
Cohere → CohereRequests per CPU-second82.89Valid—
Cohere → CohereCost window concurrency8Valid—
Cohere → CohereCost window core utilisation0.25Valid—
Cohere → CohereCost window successful requests504Valid—
Cohere → CohereCost window throughput (req/s)83Valid—
Cohere → CohereCost window threads6Valid—
Cohere → CohereNon-voluntary context switches per request0.052Valid—
Cohere → CohereCost window major page faults0Valid—
Cohere → CohereAdded time to first event p50 (µs)—Untestable—
Cohere → CohereAdded time to first event p99 (µs)—Untestable—
Cohere → CohereAdded inter-frame gap p50 (µs)—Untestable—
Cohere → CohereAdded inter-frame gap p99 (µs)—Untestable—
Cohere → CohereStreams sustained (frames/s)—Untestable—
Cohere → CohereStreams sustained—Untestable—
Cohere → CohereIdle RSS (MiB)592.8Valid—
Cohere → CohereSteady-state RSS (MiB)663.3Valid—
Cohere → CohereRecovered RSS (MiB)663.3Valid—
Cohere → CoherePeak RSS (MiB)663.3Valid—
Cohere → CoherePeak RSS high-water mark (MiB)663.3Valid—
Cohere → CohereTime to plateau (s)81Valid—
Cohere → CohereMemory growth (MiB/min)0Valid—
Cohere → CohereMemory load duration (s)302Valid—
Cohere → CohereMemory shape under load—Not measured—
Cohere → CohereIdle memory shape—Not measured—
Cohere → CohereIdle memory static1Valid—
Cohere → CohereIdle memory growth (MiB/min)0Valid—
Cohere → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Cohere → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Cohere → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Cohere → CohereThroughput (req/s)83concurrency 1; n=49950 ms
Cohere → CohereThroughput (req/s)83concurrency 1; n=499100 ms
Cohere → CohereThroughput (req/s)112concurrency 1,024; n=674Unbounded
OpenAI → CohereAdded latency p50 (µs)7,551Valid · n=790 gateway / 267,951 direct—
OpenAI → CohereAdded latency p99 (µs)8,314Valid · n=790 gateway / 267,951 direct—
OpenAI → CohereGateway p99 at concurrency 1 (µs)8,341Valid · n=790—
OpenAI → CohereDirect-to-mock p99 at concurrency 1 (µs)27Valid · n=267,951—
OpenAI → CohereCPU per request (µs)7,594Valid—
OpenAI → CohereRequests per CPU-second131.7Valid—
OpenAI → CohereCost window concurrency8Valid—
OpenAI → CohereCost window core utilisation0.25Valid—
OpenAI → CohereCost window successful requests798Valid—
OpenAI → CohereCost window throughput (req/s)132Valid—
OpenAI → CohereCost window threads13Valid—
OpenAI → CohereNon-voluntary context switches per request0.031Valid—
OpenAI → CohereCost window major page faults0Valid—
OpenAI → CohereAdded time to first event p50 (µs)—Untestable—
OpenAI → CohereAdded time to first event p99 (µs)—Untestable—
OpenAI → CohereAdded inter-frame gap p50 (µs)—Untestable—
OpenAI → CohereAdded inter-frame gap p99 (µs)—Untestable—
OpenAI → CohereStreams sustained (frames/s)—Untestable—
OpenAI → CohereStreams sustained—Untestable—
OpenAI → CohereIdle RSS (MiB)592.6Valid—
OpenAI → CohereSteady-state RSS (MiB)666.1Valid—
OpenAI → CohereRecovered RSS (MiB)666.1Valid—
OpenAI → CoherePeak RSS (MiB)666.1Valid—
OpenAI → CoherePeak RSS high-water mark (MiB)666.1Valid—
OpenAI → CohereTime to plateau (s)61Valid—
OpenAI → CohereMemory growth (MiB/min)0.002Valid—
OpenAI → CohereMemory load duration (s)301Valid—
OpenAI → CohereMemory shape under load—Not measured—
OpenAI → CohereIdle memory shape—Not measured—
OpenAI → CohereIdle memory static1Valid—
OpenAI → CohereIdle memory growth (MiB/min)0Valid—
OpenAI → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI → CohereThroughput (req/s)131concurrency 1; n=78710 ms
OpenAI → CohereThroughput (req/s)131concurrency 1; n=78750 ms
OpenAI → CohereThroughput (req/s)131concurrency 1; n=787100 ms
OpenAI → CohereThroughput (req/s)138concurrency 64; n=829Unbounded
Bedrock → Cohere——Not declared: recorded note 1—
Anthropic → CohereAdded latency p50 (µs)8,248Valid · n=723 gateway / 260,556 direct—
Anthropic → CohereAdded latency p99 (µs)9,054Valid · n=723 gateway / 260,556 direct—
Anthropic → CohereGateway p99 at concurrency 1 (µs)9,082Valid · n=723—
Anthropic → CohereDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=260,556—
Anthropic → CohereCPU per request (µs)8,407.2Valid—
Anthropic → CohereRequests per CPU-second118.9Valid—
Anthropic → CohereCost window concurrency8Valid—
Anthropic → CohereCost window core utilisation0.25Valid—
Anthropic → CohereCost window successful requests722Valid—
Anthropic → CohereCost window throughput (req/s)120Valid—
Anthropic → CohereCost window threads13Valid—
Anthropic → CohereNon-voluntary context switches per request0.025Valid—
Anthropic → CohereCost window major page faults0Valid—
Anthropic → CohereAdded time to first event p50 (µs)—Untestable—
Anthropic → CohereAdded time to first event p99 (µs)—Untestable—
Anthropic → CohereAdded inter-frame gap p50 (µs)—Untestable—
Anthropic → CohereAdded inter-frame gap p99 (µs)—Untestable—
Anthropic → CohereStreams sustained (frames/s)—Untestable—
Anthropic → CohereStreams sustained—Untestable—
Anthropic → CohereIdle RSS (MiB)592.2Valid—
Anthropic → CohereSteady-state RSS (MiB)665.8Valid—
Anthropic → CohereRecovered RSS (MiB)665.8Valid—
Anthropic → CoherePeak RSS (MiB)665.8Valid—
Anthropic → CoherePeak RSS high-water mark (MiB)665.8Valid—
Anthropic → CohereTime to plateau (s)61Valid—
Anthropic → CohereMemory growth (MiB/min)0Valid—
Anthropic → CohereMemory load duration (s)302Valid—
Anthropic → CohereMemory shape under load—Not measured—
Anthropic → CohereIdle memory shape—Not measured—
Anthropic → CohereIdle memory static1Valid—
Anthropic → CohereIdle memory growth (MiB/min)0Valid—
Anthropic → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Anthropic → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Anthropic → CohereThroughput (req/s)118concurrency 1; n=71310 ms
Anthropic → CohereThroughput (req/s)120concurrency 2; n=72250 ms
Anthropic → CohereThroughput (req/s)120concurrency 2; n=722100 ms
Anthropic → CohereThroughput (req/s)120concurrency 2; n=722Unbounded
OpenAI Responses → CohereAdded latency p50 (µs)9,710Valid · n=615 gateway / 263,786 direct—
OpenAI Responses → CohereAdded latency p99 (µs)10,423Valid · n=615 gateway / 263,786 direct—
OpenAI Responses → CohereGateway p99 at concurrency 1 (µs)10,451Valid · n=615—
OpenAI Responses → CohereDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=263,786—
OpenAI Responses → CohereCPU per request (µs)9,886.4Valid—
OpenAI Responses → CohereRequests per CPU-second101.1Valid—
OpenAI Responses → CohereCost window concurrency8Valid—
OpenAI Responses → CohereCost window core utilisation0.251Valid—
OpenAI Responses → CohereCost window successful requests616Valid—
OpenAI Responses → CohereCost window throughput (req/s)102Valid—
OpenAI Responses → CohereCost window threads13Valid—
OpenAI Responses → CohereNon-voluntary context switches per request0.047Valid—
OpenAI Responses → CohereCost window major page faults0Valid—
OpenAI Responses → CohereAdded time to first event p50 (µs)—Untestable—
OpenAI Responses → CohereAdded time to first event p99 (µs)—Untestable—
OpenAI Responses → CohereAdded inter-frame gap p50 (µs)—Untestable—
OpenAI Responses → CohereAdded inter-frame gap p99 (µs)—Untestable—
OpenAI Responses → CohereStreams sustained (frames/s)—Untestable—
OpenAI Responses → CohereStreams sustained—Untestable—
OpenAI Responses → CohereIdle RSS (MiB)592.7Valid—
OpenAI Responses → CohereSteady-state RSS (MiB)667.8Valid—
OpenAI Responses → CohereRecovered RSS (MiB)667.8Valid—
OpenAI Responses → CoherePeak RSS (MiB)667.8Valid—
OpenAI Responses → CoherePeak RSS high-water mark (MiB)667.8Valid—
OpenAI Responses → CohereTime to plateau (s)62Valid—
OpenAI Responses → CohereMemory growth (MiB/min)0Valid—
OpenAI Responses → CohereMemory load duration (s)304Valid—
OpenAI Responses → CohereMemory shape under load—Not measured—
OpenAI Responses → CohereIdle memory shape—Not measured—
OpenAI Responses → CohereIdle memory static1Valid—
OpenAI Responses → CohereIdle memory growth (MiB/min)0Valid—
OpenAI Responses → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI Responses → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI Responses → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
OpenAI Responses → CohereThroughput (req/s)104concurrency 2; n=63050 ms
OpenAI Responses → CohereThroughput (req/s)104concurrency 2; n=630100 ms
OpenAI Responses → CohereThroughput (req/s)106concurrency 64; n=637Unbounded
Gemini → CohereAdded latency p50 (µs)13,881Valid · n=428 gateway / 262,194 direct—
Gemini → CohereAdded latency p99 (µs)14,770Valid · n=428 gateway / 262,194 direct—
Gemini → CohereGateway p99 at concurrency 1 (µs)14,798Valid · n=428—
Gemini → CohereDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=262,194—
Gemini → CohereCPU per request (µs)14,342.7Valid—
Gemini → CohereRequests per CPU-second69.72Valid—
Gemini → CohereCost window concurrency8Valid—
Gemini → CohereCost window core utilisation0.251Valid—
Gemini → CohereCost window successful requests426Valid—
Gemini → CohereCost window throughput (req/s)70Valid—
Gemini → CohereCost window threads13Valid—
Gemini → CohereNon-voluntary context switches per request0.063Valid—
Gemini → CohereCost window major page faults0Valid—
Gemini → CohereAdded time to first event p50 (µs)—Untestable—
Gemini → CohereAdded time to first event p99 (µs)—Untestable—
Gemini → CohereAdded inter-frame gap p50 (µs)—Untestable—
Gemini → CohereAdded inter-frame gap p99 (µs)—Untestable—
Gemini → CohereStreams sustained (frames/s)—Untestable—
Gemini → CohereStreams sustained—Untestable—
Gemini → CohereIdle RSS (MiB)593.4Valid—
Gemini → CohereSteady-state RSS (MiB)670.8Valid—
Gemini → CohereRecovered RSS (MiB)671Valid—
Gemini → CoherePeak RSS (MiB)671Valid—
Gemini → CoherePeak RSS high-water mark (MiB)671Valid—
Gemini → CohereTime to plateau (s)63Valid—
Gemini → CohereMemory growth (MiB/min)0.096Valid—
Gemini → CohereMemory load duration (s)302Valid—
Gemini → CohereMemory shape under load—Not measured—
Gemini → CohereIdle memory shape—Not measured—
Gemini → CohereIdle memory static1Valid—
Gemini → CohereIdle memory growth (MiB/min)0Valid—
Gemini → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Gemini → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Gemini → CohereThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Gemini → CohereThroughput (req/s)71concurrency 1; n=42750 ms
Gemini → CohereThroughput (req/s)71concurrency 1; n=427100 ms
Gemini → CohereThroughput (req/s)73concurrency 256; n=439Unbounded
Cohere → Anthropic——Not declared: recorded note 1—
Bedrock → Anthropic——Not declared: recorded note 1—
OpenAI Responses → AnthropicAdded latency p50 (µs)13,061Valid · n=457 gateway / 263,938 direct—
OpenAI Responses → AnthropicAdded latency p99 (µs)13,826Valid · n=457 gateway / 263,938 direct—
OpenAI Responses → AnthropicGateway p99 at concurrency 1 (µs)13,854Valid · n=457—
OpenAI Responses → AnthropicDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=263,938—
OpenAI Responses → AnthropicCPU per request (µs)13,414.1Valid—
OpenAI Responses → AnthropicRequests per CPU-second74.55Valid—
OpenAI Responses → AnthropicCost window concurrency8Valid—
OpenAI Responses → AnthropicCost window core utilisation0.25Valid—
OpenAI Responses → AnthropicCost window successful requests454Valid—
OpenAI Responses → AnthropicCost window throughput (req/s)75Valid—
OpenAI Responses → AnthropicCost window threads13Valid—
OpenAI Responses → AnthropicNon-voluntary context switches per request0.055Valid—
OpenAI Responses → AnthropicCost window major page faults0Valid—
OpenAI Responses → AnthropicAdded time to first event p50 (µs)—Untestable—
OpenAI Responses → AnthropicAdded time to first event p99 (µs)—Untestable—
OpenAI Responses → AnthropicAdded inter-frame gap p50 (µs)—Untestable—
OpenAI Responses → AnthropicAdded inter-frame gap p99 (µs)—Untestable—
OpenAI Responses → AnthropicStreams sustained (frames/s)—Untestable—
OpenAI Responses → AnthropicStreams sustained—Untestable—
OpenAI Responses → AnthropicIdle RSS (MiB)592.7Valid—
OpenAI Responses → AnthropicSteady-state RSS (MiB)667.6Valid—
OpenAI Responses → AnthropicRecovered RSS (MiB)667.6Valid—
OpenAI Responses → AnthropicPeak RSS (MiB)667.6Valid—
OpenAI Responses → AnthropicPeak RSS high-water mark (MiB)667.6Valid—
OpenAI Responses → AnthropicTime to plateau (s)62Valid—
OpenAI Responses → AnthropicMemory growth (MiB/min)0Valid—
OpenAI Responses → AnthropicMemory load duration (s)302Valid—
OpenAI Responses → AnthropicMemory shape under load—Not measured—
OpenAI Responses → AnthropicIdle memory shape—Not measured—
OpenAI Responses → AnthropicIdle memory static1Valid—
OpenAI Responses → AnthropicIdle memory growth (MiB/min)0Valid—
OpenAI Responses → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI Responses → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI Responses → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
OpenAI Responses → AnthropicThroughput (req/s)76concurrency 1; n=45850 ms
OpenAI Responses → AnthropicThroughput (req/s)76concurrency 1; n=458100 ms
OpenAI Responses → AnthropicThroughput (req/s)76concurrency 1; n=458Unbounded
Gemini → AnthropicAdded latency p50 (µs)16,324Valid · n=364 gateway / 262,033 direct—
Gemini → AnthropicAdded latency p99 (µs)17,226Valid · n=364 gateway / 262,033 direct—
Gemini → AnthropicGateway p99 at concurrency 1 (µs)17,254Valid · n=364—
Gemini → AnthropicDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=262,033—
Gemini → AnthropicCPU per request (µs)16,832Valid—
Gemini → AnthropicRequests per CPU-second59.41Valid—
Gemini → AnthropicCost window concurrency8Valid—
Gemini → AnthropicCost window core utilisation0.25Valid—
Gemini → AnthropicCost window successful requests363Valid—
Gemini → AnthropicCost window throughput (req/s)60Valid—
Gemini → AnthropicCost window threads13Valid—
Gemini → AnthropicNon-voluntary context switches per request0.074Valid—
Gemini → AnthropicCost window major page faults0Valid—
Gemini → AnthropicAdded time to first event p50 (µs)—Untestable—
Gemini → AnthropicAdded time to first event p99 (µs)—Untestable—
Gemini → AnthropicAdded inter-frame gap p50 (µs)—Untestable—
Gemini → AnthropicAdded inter-frame gap p99 (µs)—Untestable—
Gemini → AnthropicStreams sustained (frames/s)—Untestable—
Gemini → AnthropicStreams sustained—Untestable—
Gemini → AnthropicIdle RSS (MiB)591.9Valid—
Gemini → AnthropicSteady-state RSS (MiB)671.9Valid—
Gemini → AnthropicRecovered RSS (MiB)671.9Valid—
Gemini → AnthropicPeak RSS (MiB)671.9Valid—
Gemini → AnthropicPeak RSS high-water mark (MiB)671.9Valid—
Gemini → AnthropicTime to plateau (s)63Valid—
Gemini → AnthropicMemory growth (MiB/min)0Valid—
Gemini → AnthropicMemory load duration (s)305Valid—
Gemini → AnthropicMemory shape under load—Not measured—
Gemini → AnthropicIdle memory shape—Not measured—
Gemini → AnthropicIdle memory static1Valid—
Gemini → AnthropicIdle memory growth (MiB/min)0Valid—
Gemini → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Gemini → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Gemini → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Gemini → AnthropicThroughput (req/s)60concurrency 2; n=36250 ms
Gemini → AnthropicThroughput (req/s)60concurrency 2; n=362100 ms
Gemini → AnthropicThroughput (req/s)60concurrency 2; n=362Unbounded
OpenAI → AnthropicAdded latency p50 (µs)10,313Valid · n=579 gateway / 265,135 direct—
OpenAI → AnthropicAdded latency p99 (µs)11,125Valid · n=579 gateway / 265,135 direct—
OpenAI → AnthropicGateway p99 at concurrency 1 (µs)11,153Valid · n=579—
OpenAI → AnthropicDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=265,135—
OpenAI → AnthropicCPU per request (µs)10,411.7Valid—
OpenAI → AnthropicRequests per CPU-second96.05Valid—
OpenAI → AnthropicCost window concurrency8Valid—
OpenAI → AnthropicCost window core utilisation0.25Valid—
OpenAI → AnthropicCost window successful requests583Valid—
OpenAI → AnthropicCost window throughput (req/s)97Valid—
OpenAI → AnthropicCost window threads28Valid—
OpenAI → AnthropicNon-voluntary context switches per request0.038Valid—
OpenAI → AnthropicCost window major page faults0Valid—
OpenAI → AnthropicAdded time to first event p50 (µs)13,428Valid · n=100 gateway / 100 direct—
OpenAI → AnthropicAdded time to first event p99 (µs)14,151Valid · n=100 gateway / 100 direct—
OpenAI → AnthropicAdded inter-frame gap p50 (µs)3Valid · n=63 gateway / 63 direct—
OpenAI → AnthropicAdded inter-frame gap p99 (µs)1,348Valid · n=63 gateway / 63 direct—
OpenAI → AnthropicStreams sustained (frames/s)1,232.9Valid—
OpenAI → AnthropicStreams sustained117Valid—
OpenAI → AnthropicIdle RSS (MiB)592.5Valid—
OpenAI → AnthropicSteady-state RSS (MiB)667.1Valid—
OpenAI → AnthropicRecovered RSS (MiB)667.1Valid—
OpenAI → AnthropicPeak RSS (MiB)667.1Valid—
OpenAI → AnthropicPeak RSS high-water mark (MiB)667.1Valid—
OpenAI → AnthropicTime to plateau (s)62Valid—
OpenAI → AnthropicMemory growth (MiB/min)0Valid—
OpenAI → AnthropicMemory load duration (s)304Valid—
OpenAI → AnthropicMemory shape under load—Not measured—
OpenAI → AnthropicIdle memory shape—Not measured—
OpenAI → AnthropicIdle memory static1Valid—
OpenAI → AnthropicIdle memory growth (MiB/min)0Valid—
OpenAI → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
OpenAI → AnthropicThroughput (req/s)97concurrency 1; n=58650 ms
OpenAI → AnthropicThroughput (req/s)98concurrency 8; n=589100 ms
OpenAI → AnthropicThroughput (req/s)98concurrency 8; n=589Unbounded
Anthropic → AnthropicAdded latency p50 (µs)10,994Valid · n=544 gateway / 263,642 direct—
Anthropic → AnthropicAdded latency p99 (µs)11,745Valid · n=544 gateway / 263,642 direct—
Anthropic → AnthropicGateway p99 at concurrency 1 (µs)11,773Valid · n=544—
Anthropic → AnthropicDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=263,642—
Anthropic → AnthropicCPU per request (µs)11,322.2Valid—
Anthropic → AnthropicRequests per CPU-second88.32Valid—
Anthropic → AnthropicCost window concurrency8Valid—
Anthropic → AnthropicCost window core utilisation0.25Valid—
Anthropic → AnthropicCost window successful requests537Valid—
Anthropic → AnthropicCost window throughput (req/s)89Valid—
Anthropic → AnthropicCost window threads268Valid—
Anthropic → AnthropicNon-voluntary context switches per request0.039Valid—
Anthropic → AnthropicCost window major page faults0Valid—
Anthropic → AnthropicAdded time to first event p50 (µs)14,510Valid · n=100 gateway / 100 direct—
Anthropic → AnthropicAdded time to first event p99 (µs)15,815Valid · n=100 gateway / 100 direct—
Anthropic → AnthropicAdded inter-frame gap p50 (µs)0Valid · n=63 gateway / 63 direct—
Anthropic → AnthropicAdded inter-frame gap p99 (µs)—Below resolution · n=63 gateway / 63 direct—
Anthropic → AnthropicStreams sustained (frames/s)3,187.6Valid—
Anthropic → AnthropicStreams sustained255Valid—
Anthropic → AnthropicIdle RSS (MiB)592.5Valid—
Anthropic → AnthropicSteady-state RSS (MiB)658.1Valid—
Anthropic → AnthropicRecovered RSS (MiB)658.1Valid—
Anthropic → AnthropicPeak RSS (MiB)658.1Valid—
Anthropic → AnthropicPeak RSS high-water mark (MiB)658.1Valid—
Anthropic → AnthropicTime to plateau (s)62Valid—
Anthropic → AnthropicMemory growth (MiB/min)0Valid—
Anthropic → AnthropicMemory load duration (s)300Valid—
Anthropic → AnthropicMemory shape under load—Not measured—
Anthropic → AnthropicIdle memory shape—Not measured—
Anthropic → AnthropicIdle memory static1Valid—
Anthropic → AnthropicIdle memory growth (MiB/min)0Valid—
Anthropic → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Anthropic → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Anthropic → AnthropicThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Anthropic → AnthropicThroughput (req/s)89concurrency 2; n=53850 ms
Anthropic → AnthropicThroughput (req/s)89concurrency 2; n=538100 ms
Anthropic → AnthropicThroughput (req/s)91concurrency 64; n=550Unbounded
OpenAI Responses → BedrockAdded latency p50 (µs)17,646Valid · n=339 gateway / 267,586 direct—
OpenAI Responses → BedrockAdded latency p99 (µs)18,361Valid · n=339 gateway / 267,586 direct—
OpenAI Responses → BedrockGateway p99 at concurrency 1 (µs)18,388Valid · n=339—
OpenAI Responses → BedrockDirect-to-mock p99 at concurrency 1 (µs)27Valid · n=267,586—
OpenAI Responses → BedrockCPU per request (µs)18,244Valid—
OpenAI Responses → BedrockRequests per CPU-second54.81Valid—
OpenAI Responses → BedrockCost window concurrency8Valid—
OpenAI Responses → BedrockCost window core utilisation0.25Valid—
OpenAI Responses → BedrockCost window successful requests336Valid—
OpenAI Responses → BedrockCost window throughput (req/s)55Valid—
OpenAI Responses → BedrockCost window threads17Valid—
OpenAI Responses → BedrockNon-voluntary context switches per request0.074Valid—
OpenAI Responses → BedrockCost window major page faults0Valid—
OpenAI Responses → BedrockAdded time to first event p50 (µs)—Untestable—
OpenAI Responses → BedrockAdded time to first event p99 (µs)—Untestable—
OpenAI Responses → BedrockAdded inter-frame gap p50 (µs)—Untestable—
OpenAI Responses → BedrockAdded inter-frame gap p99 (µs)—Untestable—
OpenAI Responses → BedrockStreams sustained (frames/s)—Untestable—
OpenAI Responses → BedrockStreams sustained—Untestable—
OpenAI Responses → BedrockIdle RSS (MiB)593Valid—
OpenAI Responses → BedrockSteady-state RSS (MiB)689.4Valid—
OpenAI Responses → BedrockRecovered RSS (MiB)689.4Valid—
OpenAI Responses → BedrockPeak RSS (MiB)689.4Valid—
OpenAI Responses → BedrockPeak RSS high-water mark (MiB)689.4Valid—
OpenAI Responses → BedrockTime to plateau (s)64Valid—
OpenAI Responses → BedrockMemory growth (MiB/min)0Valid—
OpenAI Responses → BedrockMemory load duration (s)301Valid—
OpenAI Responses → BedrockMemory shape under load—Not measured—
OpenAI Responses → BedrockIdle memory shape—Not measured—
OpenAI Responses → BedrockIdle memory static1Valid—
OpenAI Responses → BedrockIdle memory growth (MiB/min)0Valid—
OpenAI Responses → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI Responses → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI Responses → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
OpenAI Responses → BedrockThroughput (req/s)56concurrency 1; n=33850 ms
OpenAI Responses → BedrockThroughput (req/s)56concurrency 1; n=338100 ms
OpenAI Responses → BedrockThroughput (req/s)57concurrency 2,048; n=345Unbounded
Cohere → Bedrock——Not declared: recorded note 1—
Anthropic → BedrockAdded latency p50 (µs)16,315Valid · n=366 gateway / 262,394 direct—
Anthropic → BedrockAdded latency p99 (µs)17,156Valid · n=366 gateway / 262,394 direct—
Anthropic → BedrockGateway p99 at concurrency 1 (µs)17,184Valid · n=366—
Anthropic → BedrockDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=262,394—
Anthropic → BedrockCPU per request (µs)18,047.3Valid—
Anthropic → BedrockRequests per CPU-second55.41Valid—
Anthropic → BedrockCost window concurrency8Valid—
Anthropic → BedrockCost window core utilisation0.25Valid—
Anthropic → BedrockCost window successful requests338Valid—
Anthropic → BedrockCost window throughput (req/s)56Valid—
Anthropic → BedrockCost window threads17Valid—
Anthropic → BedrockNon-voluntary context switches per request0.074Valid—
Anthropic → BedrockCost window major page faults0Valid—
Anthropic → BedrockAdded time to first event p50 (µs)—Untestable—
Anthropic → BedrockAdded time to first event p99 (µs)—Untestable—
Anthropic → BedrockAdded inter-frame gap p50 (µs)—Untestable—
Anthropic → BedrockAdded inter-frame gap p99 (µs)—Untestable—
Anthropic → BedrockStreams sustained (frames/s)—Untestable—
Anthropic → BedrockStreams sustained—Untestable—
Anthropic → BedrockIdle RSS (MiB)592.7Valid—
Anthropic → BedrockSteady-state RSS (MiB)687.5Valid—
Anthropic → BedrockRecovered RSS (MiB)687.5Valid—
Anthropic → BedrockPeak RSS (MiB)687.5Valid—
Anthropic → BedrockPeak RSS high-water mark (MiB)687.5Valid—
Anthropic → BedrockTime to plateau (s)63Valid—
Anthropic → BedrockMemory growth (MiB/min)0Valid—
Anthropic → BedrockMemory load duration (s)305Valid—
Anthropic → BedrockMemory shape under load—Not measured—
Anthropic → BedrockIdle memory shape—Not measured—
Anthropic → BedrockIdle memory static1Valid—
Anthropic → BedrockIdle memory growth (MiB/min)0Valid—
Anthropic → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Anthropic → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Anthropic → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Anthropic → BedrockThroughput (req/s)62concurrency 2; n=37650 ms
Anthropic → BedrockThroughput (req/s)62concurrency 2; n=376100 ms
Anthropic → BedrockThroughput (req/s)63concurrency 64; n=382Unbounded
Gemini → BedrockAdded latency p50 (µs)21,239Valid · n=281 gateway / 267,100 direct—
Gemini → BedrockAdded latency p99 (µs)22,150Valid · n=281 gateway / 267,100 direct—
Gemini → BedrockGateway p99 at concurrency 1 (µs)22,177Valid · n=281—
Gemini → BedrockDirect-to-mock p99 at concurrency 1 (µs)27Valid · n=267,100—
Gemini → BedrockCPU per request (µs)22,086.3Valid—
Gemini → BedrockRequests per CPU-second45.28Valid—
Gemini → BedrockCost window concurrency8Valid—
Gemini → BedrockCost window core utilisation0.251Valid—
Gemini → BedrockCost window successful requests278Valid—
Gemini → BedrockCost window throughput (req/s)46Valid—
Gemini → BedrockCost window threads17Valid—
Gemini → BedrockNon-voluntary context switches per request0.079Valid—
Gemini → BedrockCost window major page faults0Valid—
Gemini → BedrockAdded time to first event p50 (µs)—Untestable—
Gemini → BedrockAdded time to first event p99 (µs)—Untestable—
Gemini → BedrockAdded inter-frame gap p50 (µs)—Untestable—
Gemini → BedrockAdded inter-frame gap p99 (µs)—Untestable—
Gemini → BedrockStreams sustained (frames/s)—Untestable—
Gemini → BedrockStreams sustained—Untestable—
Gemini → BedrockIdle RSS (MiB)591.9Valid—
Gemini → BedrockSteady-state RSS (MiB)692.8Valid—
Gemini → BedrockRecovered RSS (MiB)692.8Valid—
Gemini → BedrockPeak RSS (MiB)692.8Valid—
Gemini → BedrockPeak RSS high-water mark (MiB)692.8Valid—
Gemini → BedrockTime to plateau (s)64Valid—
Gemini → BedrockMemory growth (MiB/min)0.005Valid—
Gemini → BedrockMemory load duration (s)302Valid—
Gemini → BedrockMemory shape under load—Not measured—
Gemini → BedrockIdle memory shape—Not measured—
Gemini → BedrockIdle memory static1Valid—
Gemini → BedrockIdle memory growth (MiB/min)0Valid—
Gemini → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Gemini → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Gemini → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Gemini → BedrockThroughput (req/s)46concurrency 1; n=28050 ms
Gemini → BedrockThroughput (req/s)46concurrency 1; n=280100 ms
Gemini → BedrockThroughput (req/s)46concurrency 1; n=280Unbounded
Bedrock → BedrockAdded latency p50 (µs)10,032Valid · n=595 gateway / 267,885 direct—
Bedrock → BedrockAdded latency p99 (µs)10,847Valid · n=595 gateway / 267,885 direct—
Bedrock → BedrockGateway p99 at concurrency 1 (µs)10,874Valid · n=595—
Bedrock → BedrockDirect-to-mock p99 at concurrency 1 (µs)27Valid · n=267,885—
Bedrock → BedrockCPU per request (µs)10,134Valid—
Bedrock → BedrockRequests per CPU-second98.68Valid—
Bedrock → BedrockCost window concurrency8Valid—
Bedrock → BedrockCost window core utilisation0.25Valid—
Bedrock → BedrockCost window successful requests597Valid—
Bedrock → BedrockCost window throughput (req/s)99Valid—
Bedrock → BedrockCost window threads13Valid—
Bedrock → BedrockNon-voluntary context switches per request0.045Valid—
Bedrock → BedrockCost window major page faults0Valid—
Bedrock → BedrockAdded time to first event p50 (µs)—Untestable—
Bedrock → BedrockAdded time to first event p99 (µs)—Untestable—
Bedrock → BedrockAdded inter-frame gap p50 (µs)—Untestable—
Bedrock → BedrockAdded inter-frame gap p99 (µs)—Untestable—
Bedrock → BedrockStreams sustained (frames/s)—Untestable—
Bedrock → BedrockStreams sustained—Untestable—
Bedrock → BedrockIdle RSS (MiB)592.5Valid—
Bedrock → BedrockSteady-state RSS (MiB)671.4Valid—
Bedrock → BedrockRecovered RSS (MiB)671.4Valid—
Bedrock → BedrockPeak RSS (MiB)671.4Valid—
Bedrock → BedrockPeak RSS high-water mark (MiB)671.4Valid—
Bedrock → BedrockTime to plateau (s)61Valid—
Bedrock → BedrockMemory growth (MiB/min)0Valid—
Bedrock → BedrockMemory load duration (s)303Valid—
Bedrock → BedrockMemory shape under load—Not measured—
Bedrock → BedrockIdle memory shape—Not measured—
Bedrock → BedrockIdle memory static1Valid—
Bedrock → BedrockIdle memory growth (MiB/min)0Valid—
Bedrock → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Bedrock → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Bedrock → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Bedrock → BedrockThroughput (req/s)101concurrency 2; n=60750 ms
Bedrock → BedrockThroughput (req/s)101concurrency 2; n=607100 ms
Bedrock → BedrockThroughput (req/s)104concurrency 128; n=625Unbounded
OpenAI → BedrockAdded latency p50 (µs)14,005Valid · n=427 gateway / 272,464 direct—
OpenAI → BedrockAdded latency p99 (µs)14,745Valid · n=427 gateway / 272,464 direct—
OpenAI → BedrockGateway p99 at concurrency 1 (µs)14,772Valid · n=427—
OpenAI → BedrockDirect-to-mock p99 at concurrency 1 (µs)27Valid · n=272,464—
OpenAI → BedrockCPU per request (µs)14,262.3Valid—
OpenAI → BedrockRequests per CPU-second70.11Valid—
OpenAI → BedrockCost window concurrency8Valid—
OpenAI → BedrockCost window core utilisation0.25Valid—
OpenAI → BedrockCost window successful requests427Valid—
OpenAI → BedrockCost window throughput (req/s)71Valid—
OpenAI → BedrockCost window threads17Valid—
OpenAI → BedrockNon-voluntary context switches per request0.047Valid—
OpenAI → BedrockCost window major page faults0Valid—
OpenAI → BedrockAdded time to first event p50 (µs)—Untestable—
OpenAI → BedrockAdded time to first event p99 (µs)—Untestable—
OpenAI → BedrockAdded inter-frame gap p50 (µs)—Untestable—
OpenAI → BedrockAdded inter-frame gap p99 (µs)—Untestable—
OpenAI → BedrockStreams sustained (frames/s)—Untestable—
OpenAI → BedrockStreams sustained—Untestable—
OpenAI → BedrockIdle RSS (MiB)593.2Valid—
OpenAI → BedrockSteady-state RSS (MiB)689.1Valid—
OpenAI → BedrockRecovered RSS (MiB)689.1Valid—
OpenAI → BedrockPeak RSS (MiB)689.1Valid—
OpenAI → BedrockPeak RSS high-water mark (MiB)689.1Valid—
OpenAI → BedrockTime to plateau (s)63Valid—
OpenAI → BedrockMemory growth (MiB/min)0Valid—
OpenAI → BedrockMemory load duration (s)302Valid—
OpenAI → BedrockMemory shape under load—Not measured—
OpenAI → BedrockIdle memory shape—Not measured—
OpenAI → BedrockIdle memory static1Valid—
OpenAI → BedrockIdle memory growth (MiB/min)0Valid—
OpenAI → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI → BedrockThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
OpenAI → BedrockThroughput (req/s)72concurrency 2; n=43450 ms
OpenAI → BedrockThroughput (req/s)72concurrency 2; n=434100 ms
OpenAI → BedrockThroughput (req/s)72concurrency 2; n=434Unbounded
OpenAI → OpenAIAdded latency p50 (µs)8,512Valid · n=701 gateway / 267,297 direct—
OpenAI → OpenAIAdded latency p99 (µs)9,522Valid · n=701 gateway / 267,297 direct—
OpenAI → OpenAIGateway p99 at concurrency 1 (µs)9,549Valid · n=701—
OpenAI → OpenAIDirect-to-mock p99 at concurrency 1 (µs)27Valid · n=267,297—
OpenAI → OpenAICPU per request (µs)8,644.8Valid—
OpenAI → OpenAIRequests per CPU-second115.7Valid—
OpenAI → OpenAICost window concurrency8Valid—
OpenAI → OpenAICost window core utilisation0.25Valid—
OpenAI → OpenAICost window successful requests701Valid—
OpenAI → OpenAICost window throughput (req/s)116Valid—
OpenAI → OpenAICost window threads32Valid—
OpenAI → OpenAINon-voluntary context switches per request0.029Valid—
OpenAI → OpenAICost window major page faults0Valid—
OpenAI → OpenAIAdded time to first event p50 (µs)11,154Valid · n=100 gateway / 100 direct—
OpenAI → OpenAIAdded time to first event p99 (µs)11,974Valid · n=100 gateway / 100 direct—
OpenAI → OpenAIAdded inter-frame gap p50 (µs)—Below resolution · n=63 gateway / 63 direct—
OpenAI → OpenAIAdded inter-frame gap p99 (µs)9Valid · n=63 gateway / 63 direct—
OpenAI → OpenAIStreams sustained (frames/s)1,208Valid—
OpenAI → OpenAIStreams sustained62Valid—
OpenAI → OpenAIIdle RSS (MiB)593Valid—
OpenAI → OpenAISteady-state RSS (MiB)628.6Valid—
OpenAI → OpenAIRecovered RSS (MiB)628.6Valid—
OpenAI → OpenAIPeak RSS (MiB)628.6Valid—
OpenAI → OpenAIPeak RSS high-water mark (MiB)628.6Valid—
OpenAI → OpenAITime to plateau (s)61Valid—
OpenAI → OpenAIMemory growth (MiB/min)0Valid—
OpenAI → OpenAIMemory load duration (s)302Valid—
OpenAI → OpenAIMemory shape under load—Not measured—
OpenAI → OpenAIIdle memory shape—Not measured—
OpenAI → OpenAIIdle memory static1Valid—
OpenAI → OpenAIIdle memory growth (MiB/min)0Valid—
OpenAI → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI → OpenAIThroughput (req/s)117concurrency 1; n=70810 ms
OpenAI → OpenAIThroughput (req/s)119concurrency 2; n=71850 ms
OpenAI → OpenAIThroughput (req/s)119concurrency 2; n=718100 ms
OpenAI → OpenAIThroughput (req/s)119concurrency 2; n=718Unbounded
Anthropic → OpenAI——Not served: probed and answered HTTP 200, but the body is not a usable anthropic response (see body_snippet)—
OpenAI Responses → OpenAIAdded latency p50 (µs)10,676Valid · n=559 gateway / 265,389 direct—
OpenAI Responses → OpenAIAdded latency p99 (µs)11,606Valid · n=559 gateway / 265,389 direct—
OpenAI Responses → OpenAIGateway p99 at concurrency 1 (µs)11,634Valid · n=559—
OpenAI Responses → OpenAIDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=265,389—
OpenAI Responses → OpenAICPU per request (µs)10,555.6Valid—
OpenAI Responses → OpenAIRequests per CPU-second94.74Valid—
OpenAI Responses → OpenAICost window concurrency8Valid—
OpenAI Responses → OpenAICost window core utilisation0.25Valid—
OpenAI Responses → OpenAICost window successful requests576Valid—
OpenAI Responses → OpenAICost window throughput (req/s)95Valid—
OpenAI Responses → OpenAICost window threads13Valid—
OpenAI Responses → OpenAINon-voluntary context switches per request0.059Valid—
OpenAI Responses → OpenAICost window major page faults0Valid—
OpenAI Responses → OpenAIAdded time to first event p50 (µs)—Untestable—
OpenAI Responses → OpenAIAdded time to first event p99 (µs)—Untestable—
OpenAI Responses → OpenAIAdded inter-frame gap p50 (µs)—Untestable—
OpenAI Responses → OpenAIAdded inter-frame gap p99 (µs)—Untestable—
OpenAI Responses → OpenAIStreams sustained (frames/s)—Untestable—
OpenAI Responses → OpenAIStreams sustained—Untestable—
OpenAI Responses → OpenAIIdle RSS (MiB)591.9Valid—
OpenAI Responses → OpenAISteady-state RSS (MiB)630Valid—
OpenAI Responses → OpenAIRecovered RSS (MiB)630Valid—
OpenAI Responses → OpenAIPeak RSS (MiB)630Valid—
OpenAI Responses → OpenAIPeak RSS high-water mark (MiB)630Valid—
OpenAI Responses → OpenAITime to plateau (s)62Valid—
OpenAI Responses → OpenAIMemory growth (MiB/min)0.006Valid—
OpenAI Responses → OpenAIMemory load duration (s)304Valid—
OpenAI Responses → OpenAIMemory shape under load—Not measured—
OpenAI Responses → OpenAIIdle memory shape—Not measured—
OpenAI Responses → OpenAIIdle memory static1Valid—
OpenAI Responses → OpenAIIdle memory growth (MiB/min)0Valid—
OpenAI Responses → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI Responses → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI Responses → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
OpenAI Responses → OpenAIThroughput (req/s)95concurrency 2; n=57250 ms
OpenAI Responses → OpenAIThroughput (req/s)95concurrency 2; n=572100 ms
OpenAI Responses → OpenAIThroughput (req/s)95concurrency 2; n=572Unbounded
Cohere → OpenAI——Not declared: recorded note 1—
Bedrock → OpenAI——Not declared: recorded note 1—
Gemini → OpenAIAdded latency p50 (µs)14,407Valid · n=412 gateway / 263,451 direct—
Gemini → OpenAIAdded latency p99 (µs)15,293Valid · n=412 gateway / 263,451 direct—
Gemini → OpenAIGateway p99 at concurrency 1 (µs)15,321Valid · n=412—
Gemini → OpenAIDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=263,451—
Gemini → OpenAICPU per request (µs)15,086.4Valid—
Gemini → OpenAIRequests per CPU-second66.28Valid—
Gemini → OpenAICost window concurrency8Valid—
Gemini → OpenAICost window core utilisation0.25Valid—
Gemini → OpenAICost window successful requests405Valid—
Gemini → OpenAICost window throughput (req/s)67Valid—
Gemini → OpenAICost window threads13Valid—
Gemini → OpenAINon-voluntary context switches per request0.044Valid—
Gemini → OpenAICost window major page faults0Valid—
Gemini → OpenAIAdded time to first event p50 (µs)—Untestable—
Gemini → OpenAIAdded time to first event p99 (µs)—Untestable—
Gemini → OpenAIAdded inter-frame gap p50 (µs)—Untestable—
Gemini → OpenAIAdded inter-frame gap p99 (µs)—Untestable—
Gemini → OpenAIStreams sustained (frames/s)—Untestable—
Gemini → OpenAIStreams sustained—Untestable—
Gemini → OpenAIIdle RSS (MiB)592Valid—
Gemini → OpenAISteady-state RSS (MiB)632.3Valid—
Gemini → OpenAIRecovered RSS (MiB)632.3Valid—
Gemini → OpenAIPeak RSS (MiB)632.3Valid—
Gemini → OpenAIPeak RSS high-water mark (MiB)632.3Valid—
Gemini → OpenAITime to plateau (s)63Valid—
Gemini → OpenAIMemory growth (MiB/min)0.003Valid—
Gemini → OpenAIMemory load duration (s)302Valid—
Gemini → OpenAIMemory shape under load—Not measured—
Gemini → OpenAIIdle memory shape—Not measured—
Gemini → OpenAIIdle memory static1Valid—
Gemini → OpenAIIdle memory growth (MiB/min)0Valid—
Gemini → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Gemini → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Gemini → OpenAIThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Gemini → OpenAIThroughput (req/s)68concurrency 1; n=41050 ms
Gemini → OpenAIThroughput (req/s)68concurrency 1; n=410100 ms
Gemini → OpenAIThroughput (req/s)68concurrency 1; n=410Unbounded
Cohere → OpenAI Responses——Not declared: recorded note 1—
Bedrock → OpenAI Responses——Not declared: recorded note 1—
Gemini → OpenAI ResponsesAdded latency p50 (µs)17,577Valid · n=338 gateway / 262,004 direct—
Gemini → OpenAI ResponsesAdded latency p99 (µs)18,499Valid · n=338 gateway / 262,004 direct—
Gemini → OpenAI ResponsesGateway p99 at concurrency 1 (µs)18,527Valid · n=338—
Gemini → OpenAI ResponsesDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=262,004—
Gemini → OpenAI ResponsesCPU per request (µs)18,005.9Valid—
Gemini → OpenAI ResponsesRequests per CPU-second55.54Valid—
Gemini → OpenAI ResponsesCost window concurrency8Valid—
Gemini → OpenAI ResponsesCost window core utilisation0.251Valid—
Gemini → OpenAI ResponsesCost window successful requests341Valid—
Gemini → OpenAI ResponsesCost window throughput (req/s)56Valid—
Gemini → OpenAI ResponsesCost window threads13Valid—
Gemini → OpenAI ResponsesNon-voluntary context switches per request0.082Valid—
Gemini → OpenAI ResponsesCost window major page faults0Valid—
Gemini → OpenAI ResponsesAdded time to first event p50 (µs)—Untestable—
Gemini → OpenAI ResponsesAdded time to first event p99 (µs)—Untestable—
Gemini → OpenAI ResponsesAdded inter-frame gap p50 (µs)—Untestable—
Gemini → OpenAI ResponsesAdded inter-frame gap p99 (µs)—Untestable—
Gemini → OpenAI ResponsesStreams sustained (frames/s)—Untestable—
Gemini → OpenAI ResponsesStreams sustained—Untestable—
Gemini → OpenAI ResponsesIdle RSS (MiB)592.4Valid—
Gemini → OpenAI ResponsesSteady-state RSS (MiB)674.6Valid—
Gemini → OpenAI ResponsesRecovered RSS (MiB)674.6Valid—
Gemini → OpenAI ResponsesPeak RSS (MiB)674.6Valid—
Gemini → OpenAI ResponsesPeak RSS high-water mark (MiB)674.6Valid—
Gemini → OpenAI ResponsesTime to plateau (s)63Valid—
Gemini → OpenAI ResponsesMemory growth (MiB/min)0Valid—
Gemini → OpenAI ResponsesMemory load duration (s)306Valid—
Gemini → OpenAI ResponsesMemory shape under load—Not measured—
Gemini → OpenAI ResponsesIdle memory shape—Not measured—
Gemini → OpenAI ResponsesIdle memory static1Valid—
Gemini → OpenAI ResponsesIdle memory growth (MiB/min)0Valid—
Gemini → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Gemini → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Gemini → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Gemini → OpenAI ResponsesThroughput (req/s)56concurrency 1; n=33750 ms
Gemini → OpenAI ResponsesThroughput (req/s)56concurrency 1; n=337100 ms
Gemini → OpenAI ResponsesThroughput (req/s)59concurrency 2,048; n=360Unbounded
Anthropic → OpenAI ResponsesAdded latency p50 (µs)13,287Valid · n=449 gateway / 260,854 direct—
Anthropic → OpenAI ResponsesAdded latency p99 (µs)14,010Valid · n=449 gateway / 260,854 direct—
Anthropic → OpenAI ResponsesGateway p99 at concurrency 1 (µs)14,038Valid · n=449—
Anthropic → OpenAI ResponsesDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=260,854—
Anthropic → OpenAI ResponsesCPU per request (µs)13,738.7Valid—
Anthropic → OpenAI ResponsesRequests per CPU-second72.79Valid—
Anthropic → OpenAI ResponsesCost window concurrency8Valid—
Anthropic → OpenAI ResponsesCost window core utilisation0.25Valid—
Anthropic → OpenAI ResponsesCost window successful requests444Valid—
Anthropic → OpenAI ResponsesCost window throughput (req/s)73Valid—
Anthropic → OpenAI ResponsesCost window threads13Valid—
Anthropic → OpenAI ResponsesNon-voluntary context switches per request0.072Valid—
Anthropic → OpenAI ResponsesCost window major page faults0Valid—
Anthropic → OpenAI ResponsesAdded time to first event p50 (µs)—Untestable—
Anthropic → OpenAI ResponsesAdded time to first event p99 (µs)—Untestable—
Anthropic → OpenAI ResponsesAdded inter-frame gap p50 (µs)—Untestable—
Anthropic → OpenAI ResponsesAdded inter-frame gap p99 (µs)—Untestable—
Anthropic → OpenAI ResponsesStreams sustained (frames/s)—Untestable—
Anthropic → OpenAI ResponsesStreams sustained—Untestable—
Anthropic → OpenAI ResponsesIdle RSS (MiB)591.7Valid—
Anthropic → OpenAI ResponsesSteady-state RSS (MiB)661.2Valid—
Anthropic → OpenAI ResponsesRecovered RSS (MiB)661.2Valid—
Anthropic → OpenAI ResponsesPeak RSS (MiB)661.2Valid—
Anthropic → OpenAI ResponsesPeak RSS high-water mark (MiB)661.2Valid—
Anthropic → OpenAI ResponsesTime to plateau (s)63Valid—
Anthropic → OpenAI ResponsesMemory growth (MiB/min)0Valid—
Anthropic → OpenAI ResponsesMemory load duration (s)303Valid—
Anthropic → OpenAI ResponsesMemory shape under load—Not measured—
Anthropic → OpenAI ResponsesIdle memory shape—Not measured—
Anthropic → OpenAI ResponsesIdle memory static1Valid—
Anthropic → OpenAI ResponsesIdle memory growth (MiB/min)0Valid—
Anthropic → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Anthropic → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Anthropic → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Anthropic → OpenAI ResponsesThroughput (req/s)74concurrency 1; n=45050 ms
Anthropic → OpenAI ResponsesThroughput (req/s)74concurrency 1; n=450100 ms
Anthropic → OpenAI ResponsesThroughput (req/s)74concurrency 1; n=450Unbounded
OpenAI → OpenAI ResponsesAdded latency p50 (µs)10,726Valid · n=557 gateway / 261,925 direct—
OpenAI → OpenAI ResponsesAdded latency p99 (µs)11,474Valid · n=557 gateway / 261,925 direct—
OpenAI → OpenAI ResponsesGateway p99 at concurrency 1 (µs)11,502Valid · n=557—
OpenAI → OpenAI ResponsesDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=261,925—
OpenAI → OpenAI ResponsesCPU per request (µs)10,799.3Valid—
OpenAI → OpenAI ResponsesRequests per CPU-second92.6Valid—
OpenAI → OpenAI ResponsesCost window concurrency8Valid—
OpenAI → OpenAI ResponsesCost window core utilisation0.25Valid—
OpenAI → OpenAI ResponsesCost window successful requests563Valid—
OpenAI → OpenAI ResponsesCost window throughput (req/s)93Valid—
OpenAI → OpenAI ResponsesCost window threads13Valid—
OpenAI → OpenAI ResponsesNon-voluntary context switches per request0.053Valid—
OpenAI → OpenAI ResponsesCost window major page faults0Valid—
OpenAI → OpenAI ResponsesAdded time to first event p50 (µs)—Untestable—
OpenAI → OpenAI ResponsesAdded time to first event p99 (µs)—Untestable—
OpenAI → OpenAI ResponsesAdded inter-frame gap p50 (µs)—Untestable—
OpenAI → OpenAI ResponsesAdded inter-frame gap p99 (µs)—Untestable—
OpenAI → OpenAI ResponsesStreams sustained (frames/s)—Untestable—
OpenAI → OpenAI ResponsesStreams sustained—Untestable—
OpenAI → OpenAI ResponsesIdle RSS (MiB)591.8Valid—
OpenAI → OpenAI ResponsesSteady-state RSS (MiB)669.4Valid—
OpenAI → OpenAI ResponsesRecovered RSS (MiB)669.4Valid—
OpenAI → OpenAI ResponsesPeak RSS (MiB)669.4Valid—
OpenAI → OpenAI ResponsesPeak RSS high-water mark (MiB)669.4Valid—
OpenAI → OpenAI ResponsesTime to plateau (s)62Valid—
OpenAI → OpenAI ResponsesMemory growth (MiB/min)0Valid—
OpenAI → OpenAI ResponsesMemory load duration (s)305Valid—
OpenAI → OpenAI ResponsesMemory shape under load—Not measured—
OpenAI → OpenAI ResponsesIdle memory shape—Not measured—
OpenAI → OpenAI ResponsesIdle memory static1Valid—
OpenAI → OpenAI ResponsesIdle memory growth (MiB/min)0Valid—
OpenAI → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
OpenAI → OpenAI ResponsesThroughput (req/s)92concurrency 1; n=55650 ms
OpenAI → OpenAI ResponsesThroughput (req/s)92concurrency 1; n=556100 ms
OpenAI → OpenAI ResponsesThroughput (req/s)93concurrency 16; n=559Unbounded
OpenAI Responses → OpenAI ResponsesAdded latency p50 (µs)11,554Valid · n=518 gateway / 265,921 direct—
OpenAI Responses → OpenAI ResponsesAdded latency p99 (µs)12,280Valid · n=518 gateway / 265,921 direct—
OpenAI Responses → OpenAI ResponsesGateway p99 at concurrency 1 (µs)12,308Valid · n=518—
OpenAI Responses → OpenAI ResponsesDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=265,921—
OpenAI Responses → OpenAI ResponsesCPU per request (µs)11,692.3Valid—
OpenAI Responses → OpenAI ResponsesRequests per CPU-second85.53Valid—
OpenAI Responses → OpenAI ResponsesCost window concurrency8Valid—
OpenAI Responses → OpenAI ResponsesCost window core utilisation0.25Valid—
OpenAI Responses → OpenAI ResponsesCost window successful requests520Valid—
OpenAI Responses → OpenAI ResponsesCost window throughput (req/s)86Valid—
OpenAI Responses → OpenAI ResponsesCost window threads13Valid—
OpenAI Responses → OpenAI ResponsesNon-voluntary context switches per request0.063Valid—
OpenAI Responses → OpenAI ResponsesCost window major page faults0Valid—
OpenAI Responses → OpenAI ResponsesAdded time to first event p50 (µs)—Untestable—
OpenAI Responses → OpenAI ResponsesAdded time to first event p99 (µs)—Untestable—
OpenAI Responses → OpenAI ResponsesAdded inter-frame gap p50 (µs)—Untestable—
OpenAI Responses → OpenAI ResponsesAdded inter-frame gap p99 (µs)—Untestable—
OpenAI Responses → OpenAI ResponsesStreams sustained (frames/s)—Untestable—
OpenAI Responses → OpenAI ResponsesStreams sustained—Untestable—
OpenAI Responses → OpenAI ResponsesIdle RSS (MiB)592.5Valid—
OpenAI Responses → OpenAI ResponsesSteady-state RSS (MiB)627.4Valid—
OpenAI Responses → OpenAI ResponsesRecovered RSS (MiB)629.5Valid—
OpenAI Responses → OpenAI ResponsesPeak RSS (MiB)629.5Valid—
OpenAI Responses → OpenAI ResponsesPeak RSS high-water mark (MiB)629.5Valid—
OpenAI Responses → OpenAI ResponsesTime to plateau (s)62Valid—
OpenAI Responses → OpenAI ResponsesMemory growth (MiB/min)3.04Valid—
OpenAI Responses → OpenAI ResponsesMemory load duration (s)300Valid—
OpenAI Responses → OpenAI ResponsesMemory shape under load—Not measured—
OpenAI Responses → OpenAI ResponsesIdle memory shape—Not measured—
OpenAI Responses → OpenAI ResponsesIdle memory static1Valid—
OpenAI Responses → OpenAI ResponsesIdle memory growth (MiB/min)0Valid—
OpenAI Responses → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI Responses → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI Responses → OpenAI ResponsesThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
OpenAI Responses → OpenAI ResponsesThroughput (req/s)87concurrency 2; n=52450 ms
OpenAI Responses → OpenAI ResponsesThroughput (req/s)87concurrency 2; n=524100 ms
OpenAI Responses → OpenAI ResponsesThroughput (req/s)88concurrency 32; n=529Unbounded
OpenAI Responses → GeminiAdded latency p50 (µs)10,486Valid · n=570 gateway / 265,239 direct—
OpenAI Responses → GeminiAdded latency p99 (µs)11,201Valid · n=570 gateway / 265,239 direct—
OpenAI Responses → GeminiGateway p99 at concurrency 1 (µs)11,229Valid · n=570—
OpenAI Responses → GeminiDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=265,239—
OpenAI Responses → GeminiCPU per request (µs)10,628.3Valid—
OpenAI Responses → GeminiRequests per CPU-second94.09Valid—
OpenAI Responses → GeminiCost window concurrency8Valid—
OpenAI Responses → GeminiCost window core utilisation0.25Valid—
OpenAI Responses → GeminiCost window successful requests573Valid—
OpenAI Responses → GeminiCost window throughput (req/s)95Valid—
OpenAI Responses → GeminiCost window threads13Valid—
OpenAI Responses → GeminiNon-voluntary context switches per request0.042Valid—
OpenAI Responses → GeminiCost window major page faults0Valid—
OpenAI Responses → GeminiAdded time to first event p50 (µs)—Untestable—
OpenAI Responses → GeminiAdded time to first event p99 (µs)—Untestable—
OpenAI Responses → GeminiAdded inter-frame gap p50 (µs)—Untestable—
OpenAI Responses → GeminiAdded inter-frame gap p99 (µs)—Untestable—
OpenAI Responses → GeminiStreams sustained (frames/s)—Untestable—
OpenAI Responses → GeminiStreams sustained—Untestable—
OpenAI Responses → GeminiIdle RSS (MiB)592.4Valid—
OpenAI Responses → GeminiSteady-state RSS (MiB)669.4Valid—
OpenAI Responses → GeminiRecovered RSS (MiB)669.4Valid—
OpenAI Responses → GeminiPeak RSS (MiB)669.4Valid—
OpenAI Responses → GeminiPeak RSS high-water mark (MiB)669.4Valid—
OpenAI Responses → GeminiTime to plateau (s)62Valid—
OpenAI Responses → GeminiMemory growth (MiB/min)0Valid—
OpenAI Responses → GeminiMemory load duration (s)304Valid—
OpenAI Responses → GeminiMemory shape under load—Not measured—
OpenAI Responses → GeminiIdle memory shape—Not measured—
OpenAI Responses → GeminiIdle memory static1Valid—
OpenAI Responses → GeminiIdle memory growth (MiB/min)0Valid—
OpenAI Responses → GeminiThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI Responses → GeminiThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI Responses → GeminiThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
OpenAI Responses → GeminiThroughput (req/s)96concurrency 2; n=57850 ms
OpenAI Responses → GeminiThroughput (req/s)96concurrency 2; n=578100 ms
OpenAI Responses → GeminiThroughput (req/s)96concurrency 2; n=578Unbounded
Gemini → GeminiAdded latency p50 (µs)10,073Valid · n=592 gateway / 267,671 direct—
Gemini → GeminiAdded latency p99 (µs)10,889Valid · n=592 gateway / 267,671 direct—
Gemini → GeminiGateway p99 at concurrency 1 (µs)10,916Valid · n=592—
Gemini → GeminiDirect-to-mock p99 at concurrency 1 (µs)27Valid · n=267,671—
Gemini → GeminiCPU per request (µs)10,236.5Valid—
Gemini → GeminiRequests per CPU-second97.69Valid—
Gemini → GeminiCost window concurrency8Valid—
Gemini → GeminiCost window core utilisation0.25Valid—
Gemini → GeminiCost window successful requests592Valid—
Gemini → GeminiCost window throughput (req/s)98Valid—
Gemini → GeminiCost window threads13Valid—
Gemini → GeminiNon-voluntary context switches per request0.049Valid—
Gemini → GeminiCost window major page faults0Valid—
Gemini → GeminiAdded time to first event p50 (µs)—Untestable—
Gemini → GeminiAdded time to first event p99 (µs)—Untestable—
Gemini → GeminiAdded inter-frame gap p50 (µs)—Untestable—
Gemini → GeminiAdded inter-frame gap p99 (µs)—Untestable—
Gemini → GeminiStreams sustained (frames/s)—Untestable—
Gemini → GeminiStreams sustained—Untestable—
Gemini → GeminiIdle RSS (MiB)591.5Valid—
Gemini → GeminiSteady-state RSS (MiB)627.7Valid—
Gemini → GeminiRecovered RSS (MiB)627.7Valid—
Gemini → GeminiPeak RSS (MiB)627.7Valid—
Gemini → GeminiPeak RSS high-water mark (MiB)627.7Valid—
Gemini → GeminiTime to plateau (s)61Valid—
Gemini → GeminiMemory growth (MiB/min)0Valid—
Gemini → GeminiMemory load duration (s)302Valid—
Gemini → GeminiMemory shape under load—Not measured—
Gemini → GeminiIdle memory shape—Not measured—
Gemini → GeminiIdle memory static1Valid—
Gemini → GeminiIdle memory growth (MiB/min)0Valid—
Gemini → GeminiThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Gemini → GeminiThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Gemini → GeminiThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound10 ms
Gemini → GeminiThroughput (req/s)101concurrency 2; n=61250 ms
Gemini → GeminiThroughput (req/s)101concurrency 2; n=612100 ms
Gemini → GeminiThroughput (req/s)101concurrency 2; n=612Unbounded
Cohere → Gemini——Not declared: recorded note 1—
Bedrock → Gemini——Not declared: recorded note 1—
OpenAI → GeminiAdded latency p50 (µs)8,225Valid · n=725 gateway / 265,737 direct—
OpenAI → GeminiAdded latency p99 (µs)8,995Valid · n=725 gateway / 265,737 direct—
OpenAI → GeminiGateway p99 at concurrency 1 (µs)9,023Valid · n=725—
OpenAI → GeminiDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=265,737—
OpenAI → GeminiCPU per request (µs)8,281Valid—
OpenAI → GeminiRequests per CPU-second120.8Valid—
OpenAI → GeminiCost window concurrency8Valid—
OpenAI → GeminiCost window core utilisation0.25Valid—
OpenAI → GeminiCost window successful requests733Valid—
OpenAI → GeminiCost window throughput (req/s)122Valid—
OpenAI → GeminiCost window threads13Valid—
OpenAI → GeminiNon-voluntary context switches per request0.029Valid—
OpenAI → GeminiCost window major page faults0Valid—
OpenAI → GeminiAdded time to first event p50 (µs)—Untestable—
OpenAI → GeminiAdded time to first event p99 (µs)—Untestable—
OpenAI → GeminiAdded inter-frame gap p50 (µs)—Untestable—
OpenAI → GeminiAdded inter-frame gap p99 (µs)—Untestable—
OpenAI → GeminiStreams sustained (frames/s)—Untestable—
OpenAI → GeminiStreams sustained—Untestable—
OpenAI → GeminiIdle RSS (MiB)591.8Valid—
OpenAI → GeminiSteady-state RSS (MiB)665.4Valid—
OpenAI → GeminiRecovered RSS (MiB)665.4Valid—
OpenAI → GeminiPeak RSS (MiB)665.4Valid—
OpenAI → GeminiPeak RSS high-water mark (MiB)665.4Valid—
OpenAI → GeminiTime to plateau (s)61Valid—
OpenAI → GeminiMemory growth (MiB/min)0Valid—
OpenAI → GeminiMemory load duration (s)302Valid—
OpenAI → GeminiMemory shape under load—Not measured—
OpenAI → GeminiIdle memory shape—Not measured—
OpenAI → GeminiIdle memory static1Valid—
OpenAI → GeminiIdle memory growth (MiB/min)0Valid—
OpenAI → GeminiThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
OpenAI → GeminiThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
OpenAI → GeminiThroughput (req/s)121concurrency 1; n=73210 ms
OpenAI → GeminiThroughput (req/s)123concurrency 2; n=74050 ms
OpenAI → GeminiThroughput (req/s)123concurrency 2; n=740100 ms
OpenAI → GeminiThroughput (req/s)127concurrency 128; n=767Unbounded
Anthropic → GeminiAdded latency p50 (µs)8,964Valid · n=666 gateway / 263,751 direct—
Anthropic → GeminiAdded latency p99 (µs)9,686Valid · n=666 gateway / 263,751 direct—
Anthropic → GeminiGateway p99 at concurrency 1 (µs)9,714Valid · n=666—
Anthropic → GeminiDirect-to-mock p99 at concurrency 1 (µs)28Valid · n=263,751—
Anthropic → GeminiCPU per request (µs)9,144.1Valid—
Anthropic → GeminiRequests per CPU-second109.4Valid—
Anthropic → GeminiCost window concurrency8Valid—
Anthropic → GeminiCost window core utilisation0.251Valid—
Anthropic → GeminiCost window successful requests666Valid—
Anthropic → GeminiCost window throughput (req/s)110Valid—
Anthropic → GeminiCost window threads13Valid—
Anthropic → GeminiNon-voluntary context switches per request0.035Valid—
Anthropic → GeminiCost window major page faults0Valid—
Anthropic → GeminiAdded time to first event p50 (µs)—Untestable—
Anthropic → GeminiAdded time to first event p99 (µs)—Untestable—
Anthropic → GeminiAdded inter-frame gap p50 (µs)—Untestable—
Anthropic → GeminiAdded inter-frame gap p99 (µs)—Untestable—
Anthropic → GeminiStreams sustained (frames/s)—Untestable—
Anthropic → GeminiStreams sustained—Untestable—
Anthropic → GeminiIdle RSS (MiB)593Valid—
Anthropic → GeminiSteady-state RSS (MiB)666.6Valid—
Anthropic → GeminiRecovered RSS (MiB)666.6Valid—
Anthropic → GeminiPeak RSS (MiB)666.6Valid—
Anthropic → GeminiPeak RSS high-water mark (MiB)666.6Valid—
Anthropic → GeminiTime to plateau (s)61Valid—
Anthropic → GeminiMemory growth (MiB/min)0.006Valid—
Anthropic → GeminiMemory load duration (s)302Valid—
Anthropic → GeminiMemory shape under load—Not measured—
Anthropic → GeminiIdle memory shape—Not measured—
Anthropic → GeminiIdle memory static1Valid—
Anthropic → GeminiIdle memory growth (MiB/min)0Valid—
Anthropic → GeminiThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound1 ms
Anthropic → GeminiThroughput (req/s)—Bound not met; concurrency not recorded: every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound5 ms
Anthropic → GeminiThroughput (req/s)111concurrency 1; n=66710 ms
Anthropic → GeminiThroughput (req/s)113concurrency 2; n=68250 ms
Anthropic → GeminiThroughput (req/s)113concurrency 2; n=682100 ms
Anthropic → GeminiThroughput (req/s)113concurrency 2; n=682Unbounded

Probe evidence and instrument provenance

Historical capability notes may reference an earlier product version. They are not a new probe or a current support claim. Full RSS samples remain in the immutable raw JSON.

Ingress → upstreamCell stateProbe HTTP statusRecorded reasonRecorded detailEvidence scopeResponse excerptTested build
Cohere → CohereServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
OpenAI → CohereServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Bedrock → CohereNot declared—recorded note 1recorded note 1recorded note 2—ghcr.io/berriai/litellm:v1.108.3
Anthropic → CohereServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
OpenAI Responses → CohereServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Gemini → CohereServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Cohere → AnthropicNot declared—recorded note 1recorded note 1recorded note 2—ghcr.io/berriai/litellm:v1.108.3
Bedrock → AnthropicNot declared—recorded note 1recorded note 1recorded note 2—ghcr.io/berriai/litellm:v1.108.3
OpenAI Responses → AnthropicServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Gemini → AnthropicServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
OpenAI → AnthropicServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Anthropic → AnthropicServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
OpenAI Responses → BedrockServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Cohere → BedrockNot declared—recorded note 1recorded note 1recorded note 2—ghcr.io/berriai/litellm:v1.108.3
Anthropic → BedrockServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Gemini → BedrockServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Bedrock → BedrockServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
OpenAI → BedrockServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
OpenAI → OpenAIServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Anthropic → OpenAINot served200failedprobed and answered HTTP 200, but the body is not a usable anthropic response (see body_snippet)Recorded run evidencethe 2xx body is not a response of the anthropic dialect with assistant text at content[type=text].text: {"id":"resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDo2MTk3ZDQ1OWNlMzc1OTE2NzVlMTIyMTg3ZDAwOTViNzIyNDhmMTRmYTUwNzQzMDZkMzFlMzlhODkzMTM2OWQ4O3Jlc3BvbnNlX2lkOmNoYXRjbXBsLXg=","type":"message","role":"assistant","model":"gpt-4o-mini","stop_sequence":null,"usage":{"input_tokens":10,"output_tokens":2},"content":[],"stop_reason":"end_turn","stop_details":null}ghcr.io/berriai/litellm:v1.108.3
OpenAI Responses → OpenAIServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Cohere → OpenAINot declared—recorded note 1recorded note 1recorded note 2—ghcr.io/berriai/litellm:v1.108.3
Bedrock → OpenAINot declared—recorded note 1recorded note 1recorded note 2—ghcr.io/berriai/litellm:v1.108.3
Gemini → OpenAIServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Cohere → OpenAI ResponsesNot declared—recorded note 1recorded note 1recorded note 2—ghcr.io/berriai/litellm:v1.108.3
Bedrock → OpenAI ResponsesNot declared—recorded note 1recorded note 1recorded note 2—ghcr.io/berriai/litellm:v1.108.3
Gemini → OpenAI ResponsesServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Anthropic → OpenAI ResponsesServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
OpenAI → OpenAI ResponsesServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
OpenAI Responses → OpenAI ResponsesServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
OpenAI Responses → GeminiServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Gemini → GeminiServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Cohere → GeminiNot declared—recorded note 1recorded note 1recorded note 2—ghcr.io/berriai/litellm:v1.108.3
Bedrock → GeminiNot declared—recorded note 1recorded note 1recorded note 2—ghcr.io/berriai/litellm:v1.108.3
OpenAI → GeminiServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Anthropic → GeminiServed———Recorded run evidence—ghcr.io/berriai/litellm:v1.108.3
Harness commit
d240badd966933fa19eec0606b2d1f3945f18ad2
Mock SHA-256
99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8
Load generator SHA-256
not recorded
net.core.somaxconn
65,535
net.ipv4.tcp_max_syn_backlog
65,535

Stream sweep diagnostics

Per streaming-sweep concurrency, as recorded by the engine: lanes by final HTTP status, and lanes that received no response.

Ingress → upstreamConcurrencyLanes by HTTP statusLanes with no response
OpenAI → Anthropic1HTTP 200: 10
OpenAI → Anthropic2HTTP 200: 20
OpenAI → Anthropic4HTTP 200: 40
OpenAI → Anthropic8HTTP 200: 80
OpenAI → Anthropic16HTTP 200: 160
OpenAI → Anthropic32HTTP 200: 320
OpenAI → Anthropic64HTTP 200: 640
OpenAI → Anthropic128HTTP 200: 1280
OpenAI → Anthropic96HTTP 200: 960
OpenAI → Anthropic112HTTP 200: 1120
OpenAI → Anthropic120HTTP 200: 1200
OpenAI → Anthropic116HTTP 200: 1160
OpenAI → Anthropic118HTTP 200: 1180
OpenAI → Anthropic117HTTP 200: 1170
OpenAI → Anthropic117HTTP 200: 1170
OpenAI → Anthropic117HTTP 200: 1170
Anthropic → Anthropic1HTTP 200: 10
Anthropic → Anthropic2HTTP 200: 20
Anthropic → Anthropic4HTTP 200: 40
Anthropic → Anthropic8HTTP 200: 80
Anthropic → Anthropic16HTTP 200: 160
Anthropic → Anthropic32HTTP 200: 320
Anthropic → Anthropic64HTTP 200: 640
Anthropic → Anthropic128HTTP 200: 1280
Anthropic → Anthropic256HTTP 200: 2560
Anthropic → Anthropic192HTTP 200: 1920
Anthropic → Anthropic224HTTP 200: 2240
Anthropic → Anthropic240HTTP 200: 2400
Anthropic → Anthropic248HTTP 200: 2480
Anthropic → Anthropic252HTTP 200: 2520
Anthropic → Anthropic254HTTP 200: 2540
Anthropic → Anthropic255HTTP 200: 2550
Anthropic → Anthropic255HTTP 200: 2550
Anthropic → Anthropic255HTTP 200: 2550
OpenAI → OpenAI1HTTP 200: 10
OpenAI → OpenAI2HTTP 200: 20
OpenAI → OpenAI4HTTP 200: 40
OpenAI → OpenAI8HTTP 200: 80
OpenAI → OpenAI16HTTP 200: 160
OpenAI → OpenAI32HTTP 200: 320
OpenAI → OpenAI64HTTP 200: 640
OpenAI → OpenAI48HTTP 200: 480
OpenAI → OpenAI56HTTP 200: 560
OpenAI → OpenAI60HTTP 200: 600
OpenAI → OpenAI62HTTP 200: 620
OpenAI → OpenAI63HTTP 200: 630
OpenAI → OpenAI62HTTP 200: 620
OpenAI → OpenAI62HTTP 200: 620

Recorded notes

Text recorded with the cells above. Each note is shown once; rows refer to it by number.

  1. Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json
  2. Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.

Evidence for the current measurements

Configuration recorded by the run

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      use_chat_completions_api: true
      api_base: http://127.0.0.1:8000/v1
      api_key: dummy
  - model_name: gpt-4o-mini-responses
    litellm_params:
      model: openai/responses/gpt-4o-mini
      api_base: http://127.0.0.1:8000/v1
      api_key: dummy
  - model_name: gpt-4o-mini-anthropic
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_base: http://127.0.0.1:8000
      api_key: dummy
  - model_name: gpt-4o-mini-gemini
    litellm_params:
      model: gemini/gemini-1.5-flash
      api_base: http://127.0.0.1:8000
      api_key: dummy
  - model_name: command-r
    litellm_params:
      model: cohere_chat/command-r
      api_base: http://127.0.0.1:8000/v2/chat
      api_key: dummy
  - model_name: gpt-4o-mini-bedrock
    litellm_params:
      model: bedrock/converse/anthropic.claude-3-sonnet-20240229-v1:0
      aws_bedrock_runtime_endpoint: http://127.0.0.1:8000
      aws_access_key_id: AKIAMOCKACCESSKEY
      aws_secret_access_key: mock-secret-access-key
      aws_region_name: us-east-1

# Public test token for the local mock rig, never a real provider credential.
general_settings:
  master_key: sk-otb-local-mock-benchmark

Compare a shared protocol cell

Historical measurements