You’re viewing a demo workspace. Every number on this page is sample data. Sign in to use your own data

Load testing

Synthetic traffic against your endpoints to find where latency and cost break before real users do.

Run lt-0142 · Oct 22, 2026

Endpoint

support-agent

8× H100 · Llama 3.1 70B

Holds the SLO up to

64 concurrent

p95 740 ms of 1,200 ms

Throughput at that point

2,432 tok/s

Output tokens, all GPUs

Lowest cost per 1M tokens

$2.55

At 64 concurrent requests

Latency by concurrency

p50 p95
0 ms 2,000 ms 4,000 ms 6,000 ms Last step within SLO SLO 1,200 ms p50 p95 1 2 4 8 16 32 48 64 96 128

Concurrent requests

Throughput by concurrency

Output tokens per second, all GPUs

0 1,000 2,000 3,000 Last step within SLO 1 2 4 8 16 32 48 64 96 128

Concurrent requests

Steps in this run

Each step runs 3 minutes with a 30 second ramp.

Concurrent p50 p95 Tokens/s Per 1M tokens Errors Result
1 184 ms 268 ms 38 $163.16 0.0% Within SLO
2 188 ms 275 ms 76 $81.58 0.0% Within SLO
4 197 ms 290 ms 152 $40.79 0.0% Within SLO
8 214 ms 320 ms 304 $20.39 0.0% Within SLO
16 247 ms 380 ms 608 $10.20 0.0% Within SLO
32 314 ms 500 ms 1,216 $5.10 0.0% Within SLO
48 382 ms 620 ms 1,824 $3.40 0.0% Within SLO
64 449 ms 740 ms 2,432 $2.55 0.0% Within SLO
96 942 ms 1,902 ms 2,237 $2.77 0.4% Over SLO
128 2,151 ms 4,906 ms 2,043 $3.03 3.4% Over SLO

Recent runs

Run Endpoint Date Peak req/s Breaks at Result
lt-0142 support-agent Oct 22 46 64 concurrent Passed
lt-0141 support-agent · 5× H100 Oct 21 31 48 concurrent Passed
lt-0138 sql-copilot Oct 17 22 32 concurrent Degraded
lt-0133 transcribe · 1× A100 Oct 12 24 48 concurrent Passed
lt-0129 moderation Oct 8 190 96 concurrent Failed