Benchmark All DevPlace Models - Session Notes

Made with DevPlaceCode on DPE (DevPlace Playground Environment) — the online web IDE that replaces VS Code. More capable, more integrated. Much success, much wow. Extreme progress!

The Script: bench.py

Benchmarks every AI model on the devplace.net OpenAI-compatible gateway. Sends a prompt to each model, captures X-Gateway-* headers for latency, tokens/sec, cost, cache stats, and outputs formatted tables JSON.

Our Adventure

We needed to know which models were fast, cheap, and reliable. So we built a benchmark runner that hits all 40 models sequentially and collects comprehensive metrics from the gateway response headers.

The script reads model configs from the admin API, then fires each through the /openai/v1/chat/completions endpoint, parsing response headers like X-Gateway-Tokens-Per-Second, X-Gateway-Cost-USD, X-Gateway-Upstream-Latency-Ms, and more.

Output includes: summary table, per-model detailed stats, aggregate statistics (min/max/mean/median/stdev), and best-performer callouts (fastest TPS, cheapest, lowest latency, longest response).

Issues Encountered

1. Gateway header parsing: The X-Gateway-* headers carry all the telemetry. Had to map each header to the right field in the ModelStats dataclass. Some headers return empty strings instead of 0, requiring careful float/int conversion.

2. Timeout tuning: Models vary wildly in response time. Set timeout=120s to handle slow reasoning models without killing fast ones prematurely.

3. No streaming: Benchmarks use stream=False for consistent token counting. Streaming would make accurate TPS measurement harder.

Gist: https://devplace.net/gists/abc978b39a7a-benchmark-all-devplace-models