# InferSpeed benchmarks

Measured against the public API with an ordinary API key. Every number links to the raw per-request data and can be re-run with the public script.

- Machine-readable index: https://inferspeed.ai/api/v1/public/benchmarks
- Reproduce: https://inferspeed.ai/benchmarks/reproduce.py (prompt set: https://inferspeed.ai/benchmarks/prompt-set-v1.json)

## Measured by InferSpeed

Our own measurements of the public endpoint: speed, errors, stream correctness, tool calls and billing.

### Results

No reviewed run is published yet. The method and the script below are final; numbers appear here after a run has been reviewed.

## Model quality (vendor-reported)

Not measured by InferSpeed. Scores published by each model's vendor on its official model card, copied verbatim. InferSpeed did not run these evaluations and does not vouch for them. Our own measurements of the endpoint are a separate section.

### glm-5.3 — source model GLM-5.3

Publisher: Z.ai (zai-org) · source: https://huggingface.co/zai-org/glm-5.3 (revision `c5087a76d148760811ffbdedfe6ff9f194e46e38`) · checked 2026-10-14 · label: **vendor-reported**

Serving note: glm-5.3 is served from the vendor's published model through more than one route; a route may run the vendor's FP8 weights on our own hardware. The vendor measured these scores on its own setup (settings below), not on InferSpeed.

| Benchmark | Score (vendor-reported) | Vendor setting |
|---|---|---|
| Terminal Bench 2.1 | 88.2% | Claude Code 2.1.207, temperature 1.0, 64K max new tokens |
| Terminal Bench 3.0 | 28.3% | Claude Code 2.1.207, max reasoning effort, avg@3 |
| DeepSWE (v1.1) | 66.9% | mini-swe-agent, temperature 0.95, 400K context |
| NL2Repo | 58.0% | temperature 1.0, 64K max new tokens, 1M context |
| SWE-Marathon (v1.1) | 42.5% | Claude Code 2.1.207, max effort, 1M context |
| Toolathlon Verified | 73.0% | official evaluation service, pass@1 over 3 runs |
| HLE w/ Tools | 62.5% | temperature 1.0, 300K context with context management |
| CyberGym | 84.5% | Claude Code 2.1.207, max effort, Pass@1 over 1,507 tasks |

## Method (measured section)

- Requests go to the public API endpoint with an ordinary API key, the same path customers use.
- One request in flight at a time (concurrency 1): numbers describe single-user latency, not peak throughput.
- Every request streams; temperature 0; prompts, output caps and repeat counts are fixed by the versioned prompt set.
- Each model is measured on OpenAI Chat Completions, Anthropic Messages and OpenAI Responses.
- Latency includes the network path from the client to the endpoint; the client region is printed next to every run.
- Raw per-request samples are published; every aggregate can be recomputed from them.

| Metric | Definition |
|---|---|
| TTFT | Time from sending the request to the first streamed token of any kind (reasoning, text or tool call). |
| First text | Time from sending the request to the first visible text or tool call (ttfc_ms). For reasoning models the gap to TTFT is time spent reasoning. |
| End-to-end latency | Time from sending the request to the end of the stream. |
| Output tokens/s | Output tokens reported by the API (reasoning included) ÷ (end-to-end latency − TTFT), per request; the table shows the median of perf prompts. |
| Percentiles | Nearest rank over successful requests. p95 is left empty below 5 successful samples. |
| Error rate | Requests that did not succeed ÷ all requests (HTTP error, stream error, missing terminal event, empty answer, or a failed tool call). An empty answer whose stop reason is the output cap is reported as max_tokens_in_reasoning. |
| Tool-call success | Tool prompt answered with a call to get_weather whose JSON arguments contain a non-empty city. |
| Terminal event | The stream ended correctly: Chat Completions finish_reason and [DONE]; Messages message_stop with stop_reason; Responses response.completed (or response.incomplete when the output cap is reached). |
| Billing match | Charged amount equals the billed tokens × the published price (within 0.5%), and the billed tokens equal the usage the API returned. |
| Model fingerprint | Whether the response names the requested model, plus three fixed questions with one correct answer; the hash covers the normalized answers. |

## Reproduce it yourself

Needs Python 3.9+ (standard library only) and an InferSpeed API key. The script refuses to start without a spend cap and stops before it would cross it.

```sh
export INFERSPEED_API_KEY=<your key>
curl -fsSLO https://inferspeed.ai/benchmarks/reproduce.py
python3 reproduce.py                      # dry run: plan + worst-case cost
python3 reproduce.py --execute --max-spend 1 --region "<where you run it>" --out my-run.json --csv my-run.csv
```

Compare your JSON with a published run: same prompt set, same metric definitions. Your latency differs by your distance to the endpoint.