Qwen3-4B-AWQ
Run this model
Choose your operating system and enter a model repository. Launch support is checked separately from performance results.
Test conditions
- Task
- Text generation · Text → Text
- Workload
- 8K → 2K tokens · Prefix cache reuse
- Concurrency
- 1 request
- Format · Engine
- AWQ · vllm · vLLM 0.29.0
- Unit
- tok/s
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Protocol details
wbench-fixed-http-v3 · migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3 · 8192in-2048out
GPUs on this test
- Mean throughput
- 223tok/s
- Observed GPU occupancy
- 86.3GiB
- First token
- 28 ms
- Reports combined
- 1
- Mean throughput
- 57.4tok/s
- Observed GPU occupancy
- 19.5GiB
- First token
- 60 ms
- Reports combined
- 1
3 more GPUs ran this model under other tests
Sortable table
| Hardware | Mean throughput (tok/s) | P95 (s) | TTFT (ms) | Observed occupancy (GiB) | GPU power (W) | Throughput / W | VRAM per GPU (GB) | Reports combined |
|---|---|---|---|---|---|---|---|---|
| #1RTX PRO 6000 Blackwell Server Edition · 96 GBRuns, parameters and setup ↗ | 222.74 | 9.194 | 27.67 | 86.28 | 288.8 | 0.77 | 96 | 1 |
| #2L4 · 24 GBRuns, parameters and setup ↗ | 57.39 | 35.702 | 60 | 19.55 | 71.8 | 0.8 | 24 | 1 |
| 7 × RTX 3080 · 10 GBNo matching measurements | — | — | — | — | — | — | 10 | — |
| RTX 4090D · 48 GBNo matching measurements | — | — | — | — | — | — | 48 | — |
| T4 · 16 GBNo matching measurements | — | — | — | — | — | — | 16 | — |
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
How results are grouped
Results merge successful runs of the same model, precision and workload. CPU thread counts and CUDA wheel builds may differ; actual PyTorch builds are shown. Other recorded protocol settings remain separate.
Updated 11/5/2026, 9:57:52 PM