Run an AI model on your computer

1. Qwen3-4B-AWQ

   Qwen · 5 tested configurations · 3 GPUs

   • L4 · 24 GBAWQ · 8K → 2K tokens · No cache reuse

     Output throughput · includes prefill · 1 request

     55.3tok/s

     Best measured · 4 concurrent requests

     109tok/s

     Observed GPU occupancy

     19.5GiB

     First token

     1.6 s

     Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
     Test details · 2 concurrency levels

     Total output tokens divided by the full request window, including prefill and queueing.

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     vllm · vLLM 0.29.0

     Protocol

     wbench-fixed-http-v3

     Dataset

     migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

     Workload

     8192in-2048out

     Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     C1	55.3	37.17 s	1.6 s	19.54	71.9	1	9301a9ba ↗
     C4	109.43	78.88 s	3.1 s	19.55	71.9	1	9301a9ba ↗

   • 1 other test configurations on this GPU

     • AWQ · 8K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       57.4tok/s

       Best measured · 4 concurrent requests

       175tok/s

       Observed GPU occupancy

       19.5GiB

       First token

       60 ms

       Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 2 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.29.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3

       Workload

       8192in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	57.39	35.7 s	60 ms	19.55	71.8	1	7e2cbfda ↗
       C4	175.39	46.75 s	114 ms	19.55	71.9	1	7e2cbfda ↗

   • RTX PRO 6000 Blackwell Server Edition · 96 GBAWQ · 8K → 2K tokens · No cache reuse

     Output throughput · includes prefill · 1 request

     215tok/s

     Best measured · 4 concurrent requests

     517tok/s

     Observed GPU occupancy

     86.3GiB

     First token

     298 ms

     Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
     Test details · 2 concurrency levels

     Total output tokens divided by the full request window, including prefill and queueing.

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     vllm · vLLM 0.29.0

     Protocol

     wbench-fixed-http-v3

     Dataset

     migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

     Workload

     8192in-2048out

     Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     C1	214.6	9.63 s	298 ms	86.28	303.8	1	86d36e4f ↗
     C4	516.94	16.14 s	872 ms	86.28	339.5	1	86d36e4f ↗

   • 1 other test configurations on this GPU

     • AWQ · 8K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       223tok/s

       Best measured · 4 concurrent requests

       742tok/s

       Observed GPU occupancy

       86.3GiB

       First token

       28 ms

       Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 2 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.29.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3

       Workload

       8192in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	222.74	9.19 s	28 ms	86.28	288.8	1	adbfef8a ↗
       C4	741.85	11.05 s	35 ms	86.28	335	1	adbfef8a ↗

   • T4 · 16 GBAWQ · 8K → 2K tokens · No cache reuse

     Output throughput · includes prefill · 1 request

     15.4tok/s

     Best measured · 4 concurrent requests

     20.3tok/s

     Observed GPU occupancy

     12.7GiB

     First token

     28.5 s

     Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
     Test details · 2 concurrency levels

     Total output tokens divided by the full request window, including prefill and queueing.

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     vllm · vLLM 0.29.0

     Protocol

     wbench-fixed-http-v3

     Dataset

     migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

     Workload

     8192in-2048out

     Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     C1	15.39	133.36 s	28.5 s	12.74	67.9	1	fcfe2d64 ↗
     C4	20.31	468.13 s	53.1 s	12.74	67.5	1	fcfe2d64 ↗

Models shown: 1 / 1

Recipe availability is checked on each report against the published recipe executor.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

LLM run scripts require Linux x86_64 and an NVIDIA GPU. API and benchmark availability are checked separately on the report.