Run an AI model on your computer

1. WeVisOne-VL-2B

   tencent · 6 tested configurations

   • BF16 · 32K → 2K tokens · No cache reuse

     Output throughput · includes prefill · 1 request

     153tok/s

     Best measured · 16 concurrent requests

     319tok/s

     Observed GPU occupancy

     85.3GiB

     First token

     753 ms

     Use this recipe · 16 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
     Test details · 6 concurrency levels

     Total output tokens divided by the full request window, including prefill and queueing.

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     vllm · vLLM 0.28.0

     Protocol

     wbench-search-http-v1

     Dataset

     migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

     Workload

     32768in-2048out

     Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     C1	153.44	13.39 s	753 ms	85.25	343.4	1	d2914dd8 ↗
     C2	208.65	19.64 s	1.1 s	85.25	360.4	1	d2914dd8 ↗
     C4	259.19	33.21 s	1.6 s	85.25	398.9	1	d2914dd8 ↗
     C8	295.37	59.5 s	2.1 s	85.25	422	1	d2914dd8 ↗
     C12	311.33	86.14 s	2.7 s	85.25	432.4	1	d2914dd8 ↗
     C16	319.17	112.4 s	3.3 s	85.25	438	1	d2914dd8 ↗

   • 5 other test configurations on this GPU

     • BF16 · 32K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       162tok/s

       Best measured · 18 concurrent requests

       857tok/s

       Observed GPU occupancy

       85.3GiB

       First token

       71 ms

       Use this recipe · 18 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 7 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3

       Workload

       32768in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	161.57	12.72 s	71 ms	85.25	327.8	1	75198c63 ↗
       C2	269.43	15.21 s	86 ms	85.25	349.1	1	75198c63 ↗
       C4	460.41	17.83 s	106 ms	85.25	427.3	1	75198c63 ↗
       C8	644.45	25.52 s	127 ms	85.25	486.9	1	75198c63 ↗
       C16	771.04	42.77 s	184 ms	85.25	533.9	1	75198c63 ↗
       C17	812.79	43.14 s	195 ms	85.25	550.4	1	75198c63 ↗
       C18	856.64	43.41 s	196 ms	85.25	568.9	1	75198c63 ↗

     • BF16 · 64K → 2K tokens · No cache reuse

       Output throughput · includes prefill · 1 request

       101tok/s

       Best measured · 8 concurrent requests

       151tok/s

       Observed GPU occupancy

       85.3GiB

       First token

       2.3 s

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 5 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

       Workload

       65536in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	101.26	20.26 s	2.3 s	85.25	369.1	1	c5c9cf6f ↗
       C2	123.11	33.41 s	3.6 s	85.25	404.8	1	c5c9cf6f ↗
       C4	140.19	64.26 s	4.5 s	85.25	430	1	c5c9cf6f ↗
       C6	142.08	97.79 s	5.3 s	85.25	430.4	1	c5c9cf6f ↗
       C8	150.73	122.37 s	6.3 s	85.25	446.7	1	c5c9cf6f ↗

     • BF16 · 64K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       114tok/s

       Best measured · 9 concurrent requests

       433tok/s

       Observed GPU occupancy

       85.3GiB

       First token

       135 ms

       Use this recipe · 9 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 7 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3

       Workload

       65536in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	113.51	18.06 s	135 ms	85.25	341.3	1	5fee1315 ↗
       C2	153.3	26.73 s	161 ms	85.25	369.5	1	5fee1315 ↗
       C4	281.86	29.1 s	202 ms	85.25	483.8	1	5fee1315 ↗
       C6	293.74	41.93 s	214 ms	85.25	459.2	1	5fee1315 ↗
       C7	340.1	42.27 s	227 ms	85.25	496.1	1	5fee1315 ↗
       C8	387.93	42.41 s	237 ms	85.25	534.9	1	5fee1315 ↗
       C9	433.07	42.78 s	246 ms	85.25	575.8	1	5fee1315 ↗

     • BF16 · 128K → 2K tokens · No cache reuse

       Output throughput · includes prefill · 1 request

       56.1tok/s

       Best measured · 4 concurrent requests

       66.8tok/s

       Observed GPU occupancy

       85.3GiB

       First token

       8 s

       Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 4 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

       Workload

       131072in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	56.09	36.63 s	8 s	85.25	418.9	1	2c661c7e ↗
       C2	62.52	67.49 s	12 s	85.25	444.3	1	2c661c7e ↗
       C3	63.59	109.31 s	13.5 s	85.25	446.8	1	2c661c7e ↗
       C4	66.75	144.63 s	15 s	85.25	461.6	1	2c661c7e ↗

     • BF16 · 128K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       71tok/s

       Best measured · 4 concurrent requests

       165tok/s

       Observed GPU occupancy

       85.3GiB

       First token

       268 ms

       Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 4 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3

       Workload

       131072in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	71.05	28.83 s	268 ms	85.25	356.6	1	1ff0c98a ↗
       C2	89.43	45.82 s	323 ms	85.25	399.6	1	1ff0c98a ↗
       C3	135.43	45.41 s	379 ms	85.25	470.9	1	1ff0c98a ↗
       C4	165.29	49.61 s	420 ms	85.25	524	1	1ff0c98a ↗

2. Qwen3.8-27B-NVFP4

   unsloth · 11 tested configurations

   • NVFP4 · 32K → 2K tokens · No cache reuse

     Output throughput · includes prefill · 1 request

     52.8tok/s

     Best measured · 8 concurrent requests

     215tok/s

     Observed GPU occupancy

     87.4GiB

     First token

     3.3 s

     Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
     Test details · 6 concurrency levels

     Total output tokens divided by the full request window, including prefill and queueing.

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     vllm · vLLM 0.28.0

     Protocol

     wbench-search-http-v1

     Dataset

     migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

     Workload

     32768in-2048out

     Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     C1	52.78	38.8 s	3.3 s	87.37	337.3	1	aee8b33e ↗
     C2	87.2	48.71 s	4.9 s	87.37	343.4	1	aee8b33e ↗
     C4	142.14	64.91 s	7 s	87.37	375.3	1	aee8b33e ↗
     C6	187.69	79.85 s	8.1 s	87.37	402.8	1	aee8b33e ↗
     C7	202.31	85.21 s	8.7 s	87.37	411.8	1	aee8b33e ↗
     C8	215.05	94.52 s	9.3 s	87.37	421.4	1	aee8b33e ↗

   • 10 other test configurations on this GPU

     • NVFP4 · 32K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       57tok/s

       Best measured · 8 concurrent requests

       374tok/s

       Observed GPU occupancy

       87.4GiB

       First token

       222 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 6 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3

       Workload

       32768in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	56.96	36.17 s	222 ms	87.37	322.6	1	bced0642 ↗
       C2	104.14	39.34 s	355 ms	87.37	305.9	1	bced0642 ↗
       C4	198.42	41.3 s	704 ms	87.37	319.3	1	bced0642 ↗
       C6	291.12	42.23 s	1 s	87.37	342.1	1	bced0642 ↗
       C7	333.32	43.04 s	1.1 s	87.37	349.1	1	bced0642 ↗
       C8	374.04	43.82 s	1.3 s	87.37	353.2	1	bced0642 ↗

     • NVFP4 · 32K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       57.1tok/s

       Best measured · 8 concurrent requests

       373tok/s

       Observed GPU occupancy

       86.6GiB

       First token

       231 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       32768in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	57.09	35.88 s	231 ms	86.63	319	1	37f29252 ↗
       C4	197.8	41.44 s	650 ms	86.63	330.2	1	37f29252 ↗
       C8	372.88	43.98 s	1.3 s	86.63	367.3	1	37f29252 ↗

     • NVFP4 · 32K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       56.4tok/s

       Best measured · 8 concurrent requests

       371tok/s

       Observed GPU occupancy

       86.4GiB

       First token

       244 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       32768in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	56.4	36.51 s	244 ms	86.35	319.1	1	791b887d ↗
       C4	196.45	41.73 s	654 ms	86.35	315.5	1	791b887d ↗
       C8	370.68	44.24 s	1.3 s	86.36	348.1	1	791b887d ↗

     • NVFP4 · 64K → 2K tokens · No cache reuse

       Output throughput · includes prefill · 1 request

       44.9tok/s

       Best measured · 4 concurrent requests

       96.8tok/s

       Observed GPU occupancy

       87.4GiB

       First token

       8.6 s

       Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 4 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

       Workload

       65536in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	44.86	45.67 s	8.6 s	87.37	360.5	1	d2ff32ec ↗
       C2	67.59	63.21 s	12.7 s	87.37	379.1	1	d2ff32ec ↗
       C3	84.95	84.07 s	14.6 s	87.37	398.5	1	d2ff32ec ↗
       C4	96.77	105.62 s	16.3 s	87.37	415.8	1	d2ff32ec ↗

     • NVFP4 · 64K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       54.7tok/s

       Best measured · 8 concurrent requests

       340tok/s

       Observed GPU occupancy

       87.4GiB

       First token

       329 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 6 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3

       Workload

       65536in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	54.72	37.56 s	329 ms	87.37	322.4	1	d3f6b770 ↗
       C2	99.95	40.99 s	522 ms	87.37	315.9	1	d3f6b770 ↗
       C4	189.07	43.34 s	971 ms	87.37	337.8	1	d3f6b770 ↗
       C6	264.7	46.44 s	1.3 s	87.37	364	1	d3f6b770 ↗
       C7	303.08	47.32 s	1.4 s	87.37	373.4	1	d3f6b770 ↗
       C8	339.61	48.27 s	1.7 s	87.37	381.2	1	d3f6b770 ↗

     • NVFP4 · 64K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       54tok/s

       Best measured · 8 concurrent requests

       337tok/s

       Observed GPU occupancy

       86.4GiB

       First token

       361 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       65536in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	54.04	38.07 s	361 ms	86.36	320.7	1	417f69be ↗
       C4	187.12	43.81 s	879 ms	86.36	333.2	1	417f69be ↗
       C8	336.95	48.66 s	1.6 s	86.36	379.7	1	417f69be ↗

     • NVFP4 · 64K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       54.5tok/s

       Best measured · 8 concurrent requests

       339tok/s

       Observed GPU occupancy

       86.6GiB

       First token

       359 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       65536in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	54.54	37.55 s	359 ms	86.63	324.9	1	7d1a7c8e ↗
       C4	188.48	43.48 s	878 ms	86.63	350.2	1	7d1a7c8e ↗
       C8	338.65	48.43 s	1.6 s	86.63	396.9	1	7d1a7c8e ↗

     • NVFP4 · 128K → 2K tokens · No cache reuse

       Output throughput · includes prefill · 1 request

       31.5tok/s

       Best measured · 2 concurrent requests

       41.3tok/s

       Observed GPU occupancy

       87.4GiB

       First token

       24.9 s

       Use this recipe · 2 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 2 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

       Workload

       131072in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	31.54	64.93 s	24.9 s	87.37	402.7	1	73fcb7c1 ↗
       C2	41.27	111.14 s	34 s	87.37	426.3	1	73fcb7c1 ↗

     • NVFP4 · 128K → 2K tokens · No cache reuse

       Output throughput · includes prefill · 1 request

       31.5tok/s

       Best measured · 8 concurrent requests

       57.7tok/s

       Observed GPU occupancy

       86.6GiB

       First token

       24.9 s

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=isolated:repeats=3

       Workload

       131072in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	31.45	65.12 s	24.9 s	86.63	415.7	1	448b0777 ↗
       C4	50.67	220.6 s	45.8 s	86.63	469.4	1	448b0777 ↗
       C8	57.74	421.65 s	61.9 s	86.63	494.5	1	448b0777 ↗

     • NVFP4 · 128K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       50tok/s

       Best measured · 8 concurrent requests

       287tok/s

       Observed GPU occupancy

       86.4GiB

       First token

       506 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       131072in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	49.98	40.98 s	506 ms	86.36	322.7	1	68d1f2c6 ↗
       C4	171.33	47.85 s	1 s	86.36	365.5	1	68d1f2c6 ↗
       C8	287.31	57.12 s	1.9 s	86.36	425.4	1	68d1f2c6 ↗

3. Qwen3.8-27B-NVFP4

   RadixArk · 12 tested configurations

   • NVFP4 · 32K → 2K tokens · No cache reuse

     Output throughput · includes prefill · 1 request

     55.6tok/s

     Best measured · 8 concurrent requests

     225tok/s

     Observed GPU occupancy

     86GiB

     First token

     3.2 s

     Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
     Test details · 6 concurrency levels

     Total output tokens divided by the full request window, including prefill and queueing.

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     vllm · vLLM 0.28.0

     Protocol

     wbench-search-http-v1

     Dataset

     migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

     Workload

     32768in-2048out

     Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     C1	55.59	36.91 s	3.2 s	85.98	341	1	dca51ce5 ↗
     C2	91.37	46.48 s	4.6 s	85.98	342.9	1	dca51ce5 ↗
     C4	149.21	62.63 s	6.5 s	85.98	372.9	1	dca51ce5 ↗
     C6	196.73	76.24 s	7.8 s	85.98	398.9	1	dca51ce5 ↗
     C7	211.87	81.41 s	8.3 s	85.98	408.7	1	dca51ce5 ↗
     C8	224.61	90.5 s	8.9 s	85.98	414.8	1	dca51ce5 ↗

   • 11 other test configurations on this GPU

     • NVFP4 · 32K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       60.1tok/s

       Best measured · 8 concurrent requests

       393tok/s

       Observed GPU occupancy

       86GiB

       First token

       217 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 6 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3

       Workload

       32768in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	60.1	34.34 s	217 ms	85.98	323	1	9cdf45b7 ↗
       C2	109.3	37.48 s	349 ms	85.98	310.8	1	9cdf45b7 ↗
       C4	209.23	39.18 s	686 ms	85.98	324.8	1	9cdf45b7 ↗
       C6	306.94	40.06 s	992 ms	85.98	336	1	9cdf45b7 ↗
       C7	351.24	40.83 s	1.1 s	85.98	342.6	1	9cdf45b7 ↗
       C8	392.93	41.71 s	1.3 s	85.98	348.2	1	9cdf45b7 ↗

     • NVFP4 · 32K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       59.7tok/s

       Best measured · 8 concurrent requests

       394tok/s

       Observed GPU occupancy

       86.8GiB

       First token

       230 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       32768in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	59.68	34.59 s	230 ms	86.82	323.3	1	83d37469 ↗
       C4	207.31	39.54 s	664 ms	86.82	320	1	83d37469 ↗
       C8	394.04	41.6 s	1.2 s	86.82	346.1	1	83d37469 ↗

     • NVFP4 · 32K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       60.5tok/s

       Best measured · 8 concurrent requests

       395tok/s

       Observed GPU occupancy

       86.9GiB

       First token

       228 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       32768in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	60.47	33.87 s	228 ms	86.86	320.3	1	cbf1431c ↗
       C4	207.86	39.44 s	636 ms	86.86	333.6	1	cbf1431c ↗
       C8	394.52	41.6 s	1.3 s	86.86	361.6	1	cbf1431c ↗

     • NVFP4 · 64K → 2K tokens · No cache reuse

       Output throughput · includes prefill · 1 request

       47.1tok/s

       Best measured · 4 concurrent requests

       101tok/s

       Observed GPU occupancy

       86GiB

       First token

       8.4 s

       Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 4 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

       Workload

       65536in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	47.15	43.45 s	8.4 s	85.98	364.4	1	e5a201fa ↗
       C2	70.39	60.71 s	12.3 s	85.98	381.3	1	e5a201fa ↗
       C3	88.36	80.85 s	14.2 s	85.98	403.6	1	e5a201fa ↗
       C4	100.66	101.65 s	15.8 s	85.98	419.2	1	e5a201fa ↗

     • NVFP4 · 64K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       57.5tok/s

       Best measured · 8 concurrent requests

       355tok/s

       Observed GPU occupancy

       86GiB

       First token

       326 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 6 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3

       Workload

       65536in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	57.45	35.8 s	326 ms	85.98	326.1	1	67a5e6d3 ↗
       C2	104.7	39.14 s	515 ms	85.98	321	1	67a5e6d3 ↗
       C4	198.72	41.24 s	954 ms	85.98	344.2	1	67a5e6d3 ↗
       C6	277.65	44.28 s	1.3 s	85.98	358.8	1	67a5e6d3 ↗
       C7	317.41	45.18 s	1.4 s	85.98	369.2	1	67a5e6d3 ↗
       C8	354.95	46.19 s	1.6 s	85.98	377.1	1	67a5e6d3 ↗

     • NVFP4 · 64K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       57.1tok/s

       Best measured · 8 concurrent requests

       356tok/s

       Observed GPU occupancy

       86.8GiB

       First token

       358 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       65536in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	57.08	36.15 s	358 ms	86.82	326	1	275a096d ↗
       C4	196.91	41.63 s	870 ms	86.83	338.7	1	275a096d ↗
       C8	355.89	46.08 s	1.6 s	86.83	375.7	1	275a096d ↗

     • NVFP4 · 64K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       57.7tok/s

       Best measured · 8 concurrent requests

       356tok/s

       Observed GPU occupancy

       86.9GiB

       First token

       357 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       65536in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	57.67	35.51 s	357 ms	86.86	325.8	1	fe8141d2 ↗
       C4	197.42	41.52 s	870 ms	86.86	354.3	1	fe8141d2 ↗
       C8	356.25	46.05 s	1.6 s	86.86	392.4	1	fe8141d2 ↗

     • NVFP4 · 128K → 2K tokens · No cache reuse

       Output throughput · includes prefill · 1 request

       32.7tok/s

       Best measured · 2 concurrent requests

       42.5tok/s

       Observed GPU occupancy

       86GiB

       First token

       24.3 s

       Use this recipe · 2 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 2 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3

       Workload

       131072in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	32.72	62.59 s	24.3 s	85.99	406.5	1	28f51cf1 ↗
       C2	42.54	106.24 s	33.5 s	85.99	428.9	1	28f51cf1 ↗

     • NVFP4 · 128K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       53tok/s

       Best measured · 4 concurrent requests

       181tok/s

       Observed GPU occupancy

       86GiB

       First token

       435 ms

       Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 4 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-search-http-v1

       Dataset

       migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3

       Workload

       131072in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	52.97	38.72 s	435 ms	85.99	333.9	1	78fc908c ↗
       C2	97.25	42.14 s	646 ms	85.99	340.8	1	78fc908c ↗
       C3	138.66	44.36 s	822 ms	85.99	358.8	1	78fc908c ↗
       C4	181.16	45.25 s	1.1 s	85.99	375.3	1	78fc908c ↗

     • NVFP4 · 128K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       52.9tok/s

       Best measured · 8 concurrent requests

       301tok/s

       Observed GPU occupancy

       86.9GiB

       First token

       508 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       131072in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	52.9	38.72 s	508 ms	86.86	338	1	7cea351c ↗
       C4	179.81	45.65 s	1 s	86.86	387.5	1	7cea351c ↗
       C8	301.16	54.48 s	1.9 s	86.86	443.7	1	7cea351c ↗

     • NVFP4 · 128K → 2K tokens · Prefix cache reuse

       Output throughput · includes prefill · 1 request

       52.6tok/s

       Best measured · 8 concurrent requests

       301tok/s

       Observed GPU occupancy

       86.8GiB

       First token

       509 ms

       Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 concurrency levels

       Total output tokens divided by the full request window, including prefill and queueing.

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       vllm · vLLM 0.28.0

       Protocol

       wbench-fixed-http-v3

       Dataset

       synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3

       Workload

       131072in-2048out

       Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       C1	52.59	38.95 s	509 ms	86.83	328.8	1	d14962c1 ↗
       C4	179.42	45.72 s	1.1 s	86.83	371.9	1	d14962c1 ↗
       C8	300.98	54.52 s	1.9 s	86.83	426	1	d14962c1 ↗

Models shown: 3 / 5

Recipe availability is checked on each report against the published recipe executor.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

LLM run scripts require Linux x86_64 and an NVIDIA GPU. API and benchmark availability are checked separately on the report.