179 GB · 13 reports

B200 · 179 GB

Qualified results first; original reports and full tables stay below.

Text generation · Qualified results

Speeds compare only within one exact test. Recipe availability is checked on each report against the published recipe executor.

No qualified results yet · Text generation · B200 · 179 GB

13 original reports are kept as evidence but do not qualify for comparison; each report states why. This does not show that the model or GPU cannot run it.

Check model launch support View original reports Measured alternatives for this task Test your own GPU

Measured alternatives on other GPUs

Measured on different GPUs with the same or smaller listed capacity. These are references for choosing hardware, not performance or compatibility evidence for your selected GPU.

WeVisOne-VL-2B

tencent · 6 tested configurations · 1 GPU

RTX PRO 6000 Blackwell Server Edition · 96 GBBF16 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 153tok/s
Best measured · 16 concurrent requests: 319tok/s
Observed GPU occupancy: 85.3GiB
First token: 753 ms

Use this recipe · 16 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	153.44	13.39 s	753 ms	85.25	343.4	1	d2914dd8 ↗
C2	208.65	19.64 s	1.1 s	85.25	360.4	1	d2914dd8 ↗
C4	259.19	33.21 s	1.6 s	85.25	398.9	1	d2914dd8 ↗
C8	295.37	59.5 s	2.1 s	85.25	422	1	d2914dd8 ↗
C12	311.33	86.14 s	2.7 s	85.25	432.4	1	d2914dd8 ↗
C16	319.17	112.4 s	3.3 s	85.25	438	1	d2914dd8 ↗

5 other test configurations on this GPU

BF16 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 162tok/s
Best measured · 18 concurrent requests: 857tok/s
Observed GPU occupancy: 85.3GiB
First token: 71 ms

Use this recipe · 18 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 7 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	161.57	12.72 s	71 ms	85.25	327.8	1	75198c63 ↗
C2	269.43	15.21 s	86 ms	85.25	349.1	1	75198c63 ↗
C4	460.41	17.83 s	106 ms	85.25	427.3	1	75198c63 ↗
C8	644.45	25.52 s	127 ms	85.25	486.9	1	75198c63 ↗
C16	771.04	42.77 s	184 ms	85.25	533.9	1	75198c63 ↗
C17	812.79	43.14 s	195 ms	85.25	550.4	1	75198c63 ↗
C18	856.64	43.41 s	196 ms	85.25	568.9	1	75198c63 ↗

BF16 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 101tok/s
Best measured · 8 concurrent requests: 151tok/s
Observed GPU occupancy: 85.3GiB
First token: 2.3 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 5 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	101.26	20.26 s	2.3 s	85.25	369.1	1	c5c9cf6f ↗
C2	123.11	33.41 s	3.6 s	85.25	404.8	1	c5c9cf6f ↗
C4	140.19	64.26 s	4.5 s	85.25	430	1	c5c9cf6f ↗
C6	142.08	97.79 s	5.3 s	85.25	430.4	1	c5c9cf6f ↗
C8	150.73	122.37 s	6.3 s	85.25	446.7	1	c5c9cf6f ↗

BF16 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 114tok/s
Best measured · 9 concurrent requests: 433tok/s
Observed GPU occupancy: 85.3GiB
First token: 135 ms

Use this recipe · 9 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 7 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	113.51	18.06 s	135 ms	85.25	341.3	1	5fee1315 ↗
C2	153.3	26.73 s	161 ms	85.25	369.5	1	5fee1315 ↗
C4	281.86	29.1 s	202 ms	85.25	483.8	1	5fee1315 ↗
C6	293.74	41.93 s	214 ms	85.25	459.2	1	5fee1315 ↗
C7	340.1	42.27 s	227 ms	85.25	496.1	1	5fee1315 ↗
C8	387.93	42.41 s	237 ms	85.25	534.9	1	5fee1315 ↗
C9	433.07	42.78 s	246 ms	85.25	575.8	1	5fee1315 ↗

BF16 · 128K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 56.1tok/s
Best measured · 4 concurrent requests: 66.8tok/s
Observed GPU occupancy: 85.3GiB
First token: 8 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.09	36.63 s	8 s	85.25	418.9	1	2c661c7e ↗
C2	62.52	67.49 s	12 s	85.25	444.3	1	2c661c7e ↗
C3	63.59	109.31 s	13.5 s	85.25	446.8	1	2c661c7e ↗
C4	66.75	144.63 s	15 s	85.25	461.6	1	2c661c7e ↗

BF16 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 71tok/s
Best measured · 4 concurrent requests: 165tok/s
Observed GPU occupancy: 85.3GiB
First token: 268 ms

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	71.05	28.83 s	268 ms	85.25	356.6	1	1ff0c98a ↗
C2	89.43	45.82 s	323 ms	85.25	399.6	1	1ff0c98a ↗
C3	135.43	45.41 s	379 ms	85.25	470.9	1	1ff0c98a ↗
C4	165.29	49.61 s	420 ms	85.25	524	1	1ff0c98a ↗

Qwen3.8-27B-NVFP4

unsloth · 11 tested configurations · 1 GPU

RTX PRO 6000 Blackwell Server Edition · 96 GBNVFP4 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 52.8tok/s
Best measured · 8 concurrent requests: 215tok/s
Observed GPU occupancy: 87.4GiB
First token: 3.3 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.78	38.8 s	3.3 s	87.37	337.3	1	aee8b33e ↗
C2	87.2	48.71 s	4.9 s	87.37	343.4	1	aee8b33e ↗
C4	142.14	64.91 s	7 s	87.37	375.3	1	aee8b33e ↗
C6	187.69	79.85 s	8.1 s	87.37	402.8	1	aee8b33e ↗
C7	202.31	85.21 s	8.7 s	87.37	411.8	1	aee8b33e ↗
C8	215.05	94.52 s	9.3 s	87.37	421.4	1	aee8b33e ↗

10 other test configurations on this GPU

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57tok/s
Best measured · 8 concurrent requests: 374tok/s
Observed GPU occupancy: 87.4GiB
First token: 222 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.96	36.17 s	222 ms	87.37	322.6	1	bced0642 ↗
C2	104.14	39.34 s	355 ms	87.37	305.9	1	bced0642 ↗
C4	198.42	41.3 s	704 ms	87.37	319.3	1	bced0642 ↗
C6	291.12	42.23 s	1 s	87.37	342.1	1	bced0642 ↗
C7	333.32	43.04 s	1.1 s	87.37	349.1	1	bced0642 ↗
C8	374.04	43.82 s	1.3 s	87.37	353.2	1	bced0642 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.1tok/s
Best measured · 8 concurrent requests: 373tok/s
Observed GPU occupancy: 86.6GiB
First token: 231 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.09	35.88 s	231 ms	86.63	319	1	37f29252 ↗
C4	197.8	41.44 s	650 ms	86.63	330.2	1	37f29252 ↗
C8	372.88	43.98 s	1.3 s	86.63	367.3	1	37f29252 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 56.4tok/s
Best measured · 8 concurrent requests: 371tok/s
Observed GPU occupancy: 86.4GiB
First token: 244 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.4	36.51 s	244 ms	86.35	319.1	1	791b887d ↗
C4	196.45	41.73 s	654 ms	86.35	315.5	1	791b887d ↗
C8	370.68	44.24 s	1.3 s	86.36	348.1	1	791b887d ↗

NVFP4 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 44.9tok/s
Best measured · 4 concurrent requests: 96.8tok/s
Observed GPU occupancy: 87.4GiB
First token: 8.6 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	44.86	45.67 s	8.6 s	87.37	360.5	1	d2ff32ec ↗
C2	67.59	63.21 s	12.7 s	87.37	379.1	1	d2ff32ec ↗
C3	84.95	84.07 s	14.6 s	87.37	398.5	1	d2ff32ec ↗
C4	96.77	105.62 s	16.3 s	87.37	415.8	1	d2ff32ec ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54.7tok/s
Best measured · 8 concurrent requests: 340tok/s
Observed GPU occupancy: 87.4GiB
First token: 329 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.72	37.56 s	329 ms	87.37	322.4	1	d3f6b770 ↗
C2	99.95	40.99 s	522 ms	87.37	315.9	1	d3f6b770 ↗
C4	189.07	43.34 s	971 ms	87.37	337.8	1	d3f6b770 ↗
C6	264.7	46.44 s	1.3 s	87.37	364	1	d3f6b770 ↗
C7	303.08	47.32 s	1.4 s	87.37	373.4	1	d3f6b770 ↗
C8	339.61	48.27 s	1.7 s	87.37	381.2	1	d3f6b770 ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54tok/s
Best measured · 8 concurrent requests: 337tok/s
Observed GPU occupancy: 86.4GiB
First token: 361 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.04	38.07 s	361 ms	86.36	320.7	1	417f69be ↗
C4	187.12	43.81 s	879 ms	86.36	333.2	1	417f69be ↗
C8	336.95	48.66 s	1.6 s	86.36	379.7	1	417f69be ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54.5tok/s
Best measured · 8 concurrent requests: 339tok/s
Observed GPU occupancy: 86.6GiB
First token: 359 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.54	37.55 s	359 ms	86.63	324.9	1	7d1a7c8e ↗
C4	188.48	43.48 s	878 ms	86.63	350.2	1	7d1a7c8e ↗
C8	338.65	48.43 s	1.6 s	86.63	396.9	1	7d1a7c8e ↗

NVFP4 · 128K → 2K tokens · No cache reuse
Output throughput · includes prefill · 1 request
31.5tok/s
Best measured · 2 concurrent requests
41.3tok/s
Observed GPU occupancy
87.4GiB
First token
24.9 s
Use this recipe · 2 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 concurrency levels
Total output tokens divided by the full request window, including prefill and queueing.
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
vllm · vLLM 0.28.0
Protocol
wbench-search-http-v1
Dataset
migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload
131072in-2048out
Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
C1 31.54 64.93 s 24.9 s 87.37 402.7 1 73fcb7c1 ↗
C2 41.27 111.14 s 34 s 87.37 426.3 1 73fcb7c1 ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	31.54	64.93 s	24.9 s	87.37	402.7	1	73fcb7c1 ↗
C2	41.27	111.14 s	34 s	87.37	426.3	1	73fcb7c1 ↗

NVFP4 · 128K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 31.5tok/s
Best measured · 8 concurrent requests: 57.7tok/s
Observed GPU occupancy: 86.6GiB
First token: 24.9 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=isolated:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	31.45	65.12 s	24.9 s	86.63	415.7	1	448b0777 ↗
C4	50.67	220.6 s	45.8 s	86.63	469.4	1	448b0777 ↗
C8	57.74	421.65 s	61.9 s	86.63	494.5	1	448b0777 ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 50tok/s
Best measured · 8 concurrent requests: 287tok/s
Observed GPU occupancy: 86.4GiB
First token: 506 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	49.98	40.98 s	506 ms	86.36	322.7	1	68d1f2c6 ↗
C4	171.33	47.85 s	1 s	86.36	365.5	1	68d1f2c6 ↗
C8	287.31	57.12 s	1.9 s	86.36	425.4	1	68d1f2c6 ↗

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Measurements without a comparable protocol · 130 rows

These measured rates remain visible, but are excluded from comparison curves because the protocol is unknown.

Measurements without a comparable protocol · 130 rows
Model	Format	Engine	Workload	Concurrency	Throughput	GPU power (W)
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	1	86.25 tok/s	605.94
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	8	561.03 tok/s	604.92
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	16	819.46 tok/s	712.84
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	32	1,575.24 tok/s	747.1
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	64	2,779.89 tok/s	777.44
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	128	4,716.12 tok/s	851.46
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	256	6,146.38 tok/s	886.92
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	512	7,161.97 tok/s	870.65
Qwen3-30B-A3B-Thinking-FP8 ↗	FP8	vLLM	2K	1	175.82 tok/s	301.31
Qwen3-30B-A3B-Thinking-FP8 ↗	FP8	vLLM	2K	8	1,001.06 tok/s	445.14
Qwen3-30B-A3B-Thinking-FP8 ↗	FP8	vLLM	2K	16	1,667.57 tok/s	498.63
Qwen3-30B-A3B-Thinking-FP8 ↗	FP8	vLLM	2K	32	2,773.67 tok/s	535.68
Qwen3-30B-A3B-Thinking-FP8 ↗	FP8	vLLM	2K	64	4,565.05 tok/s	584.34
Qwen3-30B-A3B-Thinking-FP8 ↗	FP8	vLLM	2K	128	6,874.71 tok/s	623.02
Qwen3-30B-A3B-Thinking-FP8 ↗	FP8	vLLM	2K	256	9,988.06 tok/s	671.47
Qwen3-30B-A3B-Thinking-FP8 ↗	FP8	vLLM	2K	512	11,637 tok/s	607.38
Qwen3-30B-A3B-Thinking-FP8 ↗	FP8	vLLM	2K	1024	11,963.6 tok/s	548.46
Qwen3-30B-A3B-FP8 ↗	FP8	vLLM	2K	1	176.59 tok/s	300.15
Qwen3-30B-A3B-FP8 ↗	FP8	vLLM	2K	8	994.63 tok/s	438.22
Qwen3-30B-A3B-FP8 ↗	FP8	vLLM	2K	16	1,614.37 tok/s	487.74
Qwen3-30B-A3B-FP8 ↗	FP8	vLLM	2K	32	2,757.49 tok/s	528.38
Qwen3-30B-A3B-FP8 ↗	FP8	vLLM	2K	64	4,442.57 tok/s	572.04
Qwen3-30B-A3B-FP8 ↗	FP8	vLLM	2K	128	6,962.74 tok/s	610.61
Qwen3-30B-A3B-FP8 ↗	FP8	vLLM	2K	256	9,856.11 tok/s	669.92
Qwen3-30B-A3B-FP8 ↗	FP8	vLLM	2K	512	10,174.09 tok/s	495.21
Qwen3-30B-A3B-FP8 ↗	FP8	vLLM	2K	1024	10,874.69 tok/s	525.23
Qwen3-14B-FP8 ↗	FP8	vLLM	2K	1	151.12 tok/s	521.1
Qwen3-14B-FP8 ↗	FP8	vLLM	2K	8	1,035.84 tok/s	515.43
Qwen3-14B-FP8 ↗	FP8	vLLM	2K	16	1,632.82 tok/s	621.61
Qwen3-14B-FP8 ↗	FP8	vLLM	2K	32	2,913.66 tok/s	637.78
Qwen3-14B-FP8 ↗	FP8	vLLM	2K	64	4,854.17 tok/s	665.57
Qwen3-14B-FP8 ↗	FP8	vLLM	2K	128	7,593.55 tok/s	716.96
Qwen3-14B-FP8 ↗	FP8	vLLM	2K	256	10,715.86 tok/s	785.78
Qwen3-14B-FP8 ↗	FP8	vLLM	2K	512	11,232.71 tok/s	785.91
Qwen3-14B-FP8 ↗	FP8	vLLM	2K	1024	10,755.84 tok/s	820.15
Qwen3-8B-FP8 ↗	FP8	vLLM	2K	1	191.03 tok/s	430.09
Qwen3-8B-FP8 ↗	FP8	vLLM	2K	8	1,302.91 tok/s	431.41
Qwen3-8B-FP8 ↗	FP8	vLLM	2K	16	2,228.64 tok/s	515.94
Qwen3-8B-FP8 ↗	FP8	vLLM	2K	32	3,667.18 tok/s	533.9
Qwen3-8B-FP8 ↗	FP8	vLLM	2K	64	6,532.89 tok/s	571.09
Qwen3-8B-FP8 ↗	FP8	vLLM	2K	128	9,493.86 tok/s	631.44
Qwen3-8B-FP8 ↗	FP8	vLLM	2K	256	12,746.48 tok/s	716.54
Qwen3-8B-FP8 ↗	FP8	vLLM	2K	512	13,760.74 tok/s	753.91
Qwen3-8B-FP8 ↗	FP8	vLLM	2K	1024	13,993.83 tok/s	741.49
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	1	220 tok/s	349.62
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	8	1,455.31 tok/s	359.58
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	16	2,452.99 tok/s	416.31
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	32	4,322.48 tok/s	445.13
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	64	6,833.17 tok/s	490.52
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	128	10,080.89 tok/s	557.61
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	256	13,734.66 tok/s	646.76
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	512	14,896.39 tok/s	698.08
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	1024	16,010.54 tok/s	695.84
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	1536	16,722.14 tok/s	676.44
Qwen3-4B-Thinking-FP8 ↗	FP8	vLLM	2K	2048	16,563.9 tok/s	681.4
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	1	218.99 tok/s	348.31
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	8	1,451.23 tok/s	358.35
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	16	2,450.06 tok/s	416.08
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	32	4,185.98 tok/s	442.27
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	64	6,864.56 tok/s	484.49
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	128	10,026.59 tok/s	548.02
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	256	13,498.94 tok/s	631.75
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	512	14,627.06 tok/s	681.7
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	1024	15,364.48 tok/s	672.92
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	1536	16,324.49 tok/s	655.69
Qwen3-4B-Instruct-FP8 ↗	FP8	vLLM	2K	2048	16,001.3 tok/s	661.91
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	1	219.54 tok/s	347.43
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	8	1,461.35 tok/s	359.15
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	16	2,452.98 tok/s	419.06
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	32	4,457.44 tok/s	446.64
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	64	7,008.89 tok/s	490.64
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	128	10,697.14 tok/s	557.71
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	256	13,676.91 tok/s	650.05
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	512	14,938.73 tok/s	715.11
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	1024	16,311.85 tok/s	699.16
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	1536	17,309.44 tok/s	693.45
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	1	300.42 tok/s	297.88
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	8	2,062.89 tok/s	314.72
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	16	3,689 tok/s	344.59
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	32	6,178.8 tok/s	380.81
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	64	9,473.01 tok/s	433.31
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	128	13,187.67 tok/s	507.17
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	256	17,523.58 tok/s	598.8
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	512	20,254.86 tok/s	641.04
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	1024	21,268.84 tok/s	628.61
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	1536	20,193.98 tok/s	598.3
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	2048	20,174.85 tok/s	596.7
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	1	300.02 tok/s	295.01
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	8	2,062.8 tok/s	313.19
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	16	3,688.52 tok/s	343.57
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	32	6,183.45 tok/s	380.69
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	64	9,459.57 tok/s	433.24
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	128	13,235.65 tok/s	508.24
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	256	17,549.57 tok/s	597.61
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	512	20,290.64 tok/s	643.08
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	1024	20,949.68 tok/s	626.64
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	1536	21,003.95 tok/s	618.77
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	2048	20,824.6 tok/s	614.39
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	1	367.39 tok/s	252.35
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	8	2,671.29 tok/s	277.36
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	16	4,988.18 tok/s	304.72
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	32	8,519.59 tok/s	343.44
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	64	13,500.82 tok/s	400.79
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	128	19,526.15 tok/s	478.54
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	256	25,287.66 tok/s	561.38
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	512	29,438.45 tok/s	610.5
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	1024	30,516.57 tok/s	620.56
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	1536	29,890.3 tok/s	622.68
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	2048	30,014.61 tok/s	611.16
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	1	367.01 tok/s	248.66
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	8	2,669.65 tok/s	274.09
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	16	4,969.83 tok/s	302.41
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	32	8,504.87 tok/s	340.19
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	64	13,504.45 tok/s	401.12
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	128	19,496.51 tok/s	477.26
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	256	25,217.21 tok/s	563.13
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	512	29,464.31 tok/s	607.77
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	1024	30,751.33 tok/s	630.44
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	1536	29,794.82 tok/s	614.45
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	2048	28,948.44 tok/s	596.72
Hunyuan-1.8B-FP8 ↗	FP8	vLLM	2K	1	336.37 tok/s	302.25
Hunyuan-1.8B-FP8 ↗	FP8	vLLM	2K	8	2,166.46 tok/s	311.98
Hunyuan-1.8B-FP8 ↗	FP8	vLLM	2K	16	3,985.97 tok/s	328.59
Hunyuan-1.8B-FP8 ↗	FP8	vLLM	2K	32	6,676.99 tok/s	353.33
Hunyuan-1.8B-FP8 ↗	FP8	vLLM	2K	64	9,997.94 tok/s	394.42
Hunyuan-1.8B-FP8 ↗	FP8	vLLM	2K	128	14,066.13 tok/s	448.5
Hunyuan-1.8B-FP8 ↗	FP8	vLLM	2K	256	18,008.44 tok/s	506.75
Hunyuan-1.8B-FP8 ↗	FP8	vLLM	2K	512	15,132.27 tok/s	447.28
Hunyuan-1.8B-FP8 ↗	FP8	vLLM	2K	1024	12,735.32 tok/s	390.62
Hunyuan-1.8B-FP8 ↗	FP8	vLLM	2K	1536	13,726.93 tok/s	392.62

Original reports

Every report for this task, including quality, diagnostic and legacy runs that do not enter comparisons.

Original reports · 13 rows
Model	Test task	Format	Engine	Workload	Protocol	Quality score	Test date
Hunyuan-1.8B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/14/2026, 8:02:56 PM
Qwen3-0.6B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/15/2026, 1:06:38 AM
Qwen3-0.6B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/14/2026, 7:41:48 PM
Qwen3-1.7B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/15/2026, 12:56:08 AM
Qwen3-1.7B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/14/2026, 7:50:46 PM
Qwen3-4B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/14/2026, 8:31:06 PM
Qwen3-4B-Instruct-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/15/2026, 12:33:25 AM
Qwen3-4B-Thinking-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/15/2026, 12:44:52 AM
Qwen3-8B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/14/2026, 8:42:55 PM
Qwen3-14B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/14/2026, 8:51:34 PM
Qwen3-30B-A3B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/15/2026, 12:24:54 AM
Qwen3-30B-A3B-Thinking-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/15/2026, 12:16:00 AM
Qwen3-32B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/14/2026, 9:01:19 PM

Running benchmarks

0Updates automatically

No benchmarks are currently reported as running.

Methodology

Average of error-free runs for the selected recorded configuration. Missing methods and fallback runs are not ranked. Performance index = average / fastest matching hardware × 100. Value index = performance per purchase cost / best comparable value × 100. Quality is separate by dataset and protocol. Prices are dated seller listings, not completed transactions or market averages.

Updated 11/5/2026, 9:56:22 PM