96 GB · 196 reports

RTX PRO 6000 Blackwell Server Edition · 96 GB

Qualified results first; original reports and full tables stay below.

All tasks · Qualified results

Speeds compare only within one exact test. Recipe availability is checked on each report against the published recipe executor.

WeVisOne-VL-2B

tencent · 6 tested configurations

BF16 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 71tok/s
Best measured · 4 concurrent requests: 165tok/s
Observed GPU occupancy: 85.3GiB
First token: 268 ms

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	71.05	28.83 s	268 ms	85.25	356.6	1	1ff0c98a ↗
C2	89.43	45.82 s	323 ms	85.25	399.6	1	1ff0c98a ↗
C3	135.43	45.41 s	379 ms	85.25	470.9	1	1ff0c98a ↗
C4	165.29	49.61 s	420 ms	85.25	524	1	1ff0c98a ↗

5 other test configurations on this GPU

BF16 · 128K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 56.1tok/s
Best measured · 4 concurrent requests: 66.8tok/s
Observed GPU occupancy: 85.3GiB
First token: 8 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.09	36.63 s	8 s	85.25	418.9	1	2c661c7e ↗
C2	62.52	67.49 s	12 s	85.25	444.3	1	2c661c7e ↗
C3	63.59	109.31 s	13.5 s	85.25	446.8	1	2c661c7e ↗
C4	66.75	144.63 s	15 s	85.25	461.6	1	2c661c7e ↗

BF16 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 114tok/s
Best measured · 9 concurrent requests: 433tok/s
Observed GPU occupancy: 85.3GiB
First token: 135 ms

Use this recipe · 9 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 7 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	113.51	18.06 s	135 ms	85.25	341.3	1	5fee1315 ↗
C2	153.3	26.73 s	161 ms	85.25	369.5	1	5fee1315 ↗
C4	281.86	29.1 s	202 ms	85.25	483.8	1	5fee1315 ↗
C6	293.74	41.93 s	214 ms	85.25	459.2	1	5fee1315 ↗
C7	340.1	42.27 s	227 ms	85.25	496.1	1	5fee1315 ↗
C8	387.93	42.41 s	237 ms	85.25	534.9	1	5fee1315 ↗
C9	433.07	42.78 s	246 ms	85.25	575.8	1	5fee1315 ↗

BF16 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 101tok/s
Best measured · 8 concurrent requests: 151tok/s
Observed GPU occupancy: 85.3GiB
First token: 2.3 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 5 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	101.26	20.26 s	2.3 s	85.25	369.1	1	c5c9cf6f ↗
C2	123.11	33.41 s	3.6 s	85.25	404.8	1	c5c9cf6f ↗
C4	140.19	64.26 s	4.5 s	85.25	430	1	c5c9cf6f ↗
C6	142.08	97.79 s	5.3 s	85.25	430.4	1	c5c9cf6f ↗
C8	150.73	122.37 s	6.3 s	85.25	446.7	1	c5c9cf6f ↗

BF16 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 162tok/s
Best measured · 18 concurrent requests: 857tok/s
Observed GPU occupancy: 85.3GiB
First token: 71 ms

Use this recipe · 18 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 7 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	161.57	12.72 s	71 ms	85.25	327.8	1	75198c63 ↗
C2	269.43	15.21 s	86 ms	85.25	349.1	1	75198c63 ↗
C4	460.41	17.83 s	106 ms	85.25	427.3	1	75198c63 ↗
C8	644.45	25.52 s	127 ms	85.25	486.9	1	75198c63 ↗
C16	771.04	42.77 s	184 ms	85.25	533.9	1	75198c63 ↗
C17	812.79	43.14 s	195 ms	85.25	550.4	1	75198c63 ↗
C18	856.64	43.41 s	196 ms	85.25	568.9	1	75198c63 ↗

BF16 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 153tok/s
Best measured · 16 concurrent requests: 319tok/s
Observed GPU occupancy: 85.3GiB
First token: 753 ms

Use this recipe · 16 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	153.44	13.39 s	753 ms	85.25	343.4	1	d2914dd8 ↗
C2	208.65	19.64 s	1.1 s	85.25	360.4	1	d2914dd8 ↗
C4	259.19	33.21 s	1.6 s	85.25	398.9	1	d2914dd8 ↗
C8	295.37	59.5 s	2.1 s	85.25	422	1	d2914dd8 ↗
C12	311.33	86.14 s	2.7 s	85.25	432.4	1	d2914dd8 ↗
C16	319.17	112.4 s	3.3 s	85.25	438	1	d2914dd8 ↗

Qwen3.8-27B-NVFP4

unsloth · 11 tested configurations

NVFP4 · 128K → 2K tokens · No cache reuse
Output throughput · includes prefill · 1 request
31.5tok/s
Best measured · 2 concurrent requests
41.3tok/s
Observed GPU occupancy
87.4GiB
First token
24.9 s
Use this recipe · 2 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 concurrency levels
Total output tokens divided by the full request window, including prefill and queueing.
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
vllm · vLLM 0.28.0
Protocol
wbench-search-http-v1
Dataset
migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload
131072in-2048out
Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
C1 31.54 64.93 s 24.9 s 87.37 402.7 1 73fcb7c1 ↗
C2 41.27 111.14 s 34 s 87.37 426.3 1 73fcb7c1 ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	31.54	64.93 s	24.9 s	87.37	402.7	1	73fcb7c1 ↗
C2	41.27	111.14 s	34 s	87.37	426.3	1	73fcb7c1 ↗

10 other test configurations on this GPU

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54.7tok/s
Best measured · 8 concurrent requests: 340tok/s
Observed GPU occupancy: 87.4GiB
First token: 329 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.72	37.56 s	329 ms	87.37	322.4	1	d3f6b770 ↗
C2	99.95	40.99 s	522 ms	87.37	315.9	1	d3f6b770 ↗
C4	189.07	43.34 s	971 ms	87.37	337.8	1	d3f6b770 ↗
C6	264.7	46.44 s	1.3 s	87.37	364	1	d3f6b770 ↗
C7	303.08	47.32 s	1.4 s	87.37	373.4	1	d3f6b770 ↗
C8	339.61	48.27 s	1.7 s	87.37	381.2	1	d3f6b770 ↗

NVFP4 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 44.9tok/s
Best measured · 4 concurrent requests: 96.8tok/s
Observed GPU occupancy: 87.4GiB
First token: 8.6 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	44.86	45.67 s	8.6 s	87.37	360.5	1	d2ff32ec ↗
C2	67.59	63.21 s	12.7 s	87.37	379.1	1	d2ff32ec ↗
C3	84.95	84.07 s	14.6 s	87.37	398.5	1	d2ff32ec ↗
C4	96.77	105.62 s	16.3 s	87.37	415.8	1	d2ff32ec ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57tok/s
Best measured · 8 concurrent requests: 374tok/s
Observed GPU occupancy: 87.4GiB
First token: 222 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.96	36.17 s	222 ms	87.37	322.6	1	bced0642 ↗
C2	104.14	39.34 s	355 ms	87.37	305.9	1	bced0642 ↗
C4	198.42	41.3 s	704 ms	87.37	319.3	1	bced0642 ↗
C6	291.12	42.23 s	1 s	87.37	342.1	1	bced0642 ↗
C7	333.32	43.04 s	1.1 s	87.37	349.1	1	bced0642 ↗
C8	374.04	43.82 s	1.3 s	87.37	353.2	1	bced0642 ↗

NVFP4 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 52.8tok/s
Best measured · 8 concurrent requests: 215tok/s
Observed GPU occupancy: 87.4GiB
First token: 3.3 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.78	38.8 s	3.3 s	87.37	337.3	1	aee8b33e ↗
C2	87.2	48.71 s	4.9 s	87.37	343.4	1	aee8b33e ↗
C4	142.14	64.91 s	7 s	87.37	375.3	1	aee8b33e ↗
C6	187.69	79.85 s	8.1 s	87.37	402.8	1	aee8b33e ↗
C7	202.31	85.21 s	8.7 s	87.37	411.8	1	aee8b33e ↗
C8	215.05	94.52 s	9.3 s	87.37	421.4	1	aee8b33e ↗

NVFP4 · 128K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 31.5tok/s
Best measured · 8 concurrent requests: 57.7tok/s
Observed GPU occupancy: 86.6GiB
First token: 24.9 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=isolated:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	31.45	65.12 s	24.9 s	86.63	415.7	1	448b0777 ↗
C4	50.67	220.6 s	45.8 s	86.63	469.4	1	448b0777 ↗
C8	57.74	421.65 s	61.9 s	86.63	494.5	1	448b0777 ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54.5tok/s
Best measured · 8 concurrent requests: 339tok/s
Observed GPU occupancy: 86.6GiB
First token: 359 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.54	37.55 s	359 ms	86.63	324.9	1	7d1a7c8e ↗
C4	188.48	43.48 s	878 ms	86.63	350.2	1	7d1a7c8e ↗
C8	338.65	48.43 s	1.6 s	86.63	396.9	1	7d1a7c8e ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.1tok/s
Best measured · 8 concurrent requests: 373tok/s
Observed GPU occupancy: 86.6GiB
First token: 231 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.09	35.88 s	231 ms	86.63	319	1	37f29252 ↗
C4	197.8	41.44 s	650 ms	86.63	330.2	1	37f29252 ↗
C8	372.88	43.98 s	1.3 s	86.63	367.3	1	37f29252 ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 50tok/s
Best measured · 8 concurrent requests: 287tok/s
Observed GPU occupancy: 86.4GiB
First token: 506 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	49.98	40.98 s	506 ms	86.36	322.7	1	68d1f2c6 ↗
C4	171.33	47.85 s	1 s	86.36	365.5	1	68d1f2c6 ↗
C8	287.31	57.12 s	1.9 s	86.36	425.4	1	68d1f2c6 ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54tok/s
Best measured · 8 concurrent requests: 337tok/s
Observed GPU occupancy: 86.4GiB
First token: 361 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.04	38.07 s	361 ms	86.36	320.7	1	417f69be ↗
C4	187.12	43.81 s	879 ms	86.36	333.2	1	417f69be ↗
C8	336.95	48.66 s	1.6 s	86.36	379.7	1	417f69be ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 56.4tok/s
Best measured · 8 concurrent requests: 371tok/s
Observed GPU occupancy: 86.4GiB
First token: 244 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.4	36.51 s	244 ms	86.35	319.1	1	791b887d ↗
C4	196.45	41.73 s	654 ms	86.35	315.5	1	791b887d ↗
C8	370.68	44.24 s	1.3 s	86.36	348.1	1	791b887d ↗

Qwen3.8-27B-NVFP4

RadixArk · 12 tested configurations

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 53tok/s
Best measured · 4 concurrent requests: 181tok/s
Observed GPU occupancy: 86GiB
First token: 435 ms

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.97	38.72 s	435 ms	85.99	333.9	1	78fc908c ↗
C2	97.25	42.14 s	646 ms	85.99	340.8	1	78fc908c ↗
C3	138.66	44.36 s	822 ms	85.99	358.8	1	78fc908c ↗
C4	181.16	45.25 s	1.1 s	85.99	375.3	1	78fc908c ↗

11 other test configurations on this GPU

NVFP4 · 128K → 2K tokens · No cache reuse
Output throughput · includes prefill · 1 request
32.7tok/s
Best measured · 2 concurrent requests
42.5tok/s
Observed GPU occupancy
86GiB
First token
24.3 s
Use this recipe · 2 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 concurrency levels
Total output tokens divided by the full request window, including prefill and queueing.
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
vllm · vLLM 0.28.0
Protocol
wbench-search-http-v1
Dataset
migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload
131072in-2048out
Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
C1 32.72 62.59 s 24.3 s 85.99 406.5 1 28f51cf1 ↗
C2 42.54 106.24 s 33.5 s 85.99 428.9 1 28f51cf1 ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	32.72	62.59 s	24.3 s	85.99	406.5	1	28f51cf1 ↗
C2	42.54	106.24 s	33.5 s	85.99	428.9	1	28f51cf1 ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.5tok/s
Best measured · 8 concurrent requests: 355tok/s
Observed GPU occupancy: 86GiB
First token: 326 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.45	35.8 s	326 ms	85.98	326.1	1	67a5e6d3 ↗
C2	104.7	39.14 s	515 ms	85.98	321	1	67a5e6d3 ↗
C4	198.72	41.24 s	954 ms	85.98	344.2	1	67a5e6d3 ↗
C6	277.65	44.28 s	1.3 s	85.98	358.8	1	67a5e6d3 ↗
C7	317.41	45.18 s	1.4 s	85.98	369.2	1	67a5e6d3 ↗
C8	354.95	46.19 s	1.6 s	85.98	377.1	1	67a5e6d3 ↗

NVFP4 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 47.1tok/s
Best measured · 4 concurrent requests: 101tok/s
Observed GPU occupancy: 86GiB
First token: 8.4 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	47.15	43.45 s	8.4 s	85.98	364.4	1	e5a201fa ↗
C2	70.39	60.71 s	12.3 s	85.98	381.3	1	e5a201fa ↗
C3	88.36	80.85 s	14.2 s	85.98	403.6	1	e5a201fa ↗
C4	100.66	101.65 s	15.8 s	85.98	419.2	1	e5a201fa ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 60.1tok/s
Best measured · 8 concurrent requests: 393tok/s
Observed GPU occupancy: 86GiB
First token: 217 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	60.1	34.34 s	217 ms	85.98	323	1	9cdf45b7 ↗
C2	109.3	37.48 s	349 ms	85.98	310.8	1	9cdf45b7 ↗
C4	209.23	39.18 s	686 ms	85.98	324.8	1	9cdf45b7 ↗
C6	306.94	40.06 s	992 ms	85.98	336	1	9cdf45b7 ↗
C7	351.24	40.83 s	1.1 s	85.98	342.6	1	9cdf45b7 ↗
C8	392.93	41.71 s	1.3 s	85.98	348.2	1	9cdf45b7 ↗

NVFP4 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 55.6tok/s
Best measured · 8 concurrent requests: 225tok/s
Observed GPU occupancy: 86GiB
First token: 3.2 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	55.59	36.91 s	3.2 s	85.98	341	1	dca51ce5 ↗
C2	91.37	46.48 s	4.6 s	85.98	342.9	1	dca51ce5 ↗
C4	149.21	62.63 s	6.5 s	85.98	372.9	1	dca51ce5 ↗
C6	196.73	76.24 s	7.8 s	85.98	398.9	1	dca51ce5 ↗
C7	211.87	81.41 s	8.3 s	85.98	408.7	1	dca51ce5 ↗
C8	224.61	90.5 s	8.9 s	85.98	414.8	1	dca51ce5 ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 52.9tok/s
Best measured · 8 concurrent requests: 301tok/s
Observed GPU occupancy: 86.9GiB
First token: 508 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.9	38.72 s	508 ms	86.86	338	1	7cea351c ↗
C4	179.81	45.65 s	1 s	86.86	387.5	1	7cea351c ↗
C8	301.16	54.48 s	1.9 s	86.86	443.7	1	7cea351c ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.7tok/s
Best measured · 8 concurrent requests: 356tok/s
Observed GPU occupancy: 86.9GiB
First token: 357 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.67	35.51 s	357 ms	86.86	325.8	1	fe8141d2 ↗
C4	197.42	41.52 s	870 ms	86.86	354.3	1	fe8141d2 ↗
C8	356.25	46.05 s	1.6 s	86.86	392.4	1	fe8141d2 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 60.5tok/s
Best measured · 8 concurrent requests: 395tok/s
Observed GPU occupancy: 86.9GiB
First token: 228 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	60.47	33.87 s	228 ms	86.86	320.3	1	cbf1431c ↗
C4	207.86	39.44 s	636 ms	86.86	333.6	1	cbf1431c ↗
C8	394.52	41.6 s	1.3 s	86.86	361.6	1	cbf1431c ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 52.6tok/s
Best measured · 8 concurrent requests: 301tok/s
Observed GPU occupancy: 86.8GiB
First token: 509 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.59	38.95 s	509 ms	86.83	328.8	1	d14962c1 ↗
C4	179.42	45.72 s	1.1 s	86.83	371.9	1	d14962c1 ↗
C8	300.98	54.52 s	1.9 s	86.83	426	1	d14962c1 ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.1tok/s
Best measured · 8 concurrent requests: 356tok/s
Observed GPU occupancy: 86.8GiB
First token: 358 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.08	36.15 s	358 ms	86.82	326	1	275a096d ↗
C4	196.91	41.63 s	870 ms	86.83	338.7	1	275a096d ↗
C8	355.89	46.08 s	1.6 s	86.83	375.7	1	275a096d ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 59.7tok/s
Best measured · 8 concurrent requests: 394tok/s
Observed GPU occupancy: 86.8GiB
First token: 230 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	59.68	34.59 s	230 ms	86.82	323.3	1	83d37469 ↗
C4	207.31	39.54 s	664 ms	86.82	320	1	83d37469 ↗
C8	394.04	41.6 s	1.2 s	86.82	346.1	1	83d37469 ↗

StartLux-Decision-v1-0.8B
startlux-models · 2 tested configurations
- bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  50.3decisions/sec
  Observed GPU occupancy
  2.3GiB
  P95 latency
  20 ms
  Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  startlux-native · PyTorch 2.11.0+cu130
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 50.28 20 ms — 2.29 165 1 8c1985ea ↗
- 1 other test configurations on this GPU
  - bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
    Throughput · Batch 1
    50.3decisions/sec
    Observed GPU occupancy
    3.2GiB
    P95 latency
    20 ms
    Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
    Test details
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    startlux-native · PyTorch 2.11.0+cu130
    Protocol
    native-forward-v1
    Dataset
    typed-invoices-v1
    Workload
    decision
    Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 50.25 20 ms — 3.17 161.3 1 df141700 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.28	20 ms	—	2.29	165	1	8c1985ea ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.25	20 ms	—	3.17	161.3	1	df141700 ↗

Intern-Decision-0.8B

internlm · 4 tested configurations

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
Throughput · Batch 1
50.8decisions/sec
Best measured · Batch 17
347decisions/sec
Observed GPU occupancy
2.4GiB
P95 latency
20 ms
Use this recipe · Batch 17 ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 batch sizes
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
intern-decision-native · PyTorch 2.11.0+cu130
Protocol
native-forward-v1
Dataset
typed-invoices-v1
Workload
decision
Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B1 50.83 20 ms — 2.36 169 2 18b06f63 ↗
B17 346.72 50 ms — 3.07 378 1 6ccb367c ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.83	20 ms	—	2.36	169	2	18b06f63 ↗
B17	346.72	50 ms	—	3.07	378	1	6ccb367c ↗

3 other test configurations on this GPU

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 396 s

Throughput · Batch 1: 52.8decisions/sec
Best measured · Batch 17: 350decisions/sec
Observed GPU occupancy: 2.4GiB
P95 latency: 19 ms

Use this recipe · Batch 17 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 21 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: intern-decision-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	52.8	19 ms	—	2.36	172.8	1	5eef633e ↗
B2	95.72	21 ms	—	2.38	186.5	1	5eef633e ↗
B4	177.64	23 ms	—	2.43	242	1	5eef633e ↗
B8	306.42	26 ms	—	2.57	344.2	1	5eef633e ↗
B12	322.68	37 ms	—	37.46	371.9	1	5eef633e ↗
B13	336	39 ms	—	37.46	380.8	1	5eef633e ↗
B14	344.61	41 ms	—	37.46	392.6	1	5eef633e ↗
B15	345.75	44 ms	—	37.46	389.5	1	5eef633e ↗
B16	345.28	47 ms	—	2.85	383.9	1	5eef633e ↗
B17	349.83	49 ms	—	37.46	397.8	1	5eef633e ↗
B18	327.34	55 ms	—	37.46	383.4	1	5eef633e ↗
B20	333.21	60 ms	—	37.46	402.6	1	5eef633e ↗
B24	331.98	73 ms	—	37.46	401.2	1	5eef633e ↗
B28	327.01	86 ms	—	37.46	399.2	1	5eef633e ↗
B32	331.67	97 ms	—	3.41	397.7	1	5eef633e ↗
B48	326.46	148 ms	—	37.46	405.9	1	5eef633e ↗
B64	318.22	202 ms	—	4.51	403.2	1	5eef633e ↗
B128	298.77	430 ms	—	6.71	405.8	1	5eef633e ↗
B256	298.58	859 ms	—	11.11	407.3	1	5eef633e ↗
B512	299.67	1.71 s	—	19.9	411.2	1	5eef633e ↗
B1024	297.46	3.64 s	—	37.46	410.6	1	5eef633e ↗

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 399 s

Throughput · Batch 1: 52decisions/sec
Best measured · Batch 16: 334decisions/sec
Observed GPU occupancy: 2.4GiB
P95 latency: 20 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 21 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: intern-decision-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	52.04	20 ms	—	2.36	162.6	1	156910e7 ↗
B2	96.21	21 ms	—	2.38	213.4	1	156910e7 ↗
B4	177.08	23 ms	—	2.43	243.4	1	156910e7 ↗
B8	299.06	27 ms	—	2.57	341.2	1	156910e7 ↗
B12	306.24	40 ms	—	36.98	349.6	1	156910e7 ↗
B14	322.36	43 ms	—	36.98	365.9	1	156910e7 ↗
B15	328.14	46 ms	—	36.98	369.8	1	156910e7 ↗
B16	333.92	48 ms	—	3.19	368.4	1	156910e7 ↗
B17	331.9	52 ms	—	36.98	379.5	1	156910e7 ↗
B18	310.84	58 ms	—	36.98	365.4	1	156910e7 ↗
B20	316.13	64 ms	—	36.98	375.1	1	156910e7 ↗
B24	312.87	77 ms	—	36.98	377.5	1	156910e7 ↗
B28	309.61	91 ms	—	36.98	379.4	1	156910e7 ↗
B32	319.44	101 ms	—	3.44	376.3	1	156910e7 ↗
B40	307.24	131 ms	—	36.98	379.6	1	156910e7 ↗
B48	306.75	159 ms	—	36.98	380.4	1	156910e7 ↗
B64	304.06	213 ms	—	4.2	383.7	1	156910e7 ↗
B128	285.17	453 ms	—	6.23	387.8	1	156910e7 ↗
B256	284.61	904 ms	—	10.63	390.3	1	156910e7 ↗
B512	282.22	2.04 s	—	19.42	390.1	1	156910e7 ↗
B1024	286.53	3.58 s	—	36.98	396.2	1	156910e7 ↗

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
Throughput · Batch 1
51.5decisions/sec
Observed GPU occupancy
2.6GiB
P95 latency
20 ms
Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
Test details
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
intern-decision-native · PyTorch 2.11.0
Protocol
native-forward-v1
Dataset
typed-invoices-v1
Workload
decision
Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B1 51.51 20 ms — 2.6 164.2 1 7f2e07c0 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	51.51	20 ms	—	2.6	164.2	1	7f2e07c0 ↗

clef-flash

Cloudflare · 2 tested configurations

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
Throughput · Batch 16
47.7decisions/sec
Observed GPU occupancy
19.5GiB
P95 latency
338 ms
Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs
Test details
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
clef-native · PyTorch 2.11.0+cu130
Protocol
native-forward-v1
Dataset
typed-invoices-v1
Workload
decision
Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B16 47.71 338 ms — 19.55 499.5 2 82d98b36 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B16	47.71	338 ms	—	19.55	499.5	2	82d98b36 ↗

1 other test configurations on this GPU

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 33 s

Throughput · Batch 1: 31.8decisions/sec
Best measured · Batch 4: 39.3decisions/sec
Observed GPU occupancy: 18.9GiB
P95 latency: 32 ms

Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: clef-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	31.83	32 ms	—	18.95	388.5	1	10c64aff ↗
B4	39.27	93 ms	—	18.95	449.5	1	10c64aff ↗
B16	5.41	8.21 s	—	19.28	233.4	1	10c64aff ↗
B64	15.43	9.66 s	—	21.89	324.3	1	10c64aff ↗

StartLux-Decision-v1.1-0.8B-Instant
startlux-models · 2 tested configurations
- bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  18.7decisions/sec
  Observed GPU occupancy
  2.7GiB
  P95 latency
  54 ms
  Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  startlux-transformers · PyTorch 2.11.0+cu130
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 18.69 54 ms — 2.73 134.5 2 a4a69815 ↗
- 1 other test configurations on this GPU
  - bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
    Throughput · Batch 1
    18.8decisions/sec
    Observed GPU occupancy
    2.5GiB
    P95 latency
    54 ms
    Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
    Test details
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    startlux-transformers · PyTorch 2.11.0
    Protocol
    native-forward-v1
    Dataset
    typed-invoices-v1
    Workload
    decision
    Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 18.83 54 ms — 2.54 125.5 1 c342a9ca ↗
Intern-Decision-2B
internlm · 1 tested configuration
- bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  50.3decisions/sec
  Observed GPU occupancy
  4.9GiB
  P95 latency
  21 ms
  Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  intern-decision-native · PyTorch 2.11.0
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 50.29 21 ms — 4.9 216.2 1 e4bcd6a3 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	18.69	54 ms	—	2.73	134.5	2	a4a69815 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	18.83	54 ms	—	2.54	125.5	1	c342a9ca ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.29	21 ms	—	4.9	216.2	1	e4bcd6a3 ↗

clef-omni

Cloudflare · 1 tested configuration

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 44 s

Throughput · Batch 1: 7.5decisions/sec
Best measured · Batch 64: 7.5decisions/sec
Observed GPU occupancy: 60.9GiB
P95 latency: 134 ms

Use this recipe · Batch 64 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: clef-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	7.45	134 ms	—	60.88	232.8	1	16597f91 ↗
B4	7.47	537 ms	—	60.89	249.7	1	16597f91 ↗
B16	7.48	2.14 s	—	60.89	252.8	1	16597f91 ↗
B64	7.49	8.56 s	—	60.89	257.2	1	16597f91 ↗

clef

Cloudflare · 1 tested configuration

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 46 s

Throughput · Batch 1: 12.3decisions/sec
Observed GPU occupancy: 52.4GiB
P95 latency: 82 ms

Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: clef-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	12.32	82 ms	—	52.4	469.1	1	243b7f64 ↗
B4	1.4	8.09 s	—	52.4	225.7	1	243b7f64 ↗
B16	4.33	9.1 s	—	52.98	308.5	1	243b7f64 ↗
B64	9.39	12.46 s	—	57.95	435.9	1	243b7f64 ↗

Qwen3-4B-AWQ
Qwen · 2 tested configurations
- AWQ · 8K → 2K tokens · Prefix cache reuse
  Output throughput · includes prefill · 1 request
  223tok/s
  Best measured · 4 concurrent requests
  742tok/s
  Observed GPU occupancy
  86.3GiB
  First token
  28 ms
  Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 concurrency levels
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  vllm · vLLM 0.29.0
  Protocol
  wbench-fixed-http-v3
  Dataset
  migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
  Workload
  8192in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 222.74 9.19 s 28 ms 86.28 288.8 1 adbfef8a ↗
  C4 741.85 11.05 s 35 ms 86.28 335 1 adbfef8a ↗
- 1 other test configurations on this GPU
  - AWQ · 8K → 2K tokens · No cache reuse
    Output throughput · includes prefill · 1 request
    215tok/s
    Best measured · 4 concurrent requests
    517tok/s
    Observed GPU occupancy
    86.3GiB
    First token
    298 ms
    Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
    Test details · 2 concurrency levels
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    vllm · vLLM 0.29.0
    Protocol
    wbench-fixed-http-v3
    Dataset
    migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 214.6 9.63 s 298 ms 86.28 303.8 1 86d36e4f ↗
    C4 516.94 16.14 s 872 ms 86.28 339.5 1 86d36e4f ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	222.74	9.19 s	28 ms	86.28	288.8	1	adbfef8a ↗
C4	741.85	11.05 s	35 ms	86.28	335	1	adbfef8a ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	214.6	9.63 s	298 ms	86.28	303.8	1	86d36e4f ↗
C4	516.94	16.14 s	872 ms	86.28	339.5	1	86d36e4f ↗

embeddinggemma-2

google · 2 tested configurations

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 207vectors/sec
Best measured · Batch 16: 2,939vectors/sec
Observed GPU occupancy: 1.5GiB
P95 latency: 5 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	206.79	5 ms	—	1.48	119.3	1	196678d3 ↗
B4	764.95	6 ms	—	1.48	141.4	1	196678d3 ↗
B16	2,938.63	6 ms	—	1.48	161.8	1	196678d3 ↗

1 other test configurations on this GPU

nf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 189vectors/sec
Best measured · Batch 16: 2,572vectors/sec
Observed GPU occupancy: 1.3GiB
P95 latency: 6 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	189.23	6 ms	—	1.33	145.5	1	c673556b ↗
B4	713.32	6 ms	—	1.33	142.8	1	c673556b ↗
B16	2,571.53	7 ms	—	1.33	162.7	1	c673556b ↗

IndexTTS-2.5
IndexTeam · 1 tested configuration
- Native · Not recorded → Audio · tts_prompts · Test duration 9 s
  Throughput · Batch 1
  4.2audio-sec/sec
  Observed GPU occupancy
  6.5GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  indextts · PyTorch 2.8.0+cu128
  Protocol
  audio-native-v1
  Dataset
  tts_prompts
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 4.24 — — 6.48 221.7 1 a46ec2c4 ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	4.24	—	—	6.48	221.7	1	a46ec2c4 ↗

granite-speech-5.0-470m-turboctc

ibm-granite · 2 tested configurations

Native · Audio → Text · asr_prompts · Test duration 16 s

Throughput · Batch 1: 2,662audio-sec/sec
Best measured · Batch 32: 9,194audio-sec/sec
Observed GPU occupancy: 1.6GiB
P95 latency: Not recorded

Evidence report and run availability ↗Same test on other GPUs

Test details · 11 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: transformers-ctc · PyTorch 2.13.0
Protocol: audio-native-v1
Dataset: asr_prompts
Workload: asr

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	2,662.33	—	—	1.64	122.1	1	3be7f67e ↗
B2	4,844.87	—	—	1.67	134.9	1	3be7f67e ↗
B4	6,237.58	—	—	1.75	148	1	3be7f67e ↗
B8	7,907.41	—	—	1.9	165.1	1	3be7f67e ↗
B16	9,083.29	—	—	2.33	194.8	1	3be7f67e ↗
B32	9,193.57	—	—	3.25	243.5	1	3be7f67e ↗
B64	8,622.98	—	—	5.09	345.5	1	3be7f67e ↗
B128	8,355.43	—	—	10.1	451	1	3be7f67e ↗
B256	8,343	—	—	20.09	477.6	1	3be7f67e ↗
B512	8,391.94	—	—	40.63	492.3	1	3be7f67e ↗
B1024	8,387.34	—	—	80.89	501.3	1	3be7f67e ↗

1 other test configurations on this GPU

Native · Audio → Text · asr_prompts

Throughput · Batch 1: 3,143audio-sec/sec
Best measured · Batch 16: 8,581audio-sec/sec
Observed GPU occupancy: 1.6GiB
P95 latency: Not recorded

Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: transformers-ctc · PyTorch 2.13.0
Protocol: audio-native-v1
Dataset: asr_prompts
Workload: asr

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	3,143.16	—	—	1.64	95.8	1	d77a0cfd ↗
B8	8,162.53	—	—	1.64	105.2	1	d77a0cfd ↗
B16	8,581.45	—	—	1.92	111.9	1	d77a0cfd ↗

Qwen3-TTS-12Hz-0.6B-CustomVoice
Qwen · 1 tested configuration
- Native · Not recorded → Audio · tts_prompts · Test duration 43 s
  Throughput · Batch 1
  1.6audio-sec/sec
  Best measured · Batch 4
  5.2audio-sec/sec
  Observed GPU occupancy
  3.3GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  qwen-tts · PyTorch 2.14.1
  Protocol
  audio-native-v1
  Dataset
  tts_prompts
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 1.6 — — 3.34 154.4 1 ebb272f8 ↗
  B4 5.22 — — 4.92 161 1 ebb272f8 ↗
magpie_tts_multilingual_357m
nvidia · 1 tested configuration
- Native · Not recorded → Audio · tts_prompts · Test duration 19 s
  Throughput · Batch 1
  3.6audio-sec/sec
  Observed GPU occupancy
  4GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  nemo-magpie · PyTorch 2.13.0
  Protocol
  audio-native-v1
  Dataset
  tts_prompts
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 3.57 — — 4.01 178.8 1 666efb0f ↗
Qwen3.8-Flash-Next-NVFP4-NVIDIA
4 tested configurations
- NVFP4 · 32K → 2K tokensCPU + GPU hybrid
  Output throughput · includes prefill · 1 request
  124tok/s
  Observed GPU occupancy
  71.2GiB
  First token
  4.1 s
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  strata
  Protocol
  wbench-fixed-http-v1
  Dataset
  synthetic-tokenizer-v1:repeats=3
  Workload
  32768in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 124.44 18.26 s 4.1 s 71.22 319.5 1 4a41258c ↗
- 3 other test configurations on this GPU
  - NVFP4 · 8K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    151tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    1.1 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 151.39 15.46 s 1.1 s 71.22 292.2 1 60a45fd1 ↗
  - NVFP4 · 32K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    114tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    4.8 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    32768in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 114.25 20.98 s 4.8 s 71.22 316.1 1 4342000e ↗
  - NVFP4 · 8K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    135tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    1.3 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 134.63 19.87 s 1.3 s 71.22 280 1 f2b52c15 ↗
Qwen3.8-Flash-Next-UD-Q4_K_XL
2 tested configurations
- Q4_K_XL · 32K → 2K tokensCPU + GPU hybrid
  Output throughput · includes prefill · 1 request
  106tok/s
  Observed GPU occupancy
  79.7GiB
  First token
  6.7 s
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  strata
  Protocol
  wbench-fixed-http-v1
  Dataset
  synthetic-tokenizer-v1:repeats=3
  Workload
  32768in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 106.11 23.97 s 6.7 s 79.71 308 1 66369e97 ↗
- 1 other test configurations on this GPU
  - Q4_K_XL · 8K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    125tok/s
    Observed GPU occupancy
    79.7GiB
    First token
    1.8 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 125.06 23.75 s 1.8 s 79.71 279.5 1 18c8631f ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	1.6	—	—	3.34	154.4	1	ebb272f8 ↗
B4	5.22	—	—	4.92	161	1	ebb272f8 ↗

Qwen3-Embedding-0.6B

Qwen · 3 tested configurations

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 232vectors/sec
Best measured · Batch 32: 3,763vectors/sec
Observed GPU occupancy: 2.2GiB
P95 latency: 5 ms

Use this recipe · Batch 32 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	232.18	5 ms	—	2.22	155	1	65594583 ↗
B8	1,601.21	5 ms	—	2.22	191.2	1	65594583 ↗
B32	3,763.07	9 ms	—	2.22	208.7	1	65594583 ↗

2 other test configurations on this GPU

nf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s

Throughput · Batch 1: 196vectors/sec
Best measured · Batch 32: 3,091vectors/sec
Observed GPU occupancy: 1.6GiB
P95 latency: 5 ms

Use this recipe · Batch 32 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	195.8	5 ms	—	1.61	174.8	1	b9501833 ↗
B8	1,156.9	7 ms	—	1.61	196	1	b9501833 ↗
B32	3,091.08	10 ms	—	1.61	207.4	1	b9501833 ↗

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s

Throughput · Batch 1: 218vectors/sec
Best measured · Batch 32: 3,617vectors/sec
Observed GPU occupancy: 2.2GiB
P95 latency: 5 ms

Use this recipe · Batch 32 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	217.92	5 ms	—	2.22	156.5	1	a7d85bcf ↗
B8	1,494.89	5 ms	—	2.22	185.4	1	a7d85bcf ↗
B32	3,616.74	9 ms	—	2.22	206.8	1	a7d85bcf ↗

Kokoro-82M
hexgrad · 1 tested configuration
- fp32 · Text → Audio · kokoro-english-v1 · Test duration 6 s
  Throughput · Batch 1
  120audio-sec/sec
  Best measured · Batch 4
  125audio-sec/sec
  Observed GPU occupancy
  2GiB
  P95 latency
  29 ms
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  kokoro · PyTorch 2.13.0+cu130
  Protocol
  audio-native-v1
  Dataset
  kokoro-english-v1
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 120.3 29 ms — 2 152.1 2 061d047a ↗
  B4 125.19 124 ms — 2.03 169.7 2 061d047a ↗
whisper-tiny
openai · 1 tested configuration
- fp16 · Audio → Text · asr_prompts · Test duration 2 s
  Throughput · Batch 1
  233audio-sec/sec
  Best measured · Batch 4
  364audio-sec/sec
  Observed GPU occupancy
  1.4GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  faster-whisper · PyTorch 2.14.1
  Protocol
  audio-native-v1
  Dataset
  asr_prompts
  Workload
  asr
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 232.81 — — 1.42 113.5 2 ba5a45ac ↗
  B4 364.19 — — 1.43 115.8 2 ba5a45ac ↗

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Sortable tables by exact test condition · 231 rows

C: concurrency · B: batch · —: not measured

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
startlux-models/StartLux-Decision-v1-0.8BScaling curves ↗	bf16	startlux-native	decisionnative-forward-v1	B1	50.28	50.28	—	0.02	2.29	165	50.28	0.3	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B1	50.83	50.86	—	0.02	2.36	169	50.8	0.3	2 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
startlux-models/StartLux-Decision-v1-0.8BScaling curves ↗	bf16	startlux-native	decisionnative-forward-v1	B1	50.25	50.25	—	0.02	3.17	161.3	50.25	0.31	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
startlux-models/StartLux-Decision-v1.1-0.8B-InstantScaling curves ↗	bf16	startlux-transformers	decisionnative-forward-v1	B1	18.69	18.77	—	0.054	2.73	134.5	18.61	0.14	2 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-2BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B1	50.29	50.29	—	0.021	4.9	216.2	50.29	0.23	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
startlux-models/StartLux-Decision-v1.1-0.8B-InstantScaling curves ↗	bf16	startlux-transformers	decisionnative-forward-v1	B1	18.83	18.83	—	0.054	2.54	125.5	18.83	0.15	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B1	51.51	51.51	—	0.02	2.6	164.2	51.51	0.31	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 16 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clef-flashScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B16	47.71	47.74	—	0.338	19.55	499.5	47.67	0.1	2 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 17 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B17	346.72	346.72	—	0.05	3.07	378	346.72	0.92	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clef-flashScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B1	31.83	31.83	—	0.032	18.95	388.5	31.83	0.08	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 4 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clef-flashScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B4	39.27	39.27	—	0.093	18.95	449.5	39.27	0.09	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 16 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clef-flashScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B16	5.41	5.41	—	8.208	19.28	233.4	5.41	0.02	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 64 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clef-flashScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B64	15.43	15.43	—	9.663	21.89	324.3	15.43	0.05	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clef-omniScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B1	7.45	7.45	—	0.134	60.88	232.8	7.45	0.03	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 4 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clef-omniScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B4	7.47	7.47	—	0.537	60.89	249.7	7.47	0.03	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 16 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clef-omniScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B16	7.48	7.48	—	2.139	60.89	252.8	7.48	0.03	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 64 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clef-omniScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B64	7.49	7.49	—	8.557	60.89	257.2	7.49	0.03	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clefScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B1	12.32	12.32	—	0.082	52.4	469.1	12.32	0.03	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 4 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clefScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B4	1.4	1.4	—	8.086	52.4	225.7	1.4	0.01	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 16 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clefScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B16	4.33	4.33	—	9.102	52.98	308.5	4.33	0.01	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 64 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Cloudflare/clefScaling curves ↗	bf16	clef-native	decisionnative-forward-v1	B64	9.39	9.39	—	12.463	57.95	435.9	9.39	0.02	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B1	52.8	52.8	—	0.019	2.36	172.8	52.8	0.31	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 2 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B2	95.72	95.72	—	0.021	2.38	186.5	95.72	0.51	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 4 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B4	177.64	177.64	—	0.023	2.43	242	177.64	0.73	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 8 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B8	306.42	306.42	—	0.026	2.57	344.2	306.42	0.89	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 12 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B12	322.68	322.68	—	0.037	37.46	371.9	322.68	0.87	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 13 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B13	336	336	—	0.039	37.46	380.8	336	0.88	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 14 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B14	344.61	344.61	—	0.041	37.46	392.6	344.61	0.88	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 15 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B15	345.75	345.75	—	0.044	37.46	389.5	345.75	0.89	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 16 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B16	345.28	345.28	—	0.047	2.85	383.9	345.28	0.9	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 17 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B17	349.83	349.83	—	0.049	37.46	397.8	349.83	0.88	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 18 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B18	327.34	327.34	—	0.055	37.46	383.4	327.34	0.85	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 20 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B20	333.21	333.21	—	0.06	37.46	402.6	333.21	0.83	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 24 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B24	331.98	331.98	—	0.073	37.46	401.2	331.98	0.83	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 28 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B28	327.01	327.01	—	0.086	37.46	399.2	327.01	0.82	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 32 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B32	331.67	331.67	—	0.097	3.41	397.7	331.67	0.83	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 48 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B48	326.46	326.46	—	0.148	37.46	405.9	326.46	0.8	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 64 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B64	318.22	318.22	—	0.202	4.51	403.2	318.22	0.79	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 128 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B128	298.77	298.77	—	0.43	6.71	405.8	298.77	0.74	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 256 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B256	298.58	298.58	—	0.859	11.11	407.3	298.58	0.73	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 512 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B512	299.67	299.67	—	1.713	19.9	411.2	299.67	0.73	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 1024 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B1024	297.46	297.46	—	3.64	37.46	410.6	297.46	0.72	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 1 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B1	52.04	52.04	—	0.02	2.36	162.6	52.04	0.32	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 2 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B2	96.21	96.21	—	0.021	2.38	213.4	96.21	0.45	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 4 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B4	177.08	177.08	—	0.023	2.43	243.4	177.08	0.73	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 8 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B8	299.06	299.06	—	0.027	2.57	341.2	299.06	0.88	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 12 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B12	306.24	306.24	—	0.04	36.98	349.6	306.24	0.88	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 14 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B14	322.36	322.36	—	0.043	36.98	365.9	322.36	0.88	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 15 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B15	328.14	328.14	—	0.046	36.98	369.8	328.14	0.89	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 16 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B16	333.92	333.92	—	0.048	3.19	368.4	333.92	0.91	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 17 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B17	331.9	331.9	—	0.052	36.98	379.5	331.9	0.87	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 18 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B18	310.84	310.84	—	0.058	36.98	365.4	310.84	0.85	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 20 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B20	316.13	316.13	—	0.064	36.98	375.1	316.13	0.84	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 24 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B24	312.87	312.87	—	0.077	36.98	377.5	312.87	0.83	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 28 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B28	309.61	309.61	—	0.091	36.98	379.4	309.61	0.82	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 32 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B32	319.44	319.44	—	0.101	3.44	376.3	319.44	0.85	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 40 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B40	307.24	307.24	—	0.131	36.98	379.6	307.24	0.81	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 48 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B48	306.75	306.75	—	0.159	36.98	380.4	306.75	0.81	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 64 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B64	304.06	304.06	—	0.213	4.2	383.7	304.06	0.79	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 128 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B128	285.17	285.17	—	0.453	6.23	387.8	285.17	0.74	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 256 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B256	284.61	284.61	—	0.904	10.63	390.3	284.61	0.73	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 512 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B512	282.22	282.22	—	2.039	19.42	390.1	282.22	0.72	1 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 1024 · decisions/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
internlm/Intern-Decision-0.8BScaling curves ↗	bf16	intern-decision-native	decisionnative-forward-v1	B1024	286.53	286.53	—	3.583	36.98	396.2	286.53	0.72	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 1 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-Embedding-0.6BScaling curves ↗	nf4	sentence-transformers	embeddingnative-forward-v1	B1	195.8	195.8	—	0.005	1.61	174.8	195.8	1.12	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 1 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-Embedding-0.6BScaling curves ↗	fp16	sentence-transformers	embeddingnative-forward-v1	B1	217.92	217.92	—	0.005	2.22	156.5	217.92	1.39	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 8 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-Embedding-0.6BScaling curves ↗	nf4	sentence-transformers	embeddingnative-forward-v1	B8	1,156.9	1,156.9	—	0.007	1.61	196	1,156.9	5.9	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 8 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-Embedding-0.6BScaling curves ↗	fp16	sentence-transformers	embeddingnative-forward-v1	B8	1,494.89	1,494.89	—	0.005	2.22	185.4	1,494.89	8.06	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 32 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-Embedding-0.6BScaling curves ↗	nf4	sentence-transformers	embeddingnative-forward-v1	B32	3,091.08	3,091.08	—	0.01	1.61	207.4	3,091.08	14.9	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 32 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-Embedding-0.6BScaling curves ↗	fp16	sentence-transformers	embeddingnative-forward-v1	B32	3,616.74	3,616.74	—	0.009	2.22	206.8	3,616.74	17.49	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
google/embeddinggemma-2Scaling curves ↗	fp16	sentence-transformers	embeddingnative-forward-v1	B1	206.79	206.79	—	0.005	1.48	119.3	206.79	1.73	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
google/embeddinggemma-2Scaling curves ↗	nf4	sentence-transformers	embeddingnative-forward-v1	B1	189.23	189.23	—	0.006	1.33	145.5	189.23	1.3	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-Embedding-0.6BScaling curves ↗	fp16	sentence-transformers	embeddingnative-forward-v1	B1	232.18	232.18	—	0.005	2.22	155	232.18	1.5	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
google/embeddinggemma-2Scaling curves ↗	fp16	sentence-transformers	embeddingnative-forward-v1	B4	764.95	764.95	—	0.006	1.48	141.4	764.95	5.41	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
google/embeddinggemma-2Scaling curves ↗	nf4	sentence-transformers	embeddingnative-forward-v1	B4	713.32	713.32	—	0.006	1.33	142.8	713.32	4.99	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 8 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-Embedding-0.6BScaling curves ↗	fp16	sentence-transformers	embeddingnative-forward-v1	B8	1,601.21	1,601.21	—	0.005	2.22	191.2	1,601.21	8.37	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
google/embeddinggemma-2Scaling curves ↗	fp16	sentence-transformers	embeddingnative-forward-v1	B16	2,938.63	2,938.63	—	0.006	1.48	161.8	2,938.63	18.16	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
google/embeddinggemma-2Scaling curves ↗	nf4	sentence-transformers	embeddingnative-forward-v1	B16	2,571.53	2,571.53	—	0.007	1.33	162.7	2,571.53	15.8	1 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 32 · vectors/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-Embedding-0.6BScaling curves ↗	fp16	sentence-transformers	embeddingnative-forward-v1	B32	3,763.07	3,763.07	—	0.009	2.22	208.7	3,763.07	18.03	1 ↗

Speech recognition · Audio → Text · asr_prompts · Batch 1 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B1	3,143.16	3,143.16	—	—	1.64	95.8	3,143.16	32.81	1 ↗

Speech recognition · Audio → Text · asr_prompts · Batch 8 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B8	8,162.53	8,162.53	—	—	1.64	105.2	8,162.53	77.59	1 ↗

Speech recognition · Audio → Text · asr_prompts · Batch 16 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B16	8,581.45	8,581.45	—	—	1.92	111.9	8,581.45	76.69	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 2 s · Batch 1 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
openai/whisper-tinyScaling curves ↗	fp16	faster-whisper	asraudio-native-v1	B1	232.81	234.75	—	—	1.42	113.5	230.87	2.05	2 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 2 s · Batch 4 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
openai/whisper-tinyScaling curves ↗	fp16	faster-whisper	asraudio-native-v1	B4	364.19	365.89	—	—	1.43	115.8	362.48	3.15	2 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 1 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B1	2,662.33	2,662.33	—	—	1.64	122.1	2,662.33	21.8	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 2 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B2	4,844.87	4,844.87	—	—	1.67	134.9	4,844.87	35.91	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 4 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B4	6,237.58	6,237.58	—	—	1.75	148	6,237.58	42.14	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 8 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B8	7,907.41	7,907.41	—	—	1.9	165.1	7,907.41	47.89	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 16 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B16	9,083.29	9,083.29	—	—	2.33	194.8	9,083.29	46.64	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 32 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B32	9,193.57	9,193.57	—	—	3.25	243.5	9,193.57	37.76	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 64 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B64	8,622.98	8,622.98	—	—	5.09	345.5	8,622.98	24.96	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 128 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B128	8,355.43	8,355.43	—	—	10.1	451	8,355.43	18.53	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 256 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B256	8,343	8,343	—	—	20.09	477.6	8,343	17.47	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 512 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B512	8,391.94	8,391.94	—	—	40.63	492.3	8,391.94	17.05	1 ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 1024 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
ibm-granite/granite-speech-5.0-470m-turboctcScaling curves ↗	Native	transformers-ctc	asraudio-native-v1	B1024	8,387.34	8,387.34	—	—	80.89	501.3	8,387.34	16.73	1 ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 9 s · Batch 1 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
IndexTeam/IndexTTS-2.5Scaling curves ↗	Native	indextts	ttsaudio-native-v1	B1	4.24	4.24	—	—	6.48	221.7	4.24	0.02	1 ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 19 s · Batch 1 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
nvidia/magpie_tts_multilingual_357mScaling curves ↗	Native	nemo-magpie	ttsaudio-native-v1	B1	3.57	3.57	—	—	4.01	178.8	3.57	0.02	1 ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 43 s · Batch 1 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoiceScaling curves ↗	Native	qwen-tts	ttsaudio-native-v1	B1	1.6	1.6	—	—	3.34	154.4	1.6	0.01	1 ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 43 s · Batch 4 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoiceScaling curves ↗	Native	qwen-tts	ttsaudio-native-v1	B4	5.22	5.22	—	—	4.92	161	5.22	0.03	1 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 1 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
hexgrad/Kokoro-82MScaling curves ↗	fp32	kokoro	ttsaudio-native-v1	B1	120.3	121.58	—	0.029	2	152.1	119.01	0.79	2 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 4 · audio-sec/sec · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
hexgrad/Kokoro-82MScaling curves ↗	fp32	kokoro	ttsaudio-native-v1	B4	125.19	129.48	—	0.124	2.03	169.7	120.9	0.74	2 ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen3.8-Flash-Next-NVFP4-NVIDIAScaling curves ↗	NVFP4	strata	8192in-2048out8,192 → 2,048 tokenswbench-fixed-http-v1	C1	151.39	151.39	1,140	15.457	71.22	292.2	151.39	0.52	1 ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen3.8-Flash-Next-NVFP4-NVIDIAScaling curves ↗	NVFP4	strata	8192in-2048out8,192 → 2,048 tokenswbench-fixed-http-v1	C1	134.63	134.63	1,305.67	19.867	71.22	280	134.63	0.48	1 ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen3.8-Flash-Next-UD-Q4_K_XLScaling curves ↗	Q4_K_XL	strata	8192in-2048out8,192 → 2,048 tokenswbench-fixed-http-v1	C1	125.06	125.06	1,819	23.745	79.71	279.5	125.06	0.45	1 ↗

Text generation · 8K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-4B-AWQScaling curves ↗	AWQ	vllm	8192in-2048out8,192 → 2,048 tokenswbench-fixed-http-v3	C1	214.6	214.6	298.33	9.627	86.28	303.8	214.6	0.71	1 ↗

Text generation · 8K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-4B-AWQScaling curves ↗	AWQ	vllm	8192in-2048out8,192 → 2,048 tokenswbench-fixed-http-v3	C4	516.94	516.94	872.17	16.142	86.28	339.5	516.94	1.52	1 ↗

Text generation · 8K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-4B-AWQScaling curves ↗	AWQ	vllm	8192in-2048out8,192 → 2,048 tokenswbench-fixed-http-v3	C1	222.74	222.74	27.67	9.194	86.28	288.8	222.74	0.77	1 ↗

Text generation · 8K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen/Qwen3-4B-AWQScaling curves ↗	AWQ	vllm	8192in-2048out8,192 → 2,048 tokenswbench-fixed-http-v3	C4	741.85	741.85	35	11.05	86.28	335	741.85	2.21	1 ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen3.8-Flash-Next-NVFP4-NVIDIAScaling curves ↗	NVFP4	strata	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v1	C1	124.44	124.44	4,136	18.261	71.22	319.5	124.44	0.39	1 ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen3.8-Flash-Next-NVFP4-NVIDIAScaling curves ↗	NVFP4	strata	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v1	C1	114.25	114.25	4,774.67	20.984	71.22	316.1	114.25	0.36	1 ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
Qwen3.8-Flash-Next-UD-Q4_K_XLScaling curves ↗	Q4_K_XL	strata	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v1	C1	106.11	106.11	6,671	23.973	79.71	308	106.11	0.34	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C1	153.44	153.44	753.33	13.392	85.25	343.4	153.44	0.45	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C1	52.78	52.78	3,332.67	38.801	87.37	337.3	52.78	0.16	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C1	55.59	55.59	3,188.33	36.913	85.98	341	55.59	0.16	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C2	208.65	208.65	1,143.5	19.644	85.25	360.4	208.65	0.58	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C2	87.2	87.2	4,857.17	48.708	87.37	343.4	87.2	0.25	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C2	91.37	91.37	4,639	46.482	85.98	342.9	91.37	0.27	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C4	259.19	259.19	1,572.17	33.205	85.25	398.9	259.19	0.65	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C4	142.14	142.14	6,978.17	64.914	87.37	375.3	142.14	0.38	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C4	149.21	149.21	6,485	62.629	85.98	372.9	149.21	0.4	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C6	187.69	187.69	8,132.33	79.851	87.37	402.8	187.69	0.47	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C6	196.73	196.73	7,789.22	76.243	85.98	398.9	196.73	0.49	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C7	202.31	202.31	8,682.81	85.213	87.37	411.8	202.31	0.49	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C7	211.87	211.87	8,317.9	81.411	85.98	408.7	211.87	0.52	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C8	295.37	295.37	2,138.5	59.5	85.25	422	295.37	0.7	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C8	215.05	215.05	9,335.96	94.521	87.37	421.4	215.05	0.51	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C8	224.61	224.61	8,934.33	90.496	85.98	414.8	224.61	0.54	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 12 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C12	311.33	311.33	2,680.67	86.142	85.25	432.4	311.33	0.72	1 ↗

Text generation · 32K → 2K tokens · No cache reuse · 16 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C16	319.17	319.17	3,296.31	112.396	85.25	438	319.17	0.73	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C1	161.57	161.57	71.33	12.719	85.25	327.8	161.57	0.49	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C1	56.96	56.96	222.33	36.172	87.37	322.6	56.96	0.18	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C1	60.1	60.1	217	34.343	85.98	323	60.1	0.19	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C1	57.09	57.09	230.67	35.875	86.63	319	57.09	0.18	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C1	60.47	60.47	228.33	33.87	86.86	320.3	60.47	0.19	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C1	56.4	56.4	244.33	36.506	86.35	319.1	56.4	0.18	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C1	59.68	59.68	230.33	34.585	86.82	323.3	59.68	0.18	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C2	269.43	269.43	86.33	15.207	85.25	349.1	269.43	0.77	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C2	104.14	104.14	355.17	39.341	87.37	305.9	104.14	0.34	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C2	109.3	109.3	348.67	37.484	85.98	310.8	109.3	0.35	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C4	460.41	460.41	106.17	17.828	85.25	427.3	460.41	1.08	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C4	198.42	198.42	703.92	41.299	87.37	319.3	198.42	0.62	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C4	209.23	209.23	686.08	39.178	85.98	324.8	209.23	0.64	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C4	197.8	197.8	650.08	41.439	86.63	330.2	197.8	0.6	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C4	207.86	207.86	636.25	39.444	86.86	333.6	207.86	0.62	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C4	196.45	196.45	653.75	41.732	86.35	315.5	196.45	0.62	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C4	207.31	207.31	663.92	39.543	86.82	320	207.31	0.65	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C6	291.12	291.12	1,001.28	42.226	87.37	342.1	291.12	0.85	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C6	306.94	306.94	991.61	40.059	85.98	336	306.94	0.91	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C7	333.32	333.32	1,130.05	43.039	87.37	349.1	333.32	0.95	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C7	351.24	351.24	1,112.43	40.832	85.98	342.6	351.24	1.03	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C8	644.45	644.45	127	25.517	85.25	486.9	644.45	1.32	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C8	374.04	374.04	1,268.29	43.823	87.37	353.2	374.04	1.06	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C8	392.93	392.93	1,250.17	41.709	85.98	348.2	392.93	1.13	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C8	372.88	372.88	1,281.71	43.976	86.63	367.3	372.88	1.02	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C8	394.52	394.52	1,251.75	41.597	86.86	361.6	394.52	1.09	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C8	370.68	370.68	1,281.96	44.239	86.36	348.1	370.68	1.06	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	32768in-2048out32,768 → 2,048 tokenswbench-fixed-http-v3	C8	394.04	394.04	1,243.54	41.604	86.82	346.1	394.04	1.14	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 16 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C16	771.04	771.04	184.02	42.765	85.25	533.9	771.04	1.44	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 17 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C17	812.79	812.79	195.27	43.138	85.25	550.4	812.79	1.48	1 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 18 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	32768in-2048out32,768 → 2,048 tokenswbench-search-http-v1	C18	856.64	856.64	195.8	43.409	85.25	568.9	856.64	1.51	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C1	101.26	101.26	2,339.33	20.264	85.25	369.1	101.26	0.27	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C1	44.86	44.86	8,631.33	45.668	87.37	360.5	44.86	0.12	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C1	47.15	47.15	8,364	43.446	85.98	364.4	47.15	0.13	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C2	123.11	123.11	3,556.67	33.412	85.25	404.8	123.11	0.3	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C2	67.59	67.59	12,736.67	63.213	87.37	379.1	67.59	0.18	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C2	70.39	70.39	12,314.33	60.714	85.98	381.3	70.39	0.18	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C3	84.95	84.95	14,624.56	84.074	87.37	398.5	84.95	0.21	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C3	88.36	88.36	14,170.56	80.853	85.98	403.6	88.36	0.22	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C4	140.19	140.19	4,518.33	64.264	85.25	430	140.19	0.33	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C4	96.77	96.77	16,298.33	105.615	87.37	415.8	96.77	0.23	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C4	100.66	100.66	15,765.08	101.653	85.98	419.2	100.66	0.24	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C6	142.08	142.08	5,332.72	97.785	85.25	430.4	142.08	0.33	1 ↗

Text generation · 64K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C8	150.73	150.73	6,287.21	122.368	85.25	446.7	150.73	0.34	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C1	113.51	113.51	134.67	18.063	85.25	341.3	113.51	0.33	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C1	54.72	54.72	329	37.559	87.37	322.4	54.72	0.17	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C1	57.45	57.45	325.67	35.8	85.98	326.1	57.45	0.18	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C1	54.54	54.54	359	37.553	86.63	324.9	54.54	0.17	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C1	57.67	57.67	356.67	35.509	86.86	325.8	57.67	0.18	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C1	54.04	54.04	361.33	38.065	86.36	320.7	54.04	0.17	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C1	57.08	57.08	357.67	36.147	86.82	326	57.08	0.18	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C2	153.3	153.3	160.67	26.728	85.25	369.5	153.3	0.41	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C2	99.95	99.95	522.17	40.991	87.37	315.9	99.95	0.32	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C2	104.7	104.7	515.33	39.136	85.98	321	104.7	0.33	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C4	281.86	281.86	201.83	29.1	85.25	483.8	281.86	0.58	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C4	189.07	189.07	971	43.343	87.37	337.8	189.07	0.56	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C4	198.72	198.72	953.5	41.236	85.98	344.2	198.72	0.58	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C4	188.48	188.48	878.08	43.484	86.63	350.2	188.48	0.54	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C4	197.42	197.42	870.08	41.52	86.86	354.3	197.42	0.56	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C4	187.12	187.12	878.92	43.809	86.36	333.2	187.12	0.56	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C4	196.91	196.91	869.5	41.63	86.83	338.7	196.91	0.58	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C6	293.74	293.74	213.72	41.934	85.25	459.2	293.74	0.64	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C6	264.7	264.7	1,279.5	46.444	87.37	364	264.7	0.73	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C6	277.65	277.65	1,254.28	44.283	85.98	358.8	277.65	0.77	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C7	340.1	340.1	227.1	42.269	85.25	496.1	340.1	0.69	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C7	303.08	303.08	1,447.95	47.318	87.37	373.4	303.08	0.81	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C7	317.41	317.41	1,446.57	45.184	85.98	369.2	317.41	0.86	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C8	387.93	387.93	236.71	42.41	85.25	534.9	387.93	0.73	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C8	339.61	339.61	1,674.71	48.265	87.37	381.2	339.61	0.89	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C8	354.95	354.95	1,632.08	46.193	85.98	377.1	354.95	0.94	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C8	338.65	338.65	1,606.17	48.432	86.63	396.9	338.65	0.85	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C8	356.25	356.25	1,596.79	46.052	86.86	392.4	356.25	0.91	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C8	336.95	336.95	1,612.13	48.661	86.36	379.7	336.95	0.89	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	65536in-2048out65,536 → 2,048 tokenswbench-fixed-http-v3	C8	355.89	355.89	1,592	46.079	86.83	375.7	355.89	0.95	1 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 9 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	65536in-2048out65,536 → 2,048 tokenswbench-search-http-v1	C9	433.07	433.07	246.07	42.778	85.25	575.8	433.07	0.75	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C1	56.09	56.09	8,031	36.628	85.25	418.9	56.09	0.13	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C1	31.54	31.54	24,870	64.932	87.37	402.7	31.54	0.08	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C1	32.72	32.72	24,314.67	62.594	85.99	406.5	32.72	0.08	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C1	31.45	31.45	24,908.33	65.121	86.63	415.7	31.45	0.08	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C2	62.52	62.52	12,019.67	67.494	85.25	444.3	62.52	0.14	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C2	41.27	41.27	33,977.33	111.136	87.37	426.3	41.27	0.1	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C2	42.54	42.54	33,482.83	106.239	85.99	428.9	42.54	0.1	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C3	63.59	63.59	13,461.56	109.305	85.25	446.8	63.59	0.14	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C4	66.75	66.75	15,049.25	144.628	85.25	461.6	66.75	0.14	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C4	50.67	50.67	45,800.67	220.602	86.63	469.4	50.67	0.11	1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C8	57.74	57.74	61,931.75	421.654	86.63	494.5	57.74	0.12	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C1	71.05	71.05	267.67	28.828	85.25	356.6	71.05	0.2	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C1	52.97	52.97	434.67	38.717	85.99	333.9	52.97	0.16	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C1	52.9	52.9	508.33	38.72	86.86	338	52.9	0.16	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C1	49.98	49.98	506.33	40.979	86.36	322.7	49.98	0.15	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C1	52.59	52.59	509.33	38.947	86.83	328.8	52.59	0.16	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C2	89.43	89.43	322.5	45.82	85.25	399.6	89.43	0.22	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C2	97.25	97.25	646	42.135	85.99	340.8	97.25	0.29	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C3	135.43	135.43	378.89	45.412	85.25	470.9	135.43	0.29	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C3	138.66	138.66	822.11	44.36	85.99	358.8	138.66	0.39	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
tencent/WeVisOne-VL-2BScaling curves ↗	BF16	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C4	165.29	165.29	420.33	49.613	85.25	524	165.29	0.32	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-search-http-v1	C4	181.16	181.16	1,056.42	45.249	85.99	375.3	181.16	0.48	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C4	179.81	179.81	1,037	45.651	86.86	387.5	179.81	0.46	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C4	171.33	171.33	1,032.67	47.851	86.36	365.5	171.33	0.47	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C4	179.42	179.42	1,069	45.721	86.83	371.9	179.42	0.48	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C8	301.16	301.16	1,852.29	54.479	86.86	443.7	301.16	0.68	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
unsloth/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C8	287.31	287.31	1,878.92	57.121	86.36	425.4	287.31	0.68	1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Format	Engine	Workload	Concurrency / batch	Mean throughput	Best measured	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	GPU power (W)	Lowest measured	Throughput / W	Reports combined
RadixArk/Qwen3.8-27B-NVFP4Scaling curves ↗	NVFP4	vllm	131072in-2048out131,072 → 2,048 tokenswbench-fixed-http-v3	C8	300.98	300.98	1,857.75	54.52	86.83	426	300.98	0.71	1 ↗

Scaling curves and individual runs

One model and test protocol per curve; different workloads are never joined into one line.

Benchmark configuration

tencent/WeVisOne-VL-2B · BF16 · vllm · 131072in-2048out · wbench-search-http-v1

Mean throughput tok/s

Measured points only. Select a point to open its source report.

Mean run P95 latency s

Measured points only. Select a point to open its source report.

Mean GPU memory used GB

Measured points only. Select a point to open its source report.

Each point is the mean of recorded runs at that level. A source report may differ from the mean; all runs are listed below.

Uncached prefill is unavailable when prompts reuse cache or reliable phase timings are missing. Open a report for its recorded phase timings and cache counts.

Runs behind the curve · tencent/WeVisOne-VL-2B · tok/s · 4 rows
Concurrency	Throughput	Decode (tok/s)	Uncached prefill (tok/s)	GPU power (W)	TTFT (ms)	P95 (s)	Observed occupancy (GiB)	Test date	View report
1	71.05 tok/s	71.63	Uncached prefill unavailableView report	356.6	267.67	28.828	85.25	10/26/2026, 1:05:52 AM	1ff0c98a ↗
2	89.43 tok/s	44.99	Uncached prefill unavailableView report	399.62	322.5	45.82	85.25	10/26/2026, 1:05:52 AM	1ff0c98a ↗
3	135.43 tok/s	45.49	Uncached prefill unavailableView report	470.93	378.89	45.412	85.25	10/26/2026, 1:05:52 AM	1ff0c98a ↗
4	165.29 tok/s	41.65	Uncached prefill unavailableView report	524.02	420.33	49.613	85.25	10/26/2026, 1:05:52 AM	1ff0c98a ↗

Quality by dataset

Scores from different datasets and sample scopes are not directly comparable.

Quality by dataset · 20 rows
Model	Dataset	Format	Engine	Average score	Sample size	Runs
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit	mmlu-test-zero-shot-strict-v4-n32-mt2048-t0	AWQ	vllm	78.13%	32 / 14042 · Subset only	1 runs 78.13% · f68a1a8d ↗
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit	gsm8k-test-zero-shot-strict-v4-n32-mt2048-t0	AWQ	vllm	78.13%	32 / 1319 · Subset only	1 runs 78.13% · e4d4c4ae ↗
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit	mmlu-test-zero-shot-strict-v4-n32-mt2048-t0	AWQ	vllm	81.25%	32 / 14042 · Subset only	1 runs 81.25% · e55835c0 ↗
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit	gsm8k-test-zero-shot-strict-v4-n32-mt2048-t0	AWQ	vllm	84.38%	32 / 1319 · Subset only	1 runs 84.38% · 0e9cce27 ↗
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit	mmlu-test-zero-shot-strict-v4-n32-mt2048-t0	AWQ	vllm	81.25%	32 / 14042 · Subset only	1 runs 81.25% · 23fe3c88 ↗
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit	gsm8k-test-zero-shot-strict-v4-n32-mt2048-t0	AWQ	vllm	84.38%	32 / 1319 · Subset only	1 runs 84.38% · d42e0304 ↗
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit	mmlu-test-zero-shot-strict-v4-n32-mt2048-t0	AWQ	vllm	81.25%	32 / 14042 · Subset only	1 runs 81.25% · 8010df52 ↗
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit	gsm8k-test-zero-shot-strict-v4-n32-mt2048-t0	AWQ	vllm	78.13%	32 / 1319 · Subset only	1 runs 78.13% · 2e5e23a9 ↗
hf:nvidia/Qwen3.8-27B-NVFP4	aime2026-test-zero-shot-strict-v2-n3-mt16384-t0	NVFP4	vllm	0%	3 / 30 · Subset only	1 runs 0% · c4b1903f ↗
hf:nvidia/Qwen3.8-27B-NVFP4	gsm8k-test-zero-shot-strict-v2-n3-mt16384-t0	NVFP4	vllm	0%	3 / 1319 · Subset only	1 runs 0% · 7d12a387 ↗
hf:nvidia/Qwen3.8-27B-NVFP4	mmlu-test-zero-shot-strict-v2-n3-mt16384-t0	NVFP4	vllm	0%	3 / 14042 · Subset only	1 runs 0% · d29d7993 ↗
Qwen3.8-Flash-Next-NVFP4-NVIDIA	aime2026-test-zero-shot-strict-v1-n3-mt2048-t0	NVFP4	strata	0%	3 / 30 · Subset only	1 runs 0% · 7b6f15b0 ↗
Qwen3.8-Flash-Next-NVFP4-NVIDIA	mmlu-test-zero-shot-strict-v1-n3-mt2048-t0	NVFP4	strata	100%	3 / 14042 · Subset only	1 runs 100% · c7ef5e0c ↗
Qwen3.8-Flash-Next-NVFP4-NVIDIA	gsm8k-test-zero-shot-strict-v1-n3-mt2048-t0	NVFP4	strata	100%	3 / 1319 · Subset only	1 runs 100% · 2ec541fd ↗
Qwen3.8-Flash-Next-NVFP4-NVIDIA	aime2026-test-zero-shot-strict-v1-n3-mt2048-t0	NVFP4	strata	33.33%	3 / 30 · Subset only	1 runs 33.33% · f7729544 ↗
Qwen3.8-Flash-Next-NVFP4-NVIDIA	mmlu-test-zero-shot-strict-v1-n3-mt2048-t0	NVFP4	strata	100%	3 / 14042 · Subset only	1 runs 100% · 17483940 ↗
Qwen3.8-Flash-Next-NVFP4-NVIDIA	gsm8k-test-zero-shot-strict-v1-n3-mt2048-t0	NVFP4	strata	100%	3 / 1319 · Subset only	1 runs 100% · ed5862f7 ↗
Qwen3.8-Flash-Next-UD-Q4_K_XL	aime2026-test-zero-shot-strict-v1-n3-mt2048-t0	Q4_K_XL	strata	0%	3 / 30 · Subset only	1 runs 0% · d5277889 ↗
Qwen3.8-Flash-Next-UD-Q4_K_XL	mmlu-test-zero-shot-strict-v1-n3-mt2048-t0	Q4_K_XL	strata	100%	3 / 14042 · Subset only	1 runs 100% · b72eb056 ↗
Qwen3.8-Flash-Next-UD-Q4_K_XL	gsm8k-test-zero-shot-strict-v1-n3-mt2048-t0	Q4_K_XL	strata	100%	3 / 1319 · Subset only	1 runs 100% · b1cfdc40 ↗

Measurements without a comparable protocol · 153 rows

These measured rates remain visible, but are excluded from comparison curves because the protocol is unknown.

Measurements without a comparable protocol · 153 rows
Model	Format	Engine	Workload	Concurrency	Throughput	GPU power (W)
Qwen3-0.6B ↗	BF16	vLLM	2K	1	19.73 tok/s	83.37
Qwen2.5-0.5B ↗	Q4_K_M	Ollama	1K	1	406.75 tok/s	167.76
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	1	261.66 tok/s	180.93
Qwen2.5-0.5B ↗	Q4_K_M	Ollama	1K	8	448.48 tok/s	197.41
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	1	121.24 tok/s	201.34
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	1	125.01 tok/s	203.37
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	32	4,712.67 tok/s	206.69
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	1	179.06 tok/s	208.4
Qwen2.5-0.5B ↗	Q4_K_M	Ollama	1K	2	570.1 tok/s	210.36
Qwen3-VL-2B-FP8 ↗	FP8	vLLM	2K	8	1,124.53 tok/s	212.82
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	8	1,806.06 tok/s	213.17
Qwen2.5-1.5B ↗	Q4_K_M	Ollama	1K	1	314.88 tok/s	214.05
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	8	1,245.98 tok/s	214.11
Qwen2.5-0.5B ↗	Q4_K_M	Ollama	1K	32	581.39 tok/s	215.74
Qwen2.5-0.5B ↗	Q4_K_M	Ollama	1K	4	578.85 tok/s	215.95
Qwen2.5-0.5B ↗	Q4_K_M	Ollama	1K	16	580.86 tok/s	216.54
Qwen3-VL-2B-FP8 ↗	FP8	vLLM	2K	1	171.92 tok/s	218.33
Qwen3-VL-2B-FP8 ↗	FP8	vLLM	2K	32	3,183.95 tok/s	219.06
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	16	3,241.44 tok/s	220.84
Qwen2.5-1.5B ↗	Q4_K_M	Ollama	1K	32	440.13 tok/s	222.51
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	64	7,537.31 tok/s	224.29
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	32	3,542.33 tok/s	227.19
Qwen3-VL-2B-FP8 ↗	FP8	vLLM	2K	16	2,038.55 tok/s	227.6
Qwen2.5-1.5B ↗	Q4_K_M	Ollama	1K	16	411.66 tok/s	228.41
Qwen3-VL-4B-FP8 ↗	FP8	vLLM	2K	8	703.94 tok/s	229.64
Qwen3-VL-4B-FP8 ↗	FP8	vLLM	2K	1	102.15 tok/s	229.98
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	1	105.66 tok/s	232.53
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	16	2,304.28 tok/s	233.01
Qwen2.5-1.5B ↗	Q4_K_M	Ollama	1K	8	433.88 tok/s	234.26
Qwen3-0.6B ↗	BF16	vLLM	2K	8	1,986.29 tok/s	235.61
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	8	643.63 tok/s	235.67
Qwen3-VL-4B-FP8 ↗	FP8	vLLM	2K	32	2,001.12 tok/s	238.55
Qwen3-VL-2B-FP8 ↗	FP8	vLLM	2K	64	5,040.91 tok/s	238.57
Qwen2.5-1.5B ↗	Q4_K_M	Ollama	1K	4	434.05 tok/s	239.26
Qwen2.5-1.5B ↗	Q4_K_M	Ollama	1K	2	402.65 tok/s	241.67
Qwen3-VL-4B-FP8 ↗	FP8	vLLM	2K	16	1,204.74 tok/s	244.82
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	64	5,147.91 tok/s	245.44
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	32	2,276.18 tok/s	247.29
Qwen2.5-3B ↗	Q4_K_M	Ollama	1K	1	251.72 tok/s	248.53
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	16	1,310.26 tok/s	253.02
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	128	10,619.38 tok/s	253.25
Qwen3-VL-4B-FP8 ↗	FP8	vLLM	2K	64	3,432.74 tok/s	255.4
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	8	501.51 tok/s	256.94
Qwen3-0.6B ↗	BF16	vLLM	2K	32	4,524.91 tok/s	257.8
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	8	502.95 tok/s	261.6
Qwen2.5-3B ↗	Q4_K_M	Ollama	1K	2	306.47 tok/s	262.35
Qwen2.5-3B ↗	Q4_K_M	Ollama	1K	4	310.41 tok/s	262.62
Qwen2.5-3B ↗	Q4_K_M	Ollama	1K	16	308.6 tok/s	262.89
Qwen2.5-3B ↗	Q4_K_M	Ollama	1K	32	310.61 tok/s	263.09
Qwen2.5-3B ↗	Q4_K_M	Ollama	1K	8	308.57 tok/s	263.5
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	64	3,254.85 tok/s	264.59
Qwen3-0.6B ↗	BF16	vLLM	2K	16	3,212.79 tok/s	264.97
Qwen3-VL-2B-FP8 ↗	FP8	vLLM	2K	128	6,806.73 tok/s	268.61
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	128	7,544.12 tok/s	275.6
Qwen3-1.7B ↗	BF16	vLLM	2K	8	1,349.38 tok/s	281.45
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	16	797.5 tok/s	284.65
Qwen3-1.7B ↗	BF16	vLLM	2K	32	3,571.94 tok/s	284.81
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	16	791.06 tok/s	285.02
Qwen3-0.6B ↗	BF16	vLLM	2K	64	5,971.43 tok/s	288.07
Qwen3-VL-4B-FP8 ↗	FP8	vLLM	2K	128	5,021.53 tok/s	289.06
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	128	5,611.52 tok/s	289.17
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	256	13,064.79 tok/s	290.66
Qwen3-1.7B ↗	BF16	vLLM	2K	1	225.09 tok/s	292.97
Qwen3-0.6B ↗	BF16	vLLM	2K	128	8,165.65 tok/s	294.81
Qwen3-1.7B ↗	BF16	vLLM	2K	16	2,398.56 tok/s	296.63
Qwen3-1.7B ↗	BF16	vLLM	2K	64	5,485.36 tok/s	297.75
Qwen3-4B ↗	BF16	vLLM	2K	16	1,281.01 tok/s	299.17
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	64	1,921.03 tok/s	301.01
Qwen3-4B ↗	BF16	vLLM	2K	32	2,482.46 tok/s	301.8
Qwen3-4B ↗	BF16	vLLM	2K	8	799.81 tok/s	304.1
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	64	1,897.86 tok/s	304.25
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	32	1,195.07 tok/s	308.88
Qwen3-VL-2B-FP8 ↗	FP8	vLLM	2K	256	8,328.63 tok/s	310.02
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	32	1,214.35 tok/s	310.36
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	256	8,950.66 tok/s	314.39
Qwen3-0.6B ↗	BF16	vLLM	2K	256	9,394.89 tok/s	314.8
Qwen3-0.6B ↗	BF16	vLLM	2K	512	10,133.18 tok/s	318
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	128	2,984.25 tok/s	318.33
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	128	2,929.04 tok/s	322.27
Qwen3-0.6B-FP8 ↗	FP8	vLLM	2K	512	14,809.02 tok/s	326.59
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	8	164.11 tok/s	327.86
Qwen3-VL-2B-FP8 ↗	FP8	vLLM	2K	512	9,245.88 tok/s	328.26
Qwen3-4B ↗	BF16	vLLM	2K	64	3,482.71 tok/s	329.03
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	8	150.89 tok/s	330.42
Qwen3-1.7B ↗	BF16	vLLM	2K	128	6,746.53 tok/s	330.45
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	8	140.26 tok/s	332.58
Qwen3-1.7B ↗	BF16	vLLM	2K	256	7,824.13 tok/s	333.74
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	8	178.45 tok/s	334.35
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	256	5,906.88 tok/s	335.17
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	1	24.58 tok/s	335.46
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	16	316.04 tok/s	336.36
Qwen3-1.7B-FP8 ↗	FP8	vLLM	2K	512	9,498.06 tok/s	336.52
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	1	24.56 tok/s	336.78
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	1	24.62 tok/s	336.97
Qwen2.5-7B ↗	Q4_K_M	Ollama	1K	1	203.51 tok/s	337.12
Qwen3-1.7B ↗	BF16	vLLM	2K	512	9,361.48 tok/s	337.73
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	1	24.61 tok/s	338.6
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	16	328.98 tok/s	338.71
Qwen3-VL-4B-FP8 ↗	FP8	vLLM	2K	256	6,345.02 tok/s	340.49
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	32	498.29 tok/s	341.89
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	16	273.53 tok/s	341.98
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	16	249.29 tok/s	342.02
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	32	526.67 tok/s	342.6
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	32	585.96 tok/s	343.88
Qwen2.5-7B ↗	Q4_K_M	Ollama	1K	2	215.15 tok/s	345.8
Qwen3-4B ↗	BF16	vLLM	2K	128	5,184.54 tok/s	346.23
Qwen2.5-7B ↗	Q4_K_M	Ollama	1K	4	212.45 tok/s	346.42
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	256	4,339.44 tok/s	346.93
Qwen2.5-7B ↗	Q4_K_M	Ollama	1K	32	214.86 tok/s	347.52
Qwen2.5-7B ↗	Q4_K_M	Ollama	1K	8	214.69 tok/s	347.88
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	32	483.54 tok/s	347.96
Qwen2.5-7B ↗	Q4_K_M	Ollama	1K	16	213.85 tok/s	348.22
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	256	4,341.5 tok/s	350.73
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	512	5,618.23 tok/s	350.9
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	64	1,016.4 tok/s	352.32
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	64	1,111.66 tok/s	353.51
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	512	5,718.49 tok/s	354.64
Qwen3-4B-FP8 ↗	FP8	vLLM	2K	512	7,272.85 tok/s	356.19
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	1	31.4 tok/s	356.5
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	64	885.38 tok/s	358.14
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	64	914.64 tok/s	358.75
Qwen3-4B ↗	BF16	vLLM	2K	1	121.72 tok/s	359.47
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	1	32.11 tok/s	363.29
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	8	231.23 tok/s	365.07
Qwen3-VL-4B-FP8 ↗	FP8	vLLM	2K	512	7,385.79 tok/s	366.12
Qwen3-VL-30B-A3B-FP8 ↗	FP8	vLLM	2K	1024	7,023.62 tok/s	366.43
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	8	236.11 tok/s	374.64
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	128	1,513.36 tok/s	382.52
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	128	1,533.95 tok/s	383.66
Qwen3-4B ↗	BF16	vLLM	2K	512	6,909.38 tok/s	385
Qwen3-4B ↗	BF16	vLLM	2K	256	5,835.3 tok/s	387.88
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	128	1,458.68 tok/s	395.79
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	128	1,809.55 tok/s	398.63
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	16	440.89 tok/s	416.21
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	16	256.92 tok/s	422.67
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	256	2,427.56 tok/s	466.41
Qwen3-32B-FP8 ↗	FP8	vLLM	2K	256	2,466.97 tok/s	467.58
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	32	763.09 tok/s	472.48
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	256	2,179.85 tok/s	475.22
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	256	1,808.09 tok/s	477.82
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	32	766.98 tok/s	479.71
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	512	2,351.76 tok/s	491.25
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	512	2,799.31 tok/s	492.49
Qwen3-VL-32B-FP8 ↗	FP8	vLLM	2K	1024	3,036.65 tok/s	511.42
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	64	690.88 tok/s	541.99
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	64	1,067.57 tok/s	543.98
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	128	1,247.66 tok/s	554.24
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	128	738.36 tok/s	561.85
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	512	967.3 tok/s	578.86
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	1024	1,538.3 tok/s	580.76
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	512	1,465 tok/s	581.24
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	256	1,395.03 tok/s	581.25
Qwen2.5-72B-AWQ ↗	AWQ	vLLM	2K	256	1,393.92 tok/s	582.18

Original reports

Every report for this task, including quality, diagnostic and legacy runs that do not enter comparisons.

Original reports · 196 rows
Model	Test task	Format	Engine	Workload	Protocol	Quality score	Test date
clef ↗	Decision	bf16	clef-native	decision	native-forward-v1	—	10/23/2026, 8:57:12 AM
clef-flash ↗	Decision	bf16	clef-native	decision	native-forward-v1	—	10/23/2026, 12:06:35 PM
clef-flash ↗	Decision	bf16	clef-native	decision	native-forward-v1	—	10/23/2026, 12:02:11 PM
clef-flash ↗	Decision	bf16	clef-native	decision	native-forward-v1	—	10/23/2026, 9:04:07 AM
clef-flash ↗	Decision	bf16	clef-native	decision	native-forward-v1	—	10/14/2026, 8:26:33 AM
clef-flash ↗	Decision	bf16	clef-native	decision	native-forward-v1	—	10/14/2026, 6:35:30 AM
clef-omni ↗	Decision	bf16	clef-native	decision	native-forward-v1	—	10/23/2026, 9:01:55 AM
DeepSeek-R1-Distill-Qwen-1.5B ↗	Text generation	BF16	vLLM	1K	wbench-http	—	10/14/2026, 8:52:17 AM
embeddinggemma-2 ↗	Embedding	fp16	sentence-transformers	embedding	native-forward-v1	—	10/19/2026, 10:00:34 PM
embeddinggemma-2 ↗	Embedding	nf4	sentence-transformers	embedding	native-forward-v1	—	10/19/2026, 9:52:34 PM
granite-speech-5.0-470m-turboctc ↗	Speech recognition	Native	transformers-ctc	asr	audio-native-v1	—	10/19/2026, 8:09:36 AM
granite-speech-5.0-470m-turboctc ↗	Speech recognition	Native	transformers-ctc	asr	audio-native-v1	—	10/19/2026, 6:57:47 AM
granite-speech-5.0-470m-turboctc ↗	Speech recognition	fp16	transformers-ctc	asr	audio-native-v1	—	10/19/2026, 6:48:01 AM
IndexTTS-2.5 ↗	Speech synthesis	Native	indextts	tts	audio-native-v1	—	10/19/2026, 8:09:44 AM
Intern-Decision-0.8B ↗	Decision	bf16	intern-decision-native	decision	native-forward-v1	—	10/23/2026, 12:07:54 PM
Intern-Decision-0.8B ↗	Decision	bf16	intern-decision-native	decision	native-forward-v1	—	10/23/2026, 12:06:48 PM
Intern-Decision-0.8B ↗	Decision	bf16	intern-decision-native	decision	native-forward-v1	—	10/23/2026, 12:00:13 PM
Intern-Decision-0.8B ↗	Decision	bf16	intern-decision-native	decision	native-forward-v1	—	10/23/2026, 10:11:05 AM
Intern-Decision-0.8B ↗	Decision	bf16	intern-decision-native	decision	native-forward-v1	—	10/23/2026, 10:02:56 AM
Intern-Decision-0.8B ↗	Decision	bf16	intern-decision-native	decision	native-forward-v1	—	10/23/2026, 9:23:01 AM
Intern-Decision-2B ↗	Decision	bf16	intern-decision-native	decision	native-forward-v1	—	10/23/2026, 9:24:25 AM
Kokoro-82M ↗	Speech synthesis	fp32	kokoro	tts	audio-native-v1	—	10/14/2026, 8:25:11 AM
Kokoro-82M ↗	Speech synthesis	fp32	kokoro	tts	audio-native-v1	—	10/14/2026, 6:23:39 AM
Llama-3.1-8B-Instruct-NVFP4 ↗	Text generation	NVFP4	vllm	8192in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:12:13 PM
Llama-3.1-8B-Instruct-NVFP4 ↗	Text generation	NVFP4	vllm	2048in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:11:46 PM
Llama-3.1-8B-Instruct-NVFP4 ↗	Text generation	NVFP4	vllm	512in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:11:21 PM
magpie_tts_multilingual_357m ↗	Speech synthesis	Native	nemo-magpie	tts	audio-native-v1	—	10/19/2026, 6:57:57 AM
magpie_tts_multilingual_357m ↗	Speech synthesis	fp16	nemo-magpie	tts	audio-native-v1	—	10/19/2026, 6:48:05 AM
NVIDIA-Nemotron-Nano-9B-v2-NVFP4 ↗	Text generation	NVFP4	vllm	8192in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:16:58 PM
NVIDIA-Nemotron-Nano-9B-v2-NVFP4 ↗	Text generation	NVFP4	vllm	2048in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:16:28 PM
NVIDIA-Nemotron-Nano-9B-v2-NVFP4 ↗	Text generation	NVFP4	vllm	512in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:15:49 PM
NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD ↗	Text generation	NVFP4	vllm	2048in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:21:34 PM
NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD ↗	Text generation	NVFP4	vllm	512in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:20:57 PM
Qwen2.5-0.5B ↗	Text generation	Q4_K_M	Ollama	1K	Method unknown	—	10/14/2026, 8:27:58 AM
qwen2.5-0.5b-instruct-q4_k_m.gguf ↗	Text generation	Q4_K_M	llama.cpp	1K	wbench-http	—	10/14/2026, 9:06:17 AM
Qwen2.5-1.5B ↗	Text generation	Q4_K_M	Ollama	1K	Method unknown	—	10/14/2026, 8:32:19 AM
qwen2.5-1.5b-instruct-q4_k_m.gguf ↗	Text generation	Q4_K_M	llama.cpp	1K	wbench-http	—	10/14/2026, 9:08:55 AM
Qwen2.5-3B ↗	Text generation	Q4_K_M	Ollama	1K	Method unknown	—	10/14/2026, 8:37:06 AM
Qwen2.5-3B-Instruct ↗	Text generation	BF16	vllm	32K	vllm-official	—	10/26/2026, 6:36:29 AM
Qwen2.5-3B-Instruct ↗	Text generation	BF16	vllm	32K	vllm-official	—	10/26/2026, 6:32:34 AM
Qwen2.5-3B-Instruct ↗	Text generation	BF16	vllm	4K	vllm-official	—	10/26/2026, 6:31:24 AM
Qwen2.5-3B-Instruct ↗	Text generation	BF16	vllm	4K	vllm-official	—	10/26/2026, 6:30:25 AM
qwen2.5-3b-instruct-q4_k_m.gguf ↗	Text generation	Q4_K_M	llama.cpp	1K	wbench-http	—	10/14/2026, 9:11:33 AM
Qwen2.5-7B ↗	Text generation	Q4_K_M	Ollama	1K	Method unknown	—	10/14/2026, 8:42:41 AM
Qwen2.5-72B-AWQ ↗	Text generation	AWQ	vLLM	2K	Method unknown	—	1/13/2026, 1:08:31 AM
Qwen2.5-72B-AWQ ↗	Text generation	AWQ	vLLM	2K	Method unknown	—	1/12/2026, 9:06:29 PM
Qwen3-0.6B ↗	Text generation	BF16	vLLM	1K	wbench-http	—	10/14/2026, 8:46:02 AM
Qwen3-0.6B ↗	Text generation	BF16	vLLM	2K	Method unknown	—	1/11/2026, 3:59:24 PM
Qwen3-0.6B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/12/2026, 7:18:04 PM
Qwen3-0.6B-GPTQ ↗	Text generation	GPTQ	vLLM	1K	wbench-http	—	10/14/2026, 8:25:21 AM
Qwen3-0.6B-Q8 ↗	Text generation	Q8_0	llama.cpp	1K	wbench-http	—	10/14/2026, 9:14:07 AM
Qwen3-1.7B ↗	Text generation	BF16	vLLM	1K	wbench-http	—	10/14/2026, 8:58:30 AM
Qwen3-1.7B ↗	Text generation	BF16	vLLM	2K	Method unknown	—	1/11/2026, 4:06:46 PM
Qwen3-1.7B-FP8 ↗	Text generation	FP8	vllm	32K	vllm-official	—	10/26/2026, 6:27:28 AM
Qwen3-1.7B-FP8 ↗	Text generation	FP8	vllm	32K	vllm-official	—	10/26/2026, 6:23:11 AM
Qwen3-1.7B-FP8 ↗	Text generation	FP8	vllm	4K	vllm-official	—	10/26/2026, 6:22:18 AM
Qwen3-1.7B-FP8 ↗	Text generation	FP8	vllm	4K	vllm-official	—	10/26/2026, 6:21:16 AM
Qwen3-1.7B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/12/2026, 7:25:25 PM
Qwen3-1.7B-Q8 ↗	Text generation	Q8_0	llama.cpp	1K	wbench-http	—	10/14/2026, 9:16:40 AM
Qwen3-4B ↗	Text generation	BF16	vLLM	1K	wbench-http	—	10/14/2026, 9:04:46 AM
Qwen3-4B ↗	Text generation	BF16	vLLM	2K	Method unknown	—	1/11/2026, 4:14:58 PM
Qwen3-4B-AWQ ↗	Text generation	AWQ	vllm	8192in-2048out	wbench-fixed-http-v3	—	10/23/2026, 6:41:45 AM
Qwen3-4B-AWQ ↗	Text generation	AWQ	vllm	8192in-2048out	wbench-fixed-http-v3	—	10/23/2026, 6:39:18 AM
Qwen3-4B-AWQ ↗	Text generation	AWQ	vLLM	1K	wbench-http	—	10/14/2026, 9:10:51 AM
Qwen3-4B-FP8 ↗	Text generation	FP8	vllm	32K	vllm-official	—	10/26/2026, 6:47:58 AM
Qwen3-4B-FP8 ↗	Text generation	FP8	vllm	32K	vllm-official	—	10/26/2026, 6:43:10 AM
Qwen3-4B-FP8 ↗	Text generation	FP8	vllm	4K	vllm-official	—	10/26/2026, 6:41:49 AM
Qwen3-4B-FP8 ↗	Text generation	FP8	vllm	4K	vllm-official	—	10/26/2026, 6:40:24 AM
Qwen3-4B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/12/2026, 8:05:29 PM
Qwen3-4B-Q4 ↗	Text generation	Q4_K_M	llama.cpp	1K	wbench-http	—	10/14/2026, 9:19:13 AM
Qwen3-4B-Q8_0 ↗	Text generation	Q8_0	llama.cpp	1K	wbench-http	—	10/14/2026, 9:48:49 AM
Qwen3-8B ↗	Text generation	BF16	vllm	2K	vllm-official	—	10/14/2026, 6:16:09 AM
Qwen3-8B ↗	Text generation	BF16	vllm	2K	vllm-official	—	10/14/2026, 6:16:08 AM
Qwen3-8B ↗	Text generation	BF16	vllm	2K	vllm-official	—	10/14/2026, 6:16:07 AM
Qwen3-8B-AWQ ↗	Text generation	AWQ	vLLM	1K	wbench-http	—	10/14/2026, 9:17:31 AM
Qwen3-8B-NVFP4 ↗	Text generation	NVFP4	vllm	8192in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:08:42 PM
Qwen3-8B-NVFP4 ↗	Text generation	NVFP4	vllm	2048in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:08:10 PM
Qwen3-8B-NVFP4 ↗	Text generation	NVFP4	vllm	512in-512out	wbench-fixed-http-v1	—	10/18/2026, 6:07:42 PM
Qwen3-8B-Q4 ↗	Text generation	Q4_K_M	llama.cpp	1K	wbench-http	—	10/14/2026, 9:21:56 AM
Qwen3-14B-AWQ ↗	Text generation	AWQ	vLLM	1K	wbench-http	—	10/14/2026, 9:24:27 AM
Qwen3-14B-Q4 ↗	Text generation	Q4_K_M	llama.cpp	1K	wbench-http	—	10/14/2026, 9:24:54 AM
Qwen3-32B-FP8 ↗	Text generation	FP8	vLLM	1K	wbench-http	—	10/14/2026, 8:39:06 AM
Qwen3-32B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/13/2026, 1:35:30 AM
Qwen3-32B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/12/2026, 9:58:17 PM
Qwen3-32B-Q4 ↗	Text generation	Q4_K_M	llama.cpp	1K	wbench-http	—	10/14/2026, 9:29:00 AM
Qwen3-Embedding-0.6B ↗	Embedding	fp16	sentence-transformers	embedding	native-forward-v1	—	10/14/2026, 8:39:45 AM
Qwen3-Embedding-0.6B ↗	Embedding	nf4	sentence-transformers	embedding	native-forward-v1	—	10/14/2026, 6:35:06 AM
Qwen3-Embedding-0.6B ↗	Embedding	fp16	sentence-transformers	embedding	native-forward-v1	—	10/14/2026, 6:31:26 AM
Qwen3-TTS-12Hz-0.6B-CustomVoice ↗	Speech synthesis	Native	qwen-tts	tts	audio-native-v1	—	10/19/2026, 8:09:31 AM
Qwen3-VL-2B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/12/2026, 7:57:22 PM
Qwen3-VL-4B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/12/2026, 8:13:21 PM
Qwen3-VL-30B-A3B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/13/2026, 1:45:48 AM
Qwen3-VL-30B-A3B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/12/2026, 10:17:10 PM
Qwen3-VL-32B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/13/2026, 1:19:39 AM
Qwen3-VL-32B-FP8 ↗	Text generation	FP8	vLLM	2K	Method unknown	—	1/12/2026, 9:30:01 PM
Qwen3.5-0.8B ↗	Text generation	BF16	vllm	32K	vllm-official	—	10/26/2026, 6:15:09 AM
Qwen3.5-0.8B ↗	Text generation	BF16	vllm	32K	vllm-official	—	10/26/2026, 6:13:41 AM
Qwen3.5-0.8B ↗	Text generation	BF16	vllm	4K	vllm-official	—	10/26/2026, 6:13:12 AM
Qwen3.5-0.8B ↗	Text generation	BF16	vllm	4K	vllm-official	—	10/26/2026, 6:12:41 AM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/22/2026, 2:21:46 AM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 6:16:25 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 6:14:59 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 6:10:03 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 6:08:52 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 6:05:25 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 6:04:00 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:59:16 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:58:06 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:54:40 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:53:26 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:48:23 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:47:15 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:43:49 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:42:31 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:37:23 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:36:06 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	mmlu-test-zero-shot-strict-v4-n32-mt2048-t0	quality-v1	78.13%	10/21/2026, 5:30:58 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	gsm8k-test-zero-shot-strict-v4-n32-mt2048-t0	quality-v1	78.13%	10/21/2026, 5:28:11 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:23:51 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:23:26 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:22:29 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:22:04 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	mmlu-test-zero-shot-strict-v4-n32-mt2048-t0	quality-v1	81.25%	10/21/2026, 5:19:15 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	gsm8k-test-zero-shot-strict-v4-n32-mt2048-t0	quality-v1	84.38%	10/21/2026, 5:16:42 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:12:28 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:12:04 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:11:08 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:10:44 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	mmlu-test-zero-shot-strict-v4-n32-mt2048-t0	quality-v1	81.25%	10/21/2026, 5:08:00 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	gsm8k-test-zero-shot-strict-v4-n32-mt2048-t0	quality-v1	84.38%	10/21/2026, 5:05:35 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:01:27 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 5:01:01 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 4:59:56 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 4:59:33 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 4:59:08 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 4:59:07 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	mmlu-test-zero-shot-strict-v4-n32-mt2048-t0	quality-v1	81.25%	10/21/2026, 4:56:50 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	gsm8k-test-zero-shot-strict-v4-n32-mt2048-t0	quality-v1	78.13%	10/21/2026, 4:54:23 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	8192in-128out	wbench-fixed-http-v3	—	10/21/2026, 4:49:46 PM
Qwen3.6-35B-A3B-AWQ-4bit ↗	Text generation	AWQ	vllm	1024in-128out	wbench-fixed-http-v3	—	10/21/2026, 4:48:19 PM
Qwen3.6-35B-A3B-FP8 ↗	Text generation	FP8	vllm	32K	vllm-official	—	10/26/2026, 5:57:30 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	131072in-2048out	wbench-search-http-v1	—	10/25/2026, 9:17:12 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	65536in-2048out	wbench-search-http-v1	—	10/25/2026, 9:03:28 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	65536in-2048out	wbench-search-http-v1	—	10/25/2026, 8:40:15 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	32768in-2048out	wbench-search-http-v1	—	10/25/2026, 8:18:10 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	32768in-2048out	wbench-search-http-v1	—	10/25/2026, 7:57:47 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	131072in-2048out	wbench-search-http-v1	—	10/25/2026, 7:24:36 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	131072in-2048out	wbench-search-http-v1	—	10/25/2026, 7:09:07 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	65536in-2048out	wbench-search-http-v1	—	10/25/2026, 6:55:57 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	65536in-2048out	wbench-search-http-v1	—	10/25/2026, 6:34:21 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	32768in-2048out	wbench-search-http-v1	—	10/25/2026, 6:13:45 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	32768in-2048out	wbench-search-http-v1	—	10/25/2026, 5:54:42 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	131072in-2048out	wbench-fixed-http-v3	—	10/19/2026, 9:20:04 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	65536in-2048out	wbench-fixed-http-v3	—	10/19/2026, 8:34:56 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	32768in-2048out	wbench-fixed-http-v3	—	10/19/2026, 8:07:59 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	131072in-2048out	wbench-fixed-http-v3	—	10/19/2026, 7:40:50 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	65536in-2048out	wbench-fixed-http-v3	—	10/19/2026, 6:58:25 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	32768in-2048out	wbench-fixed-http-v3	—	10/19/2026, 6:33:20 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	131072in-2048out	wbench-fixed-http-v3	—	10/19/2026, 2:00:54 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	65536in-2048out	wbench-fixed-http-v3	—	10/19/2026, 1:18:10 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	32768in-2048out	wbench-fixed-http-v3	—	10/19/2026, 12:52:39 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	131072in-2048out	wbench-fixed-http-v3	—	10/19/2026, 12:27:36 AM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	65536in-2048out	wbench-fixed-http-v3	—	10/18/2026, 11:46:41 PM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	32768in-2048out	wbench-fixed-http-v3	—	10/18/2026, 11:22:38 PM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	aime2026-test-zero-shot-strict-v2-n3-mt16384-t0	quality-v1	0%	10/14/2026, 1:10:28 PM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	gsm8k-test-zero-shot-strict-v2-n3-mt16384-t0	quality-v1	0%	10/14/2026, 1:10:26 PM
Qwen3.8-27B-NVFP4 ↗	Text generation	NVFP4	vllm	mmlu-test-zero-shot-strict-v2-n3-mt16384-t0	quality-v1	0%	10/14/2026, 1:10:24 PM
Qwen3.8-Flash-Next-NVFP4-NVIDIA ↗	Text generation	NVFP4	strata	aime2026-test-zero-shot-strict-v1-n3-mt2048-t0	quality-v1	0%	10/14/2026, 10:50:49 AM
Qwen3.8-Flash-Next-NVFP4-NVIDIA ↗	Text generation	NVFP4	strata	mmlu-test-zero-shot-strict-v1-n3-mt2048-t0	quality-v1	100%	10/14/2026, 10:50:09 AM
Qwen3.8-Flash-Next-NVFP4-NVIDIA ↗	Text generation	NVFP4	strata	gsm8k-test-zero-shot-strict-v1-n3-mt2048-t0	quality-v1	100%	10/14/2026, 10:50:05 AM
Qwen3.8-Flash-Next-NVFP4-NVIDIA ↗	Text generation	NVFP4	strata	32768in-2048out	wbench-fixed-http-v1	—	10/14/2026, 10:49:58 AM
Qwen3.8-Flash-Next-NVFP4-NVIDIA ↗	Text generation	NVFP4	strata	8192in-2048out	wbench-fixed-http-v1	—	10/14/2026, 10:49:01 AM
Qwen3.8-Flash-Next-NVFP4-NVIDIA ↗	Text generation	NVFP4	strata	aime2026-test-zero-shot-strict-v1-n3-mt2048-t0	quality-v1	33.33%	10/14/2026, 10:42:50 AM
Qwen3.8-Flash-Next-NVFP4-NVIDIA ↗	Text generation	NVFP4	strata	mmlu-test-zero-shot-strict-v1-n3-mt2048-t0	quality-v1	100%	10/14/2026, 10:42:03 AM
Qwen3.8-Flash-Next-NVFP4-NVIDIA ↗	Text generation	NVFP4	strata	gsm8k-test-zero-shot-strict-v1-n3-mt2048-t0	quality-v1	100%	10/14/2026, 10:41:57 AM
Qwen3.8-Flash-Next-NVFP4-NVIDIA ↗	Text generation	NVFP4	strata	32768in-2048out	wbench-fixed-http-v1	—	10/14/2026, 10:41:46 AM
Qwen3.8-Flash-Next-NVFP4-NVIDIA ↗	Text generation	NVFP4	strata	8192in-2048out	wbench-fixed-http-v1	—	10/14/2026, 10:40:44 AM
Qwen3.8-Flash-Next-UD-Q4_K_XL ↗	Text generation	Q4_K_XL	strata	aime2026-test-zero-shot-strict-v1-n3-mt2048-t0	quality-v1	0%	10/14/2026, 10:19:21 AM
Qwen3.8-Flash-Next-UD-Q4_K_XL ↗	Text generation	Q4_K_XL	strata	mmlu-test-zero-shot-strict-v1-n3-mt2048-t0	quality-v1	100%	10/14/2026, 10:18:19 AM
Qwen3.8-Flash-Next-UD-Q4_K_XL ↗	Text generation	Q4_K_XL	strata	gsm8k-test-zero-shot-strict-v1-n3-mt2048-t0	quality-v1	100%	10/14/2026, 10:18:10 AM
Qwen3.8-Flash-Next-UD-Q4_K_XL ↗	Text generation	Q4_K_XL	strata	32768in-2048out	wbench-fixed-http-v1	—	10/14/2026, 10:17:57 AM
Qwen3.8-Flash-Next-UD-Q4_K_XL ↗	Text generation	Q4_K_XL	strata	8192in-2048out	wbench-fixed-http-v1	—	10/14/2026, 10:16:49 AM
StartLux-Decision-v1-0.8B ↗	Decision	bf16	startlux-native	decision	native-forward-v1	—	10/23/2026, 12:09:03 PM
StartLux-Decision-v1-0.8B ↗	Decision	bf16	startlux-native	decision	native-forward-v1	—	10/23/2026, 12:06:36 PM
StartLux-Decision-v1.1-0.8B-Instant ↗	Decision	bf16	startlux-transformers	decision	native-forward-v1	—	10/23/2026, 12:06:09 PM
StartLux-Decision-v1.1-0.8B-Instant ↗	Decision	bf16	startlux-transformers	decision	native-forward-v1	—	10/23/2026, 12:04:32 PM
StartLux-Decision-v1.1-0.8B-Instant ↗	Decision	bf16	startlux-transformers	decision	native-forward-v1	—	10/23/2026, 9:23:32 AM
WeVisOne-VL-2B ↗	Text generation	BF16	vllm	131072in-2048out	wbench-search-http-v1	—	10/26/2026, 1:05:52 AM
WeVisOne-VL-2B ↗	Text generation	BF16	vllm	131072in-2048out	wbench-search-http-v1	—	10/26/2026, 12:51:38 AM
WeVisOne-VL-2B ↗	Text generation	BF16	vllm	65536in-2048out	wbench-search-http-v1	—	10/26/2026, 12:25:56 AM
WeVisOne-VL-2B ↗	Text generation	BF16	vllm	65536in-2048out	wbench-search-http-v1	—	10/26/2026, 12:06:16 AM
WeVisOne-VL-2B ↗	Text generation	BF16	vllm	32768in-2048out	wbench-search-http-v1	—	10/25/2026, 11:42:06 PM
WeVisOne-VL-2B ↗	Text generation	BF16	vllm	32768in-2048out	wbench-search-http-v1	—	10/25/2026, 11:26:15 PM
WeVisOne-VL-2B ↗	Text generation	BF16	vllm	32768in-2048out	wbench-search-http-v1	—	10/25/2026, 10:40:36 PM
whisper-tiny ↗	Speech recognition	fp16	faster-whisper	asr	audio-native-v1	—	10/14/2026, 8:24:13 AM
whisper-tiny ↗	Speech recognition	fp16	faster-whisper	asr	audio-native-v1	—	10/14/2026, 6:18:59 AM

Running benchmarks

0Updates automatically

No benchmarks are currently reported as running.

RTX PRO 6000 Blackwell Server Edition · 96 GB · 403 queued · 0 running · 0 awaiting reassignment · 6 succeeded · 3 failed or exhausted

Methodology

Average of error-free runs for the selected recorded configuration. Missing methods and fallback runs are not ranked. Performance index = average / fastest matching hardware × 100. Value index = performance per purchase cost / best comparable value × 100. Quality is separate by dataset and protocol. Prices are dated seller listings, not completed transactions or market averages.

Updated 11/5/2026, 11:07:29 PM

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	120.3	29 ms	—	2	152.1	2	061d047a ↗
B4	125.19	124 ms	—	2.03	169.7	2	061d047a ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	232.81	—	—	1.42	113.5	2	ba5a45ac ↗
B4	364.19	—	—	1.43	115.8	2	ba5a45ac ↗