20 models with qualified results

Models

Measured configurations by model. Open a model to compare GPUs on one exact test.

Get your startup command

Choose your operating system and enter a model repository. Launch support is checked separately from performance results.

All tasks · RTX PRO 6000 Blackwell Server Edition · 96 GB

Speeds compare only within one exact test. Recipe availability is checked on each report against the published recipe executor.

WeVisOne-VL-2B

tencent · 6 tested configurations

BF16 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 153tok/s
Best measured · 16 concurrent requests: 319tok/s
Observed GPU occupancy: 85.3GiB
First token: 753 ms

Use this recipe · 16 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	153.44	13.39 s	753 ms	85.25	343.4	1	d2914dd8 ↗
C2	208.65	19.64 s	1.1 s	85.25	360.4	1	d2914dd8 ↗
C4	259.19	33.21 s	1.6 s	85.25	398.9	1	d2914dd8 ↗
C8	295.37	59.5 s	2.1 s	85.25	422	1	d2914dd8 ↗
C12	311.33	86.14 s	2.7 s	85.25	432.4	1	d2914dd8 ↗
C16	319.17	112.4 s	3.3 s	85.25	438	1	d2914dd8 ↗

5 other test configurations on this GPU

BF16 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 162tok/s
Best measured · 18 concurrent requests: 857tok/s
Observed GPU occupancy: 85.3GiB
First token: 71 ms

Use this recipe · 18 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 7 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	161.57	12.72 s	71 ms	85.25	327.8	1	75198c63 ↗
C2	269.43	15.21 s	86 ms	85.25	349.1	1	75198c63 ↗
C4	460.41	17.83 s	106 ms	85.25	427.3	1	75198c63 ↗
C8	644.45	25.52 s	127 ms	85.25	486.9	1	75198c63 ↗
C16	771.04	42.77 s	184 ms	85.25	533.9	1	75198c63 ↗
C17	812.79	43.14 s	195 ms	85.25	550.4	1	75198c63 ↗
C18	856.64	43.41 s	196 ms	85.25	568.9	1	75198c63 ↗

BF16 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 101tok/s
Best measured · 8 concurrent requests: 151tok/s
Observed GPU occupancy: 85.3GiB
First token: 2.3 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 5 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	101.26	20.26 s	2.3 s	85.25	369.1	1	c5c9cf6f ↗
C2	123.11	33.41 s	3.6 s	85.25	404.8	1	c5c9cf6f ↗
C4	140.19	64.26 s	4.5 s	85.25	430	1	c5c9cf6f ↗
C6	142.08	97.79 s	5.3 s	85.25	430.4	1	c5c9cf6f ↗
C8	150.73	122.37 s	6.3 s	85.25	446.7	1	c5c9cf6f ↗

BF16 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 114tok/s
Best measured · 9 concurrent requests: 433tok/s
Observed GPU occupancy: 85.3GiB
First token: 135 ms

Use this recipe · 9 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 7 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	113.51	18.06 s	135 ms	85.25	341.3	1	5fee1315 ↗
C2	153.3	26.73 s	161 ms	85.25	369.5	1	5fee1315 ↗
C4	281.86	29.1 s	202 ms	85.25	483.8	1	5fee1315 ↗
C6	293.74	41.93 s	214 ms	85.25	459.2	1	5fee1315 ↗
C7	340.1	42.27 s	227 ms	85.25	496.1	1	5fee1315 ↗
C8	387.93	42.41 s	237 ms	85.25	534.9	1	5fee1315 ↗
C9	433.07	42.78 s	246 ms	85.25	575.8	1	5fee1315 ↗

BF16 · 128K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 56.1tok/s
Best measured · 4 concurrent requests: 66.8tok/s
Observed GPU occupancy: 85.3GiB
First token: 8 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.09	36.63 s	8 s	85.25	418.9	1	2c661c7e ↗
C2	62.52	67.49 s	12 s	85.25	444.3	1	2c661c7e ↗
C3	63.59	109.31 s	13.5 s	85.25	446.8	1	2c661c7e ↗
C4	66.75	144.63 s	15 s	85.25	461.6	1	2c661c7e ↗

BF16 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 71tok/s
Best measured · 4 concurrent requests: 165tok/s
Observed GPU occupancy: 85.3GiB
First token: 268 ms

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	71.05	28.83 s	268 ms	85.25	356.6	1	1ff0c98a ↗
C2	89.43	45.82 s	323 ms	85.25	399.6	1	1ff0c98a ↗
C3	135.43	45.41 s	379 ms	85.25	470.9	1	1ff0c98a ↗
C4	165.29	49.61 s	420 ms	85.25	524	1	1ff0c98a ↗

Qwen3.8-27B-NVFP4

unsloth · 11 tested configurations

NVFP4 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 52.8tok/s
Best measured · 8 concurrent requests: 215tok/s
Observed GPU occupancy: 87.4GiB
First token: 3.3 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.78	38.8 s	3.3 s	87.37	337.3	1	aee8b33e ↗
C2	87.2	48.71 s	4.9 s	87.37	343.4	1	aee8b33e ↗
C4	142.14	64.91 s	7 s	87.37	375.3	1	aee8b33e ↗
C6	187.69	79.85 s	8.1 s	87.37	402.8	1	aee8b33e ↗
C7	202.31	85.21 s	8.7 s	87.37	411.8	1	aee8b33e ↗
C8	215.05	94.52 s	9.3 s	87.37	421.4	1	aee8b33e ↗

10 other test configurations on this GPU

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57tok/s
Best measured · 8 concurrent requests: 374tok/s
Observed GPU occupancy: 87.4GiB
First token: 222 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.96	36.17 s	222 ms	87.37	322.6	1	bced0642 ↗
C2	104.14	39.34 s	355 ms	87.37	305.9	1	bced0642 ↗
C4	198.42	41.3 s	704 ms	87.37	319.3	1	bced0642 ↗
C6	291.12	42.23 s	1 s	87.37	342.1	1	bced0642 ↗
C7	333.32	43.04 s	1.1 s	87.37	349.1	1	bced0642 ↗
C8	374.04	43.82 s	1.3 s	87.37	353.2	1	bced0642 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.1tok/s
Best measured · 8 concurrent requests: 373tok/s
Observed GPU occupancy: 86.6GiB
First token: 231 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.09	35.88 s	231 ms	86.63	319	1	37f29252 ↗
C4	197.8	41.44 s	650 ms	86.63	330.2	1	37f29252 ↗
C8	372.88	43.98 s	1.3 s	86.63	367.3	1	37f29252 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 56.4tok/s
Best measured · 8 concurrent requests: 371tok/s
Observed GPU occupancy: 86.4GiB
First token: 244 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	56.4	36.51 s	244 ms	86.35	319.1	1	791b887d ↗
C4	196.45	41.73 s	654 ms	86.35	315.5	1	791b887d ↗
C8	370.68	44.24 s	1.3 s	86.36	348.1	1	791b887d ↗

NVFP4 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 44.9tok/s
Best measured · 4 concurrent requests: 96.8tok/s
Observed GPU occupancy: 87.4GiB
First token: 8.6 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	44.86	45.67 s	8.6 s	87.37	360.5	1	d2ff32ec ↗
C2	67.59	63.21 s	12.7 s	87.37	379.1	1	d2ff32ec ↗
C3	84.95	84.07 s	14.6 s	87.37	398.5	1	d2ff32ec ↗
C4	96.77	105.62 s	16.3 s	87.37	415.8	1	d2ff32ec ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54.7tok/s
Best measured · 8 concurrent requests: 340tok/s
Observed GPU occupancy: 87.4GiB
First token: 329 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.72	37.56 s	329 ms	87.37	322.4	1	d3f6b770 ↗
C2	99.95	40.99 s	522 ms	87.37	315.9	1	d3f6b770 ↗
C4	189.07	43.34 s	971 ms	87.37	337.8	1	d3f6b770 ↗
C6	264.7	46.44 s	1.3 s	87.37	364	1	d3f6b770 ↗
C7	303.08	47.32 s	1.4 s	87.37	373.4	1	d3f6b770 ↗
C8	339.61	48.27 s	1.7 s	87.37	381.2	1	d3f6b770 ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54tok/s
Best measured · 8 concurrent requests: 337tok/s
Observed GPU occupancy: 86.4GiB
First token: 361 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.04	38.07 s	361 ms	86.36	320.7	1	417f69be ↗
C4	187.12	43.81 s	879 ms	86.36	333.2	1	417f69be ↗
C8	336.95	48.66 s	1.6 s	86.36	379.7	1	417f69be ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 54.5tok/s
Best measured · 8 concurrent requests: 339tok/s
Observed GPU occupancy: 86.6GiB
First token: 359 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	54.54	37.55 s	359 ms	86.63	324.9	1	7d1a7c8e ↗
C4	188.48	43.48 s	878 ms	86.63	350.2	1	7d1a7c8e ↗
C8	338.65	48.43 s	1.6 s	86.63	396.9	1	7d1a7c8e ↗

NVFP4 · 128K → 2K tokens · No cache reuse
Output throughput · includes prefill · 1 request
31.5tok/s
Best measured · 2 concurrent requests
41.3tok/s
Observed GPU occupancy
87.4GiB
First token
24.9 s
Use this recipe · 2 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 concurrency levels
Total output tokens divided by the full request window, including prefill and queueing.
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
vllm · vLLM 0.28.0
Protocol
wbench-search-http-v1
Dataset
migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload
131072in-2048out
Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
C1 31.54 64.93 s 24.9 s 87.37 402.7 1 73fcb7c1 ↗
C2 41.27 111.14 s 34 s 87.37 426.3 1 73fcb7c1 ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	31.54	64.93 s	24.9 s	87.37	402.7	1	73fcb7c1 ↗
C2	41.27	111.14 s	34 s	87.37	426.3	1	73fcb7c1 ↗

NVFP4 · 128K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 31.5tok/s
Best measured · 8 concurrent requests: 57.7tok/s
Observed GPU occupancy: 86.6GiB
First token: 24.9 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=isolated:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	31.45	65.12 s	24.9 s	86.63	415.7	1	448b0777 ↗
C4	50.67	220.6 s	45.8 s	86.63	469.4	1	448b0777 ↗
C8	57.74	421.65 s	61.9 s	86.63	494.5	1	448b0777 ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 50tok/s
Best measured · 8 concurrent requests: 287tok/s
Observed GPU occupancy: 86.4GiB
First token: 506 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	49.98	40.98 s	506 ms	86.36	322.7	1	68d1f2c6 ↗
C4	171.33	47.85 s	1 s	86.36	365.5	1	68d1f2c6 ↗
C8	287.31	57.12 s	1.9 s	86.36	425.4	1	68d1f2c6 ↗

Qwen3.8-27B-NVFP4

RadixArk · 12 tested configurations

NVFP4 · 32K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 55.6tok/s
Best measured · 8 concurrent requests: 225tok/s
Observed GPU occupancy: 86GiB
First token: 3.2 s

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	55.59	36.91 s	3.2 s	85.98	341	1	dca51ce5 ↗
C2	91.37	46.48 s	4.6 s	85.98	342.9	1	dca51ce5 ↗
C4	149.21	62.63 s	6.5 s	85.98	372.9	1	dca51ce5 ↗
C6	196.73	76.24 s	7.8 s	85.98	398.9	1	dca51ce5 ↗
C7	211.87	81.41 s	8.3 s	85.98	408.7	1	dca51ce5 ↗
C8	224.61	90.5 s	8.9 s	85.98	414.8	1	dca51ce5 ↗

11 other test configurations on this GPU

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 60.1tok/s
Best measured · 8 concurrent requests: 393tok/s
Observed GPU occupancy: 86GiB
First token: 217 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	60.1	34.34 s	217 ms	85.98	323	1	9cdf45b7 ↗
C2	109.3	37.48 s	349 ms	85.98	310.8	1	9cdf45b7 ↗
C4	209.23	39.18 s	686 ms	85.98	324.8	1	9cdf45b7 ↗
C6	306.94	40.06 s	992 ms	85.98	336	1	9cdf45b7 ↗
C7	351.24	40.83 s	1.1 s	85.98	342.6	1	9cdf45b7 ↗
C8	392.93	41.71 s	1.3 s	85.98	348.2	1	9cdf45b7 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 59.7tok/s
Best measured · 8 concurrent requests: 394tok/s
Observed GPU occupancy: 86.8GiB
First token: 230 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	59.68	34.59 s	230 ms	86.82	323.3	1	83d37469 ↗
C4	207.31	39.54 s	664 ms	86.82	320	1	83d37469 ↗
C8	394.04	41.6 s	1.2 s	86.82	346.1	1	83d37469 ↗

NVFP4 · 32K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 60.5tok/s
Best measured · 8 concurrent requests: 395tok/s
Observed GPU occupancy: 86.9GiB
First token: 228 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 32768in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	60.47	33.87 s	228 ms	86.86	320.3	1	cbf1431c ↗
C4	207.86	39.44 s	636 ms	86.86	333.6	1	cbf1431c ↗
C8	394.52	41.6 s	1.3 s	86.86	361.6	1	cbf1431c ↗

NVFP4 · 64K → 2K tokens · No cache reuse

Output throughput · includes prefill · 1 request: 47.1tok/s
Best measured · 4 concurrent requests: 101tok/s
Observed GPU occupancy: 86GiB
First token: 8.4 s

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	47.15	43.45 s	8.4 s	85.98	364.4	1	e5a201fa ↗
C2	70.39	60.71 s	12.3 s	85.98	381.3	1	e5a201fa ↗
C3	88.36	80.85 s	14.2 s	85.98	403.6	1	e5a201fa ↗
C4	100.66	101.65 s	15.8 s	85.98	419.2	1	e5a201fa ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.5tok/s
Best measured · 8 concurrent requests: 355tok/s
Observed GPU occupancy: 86GiB
First token: 326 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 6 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.45	35.8 s	326 ms	85.98	326.1	1	67a5e6d3 ↗
C2	104.7	39.14 s	515 ms	85.98	321	1	67a5e6d3 ↗
C4	198.72	41.24 s	954 ms	85.98	344.2	1	67a5e6d3 ↗
C6	277.65	44.28 s	1.3 s	85.98	358.8	1	67a5e6d3 ↗
C7	317.41	45.18 s	1.4 s	85.98	369.2	1	67a5e6d3 ↗
C8	354.95	46.19 s	1.6 s	85.98	377.1	1	67a5e6d3 ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.1tok/s
Best measured · 8 concurrent requests: 356tok/s
Observed GPU occupancy: 86.8GiB
First token: 358 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.08	36.15 s	358 ms	86.82	326	1	275a096d ↗
C4	196.91	41.63 s	870 ms	86.83	338.7	1	275a096d ↗
C8	355.89	46.08 s	1.6 s	86.83	375.7	1	275a096d ↗

NVFP4 · 64K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 57.7tok/s
Best measured · 8 concurrent requests: 356tok/s
Observed GPU occupancy: 86.9GiB
First token: 357 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 65536in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	57.67	35.51 s	357 ms	86.86	325.8	1	fe8141d2 ↗
C4	197.42	41.52 s	870 ms	86.86	354.3	1	fe8141d2 ↗
C8	356.25	46.05 s	1.6 s	86.86	392.4	1	fe8141d2 ↗

NVFP4 · 128K → 2K tokens · No cache reuse
Output throughput · includes prefill · 1 request
32.7tok/s
Best measured · 2 concurrent requests
42.5tok/s
Observed GPU occupancy
86GiB
First token
24.3 s
Use this recipe · 2 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 concurrency levels
Total output tokens divided by the full request window, including prefill and queueing.
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
vllm · vLLM 0.28.0
Protocol
wbench-search-http-v1
Dataset
migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
Workload
131072in-2048out
Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
C1 32.72 62.59 s 24.3 s 85.99 406.5 1 28f51cf1 ↗
C2 42.54 106.24 s 33.5 s 85.99 428.9 1 28f51cf1 ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	32.72	62.59 s	24.3 s	85.99	406.5	1	28f51cf1 ↗
C2	42.54	106.24 s	33.5 s	85.99	428.9	1	28f51cf1 ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 53tok/s
Best measured · 4 concurrent requests: 181tok/s
Observed GPU occupancy: 86GiB
First token: 435 ms

Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-search-http-v1
Dataset: migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.97	38.72 s	435 ms	85.99	333.9	1	78fc908c ↗
C2	97.25	42.14 s	646 ms	85.99	340.8	1	78fc908c ↗
C3	138.66	44.36 s	822 ms	85.99	358.8	1	78fc908c ↗
C4	181.16	45.25 s	1.1 s	85.99	375.3	1	78fc908c ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 52.9tok/s
Best measured · 8 concurrent requests: 301tok/s
Observed GPU occupancy: 86.9GiB
First token: 508 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.9	38.72 s	508 ms	86.86	338	1	7cea351c ↗
C4	179.81	45.65 s	1 s	86.86	387.5	1	7cea351c ↗
C8	301.16	54.48 s	1.9 s	86.86	443.7	1	7cea351c ↗

NVFP4 · 128K → 2K tokens · Prefix cache reuse

Output throughput · includes prefill · 1 request: 52.6tok/s
Best measured · 8 concurrent requests: 301tok/s
Observed GPU occupancy: 86.8GiB
First token: 509 ms

Use this recipe · 8 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 concurrency levels

Total output tokens divided by the full request window, including prefill and queueing.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: vllm · vLLM 0.28.0
Protocol: wbench-fixed-http-v3
Dataset: synthetic-tokenizer-v3:cache=prefix-reuse:repeats=3
Workload: 131072in-2048out

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	52.59	38.95 s	509 ms	86.83	328.8	1	d14962c1 ↗
C4	179.42	45.72 s	1.1 s	86.83	371.9	1	d14962c1 ↗
C8	300.98	54.52 s	1.9 s	86.83	426	1	d14962c1 ↗

StartLux-Decision-v1-0.8B
startlux-models · 2 tested configurations
- bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  50.3decisions/sec
  Observed GPU occupancy
  3.2GiB
  P95 latency
  20 ms
  Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  startlux-native · PyTorch 2.11.0+cu130
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 50.25 20 ms — 3.17 161.3 1 df141700 ↗
- 1 other test configurations on this GPU
  - bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
    Throughput · Batch 1
    50.3decisions/sec
    Observed GPU occupancy
    2.3GiB
    P95 latency
    20 ms
    Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
    Test details
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    startlux-native · PyTorch 2.11.0+cu130
    Protocol
    native-forward-v1
    Dataset
    typed-invoices-v1
    Workload
    decision
    Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 50.28 20 ms — 2.29 165 1 8c1985ea ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.25	20 ms	—	3.17	161.3	1	df141700 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.28	20 ms	—	2.29	165	1	8c1985ea ↗

Intern-Decision-0.8B

internlm · 4 tested configurations

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
Throughput · Batch 1
50.8decisions/sec
Best measured · Batch 17
347decisions/sec
Observed GPU occupancy
2.4GiB
P95 latency
20 ms
Use this recipe · Batch 17 ↗Evidence report and run availability ↗Same test on other GPUs
Test details · 2 batch sizes
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
intern-decision-native · PyTorch 2.11.0+cu130
Protocol
native-forward-v1
Dataset
typed-invoices-v1
Workload
decision
Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B1 50.83 20 ms — 2.36 169 2 18b06f63 ↗
B17 346.72 50 ms — 3.07 378 1 6ccb367c ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.83	20 ms	—	2.36	169	2	18b06f63 ↗
B17	346.72	50 ms	—	3.07	378	1	6ccb367c ↗

3 other test configurations on this GPU

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 399 s

Throughput · Batch 1: 52decisions/sec
Best measured · Batch 16: 334decisions/sec
Observed GPU occupancy: 2.4GiB
P95 latency: 20 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 21 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: intern-decision-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	52.04	20 ms	—	2.36	162.6	1	156910e7 ↗
B2	96.21	21 ms	—	2.38	213.4	1	156910e7 ↗
B4	177.08	23 ms	—	2.43	243.4	1	156910e7 ↗
B8	299.06	27 ms	—	2.57	341.2	1	156910e7 ↗
B12	306.24	40 ms	—	36.98	349.6	1	156910e7 ↗
B14	322.36	43 ms	—	36.98	365.9	1	156910e7 ↗
B15	328.14	46 ms	—	36.98	369.8	1	156910e7 ↗
B16	333.92	48 ms	—	3.19	368.4	1	156910e7 ↗
B17	331.9	52 ms	—	36.98	379.5	1	156910e7 ↗
B18	310.84	58 ms	—	36.98	365.4	1	156910e7 ↗
B20	316.13	64 ms	—	36.98	375.1	1	156910e7 ↗
B24	312.87	77 ms	—	36.98	377.5	1	156910e7 ↗
B28	309.61	91 ms	—	36.98	379.4	1	156910e7 ↗
B32	319.44	101 ms	—	3.44	376.3	1	156910e7 ↗
B40	307.24	131 ms	—	36.98	379.6	1	156910e7 ↗
B48	306.75	159 ms	—	36.98	380.4	1	156910e7 ↗
B64	304.06	213 ms	—	4.2	383.7	1	156910e7 ↗
B128	285.17	453 ms	—	6.23	387.8	1	156910e7 ↗
B256	284.61	904 ms	—	10.63	390.3	1	156910e7 ↗
B512	282.22	2.04 s	—	19.42	390.1	1	156910e7 ↗
B1024	286.53	3.58 s	—	36.98	396.2	1	156910e7 ↗

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
Throughput · Batch 1
51.5decisions/sec
Observed GPU occupancy
2.6GiB
P95 latency
20 ms
Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
Test details
Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
Engine
intern-decision-native · PyTorch 2.11.0
Protocol
native-forward-v1
Dataset
typed-invoices-v1
Workload
decision
Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
B1 51.51 20 ms — 2.6 164.2 1 7f2e07c0 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	51.51	20 ms	—	2.6	164.2	1	7f2e07c0 ↗

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 396 s

Throughput · Batch 1: 52.8decisions/sec
Best measured · Batch 17: 350decisions/sec
Observed GPU occupancy: 2.4GiB
P95 latency: 19 ms

Use this recipe · Batch 17 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 21 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: intern-decision-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	52.8	19 ms	—	2.36	172.8	1	5eef633e ↗
B2	95.72	21 ms	—	2.38	186.5	1	5eef633e ↗
B4	177.64	23 ms	—	2.43	242	1	5eef633e ↗
B8	306.42	26 ms	—	2.57	344.2	1	5eef633e ↗
B12	322.68	37 ms	—	37.46	371.9	1	5eef633e ↗
B13	336	39 ms	—	37.46	380.8	1	5eef633e ↗
B14	344.61	41 ms	—	37.46	392.6	1	5eef633e ↗
B15	345.75	44 ms	—	37.46	389.5	1	5eef633e ↗
B16	345.28	47 ms	—	2.85	383.9	1	5eef633e ↗
B17	349.83	49 ms	—	37.46	397.8	1	5eef633e ↗
B18	327.34	55 ms	—	37.46	383.4	1	5eef633e ↗
B20	333.21	60 ms	—	37.46	402.6	1	5eef633e ↗
B24	331.98	73 ms	—	37.46	401.2	1	5eef633e ↗
B28	327.01	86 ms	—	37.46	399.2	1	5eef633e ↗
B32	331.67	97 ms	—	3.41	397.7	1	5eef633e ↗
B48	326.46	148 ms	—	37.46	405.9	1	5eef633e ↗
B64	318.22	202 ms	—	4.51	403.2	1	5eef633e ↗
B128	298.77	430 ms	—	6.71	405.8	1	5eef633e ↗
B256	298.58	859 ms	—	11.11	407.3	1	5eef633e ↗
B512	299.67	1.71 s	—	19.9	411.2	1	5eef633e ↗
B1024	297.46	3.64 s	—	37.46	410.6	1	5eef633e ↗

clef-flash

Cloudflare · 2 tested configurations

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 33 s

Throughput · Batch 1: 31.8decisions/sec
Best measured · Batch 4: 39.3decisions/sec
Observed GPU occupancy: 18.9GiB
P95 latency: 32 ms

Use this recipe · Batch 4 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: clef-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	31.83	32 ms	—	18.95	388.5	1	10c64aff ↗
B4	39.27	93 ms	—	18.95	449.5	1	10c64aff ↗
B16	5.41	8.21 s	—	19.28	233.4	1	10c64aff ↗
B64	15.43	9.66 s	—	21.89	324.3	1	10c64aff ↗

1 other test configurations on this GPU
- bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 16
  47.7decisions/sec
  Observed GPU occupancy
  19.5GiB
  P95 latency
  338 ms
  Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  clef-native · PyTorch 2.11.0+cu130
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B16 47.71 338 ms — 19.55 499.5 2 82d98b36 ↗

StartLux-Decision-v1.1-0.8B-Instant
startlux-models · 2 tested configurations
- bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  18.7decisions/sec
  Observed GPU occupancy
  2.7GiB
  P95 latency
  54 ms
  Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  startlux-transformers · PyTorch 2.11.0+cu130
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 18.69 54 ms — 2.73 134.5 2 a4a69815 ↗
- 1 other test configurations on this GPU
  - bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
    Throughput · Batch 1
    18.8decisions/sec
    Observed GPU occupancy
    2.5GiB
    P95 latency
    54 ms
    Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
    Test details
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    startlux-transformers · PyTorch 2.11.0
    Protocol
    native-forward-v1
    Dataset
    typed-invoices-v1
    Workload
    decision
    Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    B1 18.83 54 ms — 2.54 125.5 1 c342a9ca ↗
Intern-Decision-2B
internlm · 1 tested configuration
- bf16 · Text → Structured data · typed-invoices-v1 · Test duration 6 s
  Throughput · Batch 1
  50.3decisions/sec
  Observed GPU occupancy
  4.9GiB
  P95 latency
  21 ms
  Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  intern-decision-native · PyTorch 2.11.0
  Protocol
  native-forward-v1
  Dataset
  typed-invoices-v1
  Workload
  decision
  Batch size Mean throughput (decisions/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 50.29 21 ms — 4.9 216.2 1 e4bcd6a3 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B16	47.71	338 ms	—	19.55	499.5	2	82d98b36 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	18.69	54 ms	—	2.73	134.5	2	a4a69815 ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	18.83	54 ms	—	2.54	125.5	1	c342a9ca ↗

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	50.29	21 ms	—	4.9	216.2	1	e4bcd6a3 ↗

clef-omni

Cloudflare · 1 tested configuration

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 44 s

Throughput · Batch 1: 7.5decisions/sec
Best measured · Batch 64: 7.5decisions/sec
Observed GPU occupancy: 60.9GiB
P95 latency: 134 ms

Use this recipe · Batch 64 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: clef-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	7.45	134 ms	—	60.88	232.8	1	16597f91 ↗
B4	7.47	537 ms	—	60.89	249.7	1	16597f91 ↗
B16	7.48	2.14 s	—	60.89	252.8	1	16597f91 ↗
B64	7.49	8.56 s	—	60.89	257.2	1	16597f91 ↗

clef

Cloudflare · 1 tested configuration

bf16 · Text → Structured data · typed-invoices-v1 · Test duration 46 s

Throughput · Batch 1: 12.3decisions/sec
Observed GPU occupancy: 52.4GiB
P95 latency: 82 ms

Use this recipe · Batch 1 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 4 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: clef-native · PyTorch 2.11.0
Protocol: native-forward-v1
Dataset: typed-invoices-v1
Workload: decision

Batch size	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	12.32	82 ms	—	52.4	469.1	1	243b7f64 ↗
B4	1.4	8.09 s	—	52.4	225.7	1	243b7f64 ↗
B16	4.33	9.1 s	—	52.98	308.5	1	243b7f64 ↗
B64	9.39	12.46 s	—	57.95	435.9	1	243b7f64 ↗

Qwen3-4B-AWQ
Qwen · 2 tested configurations
- AWQ · 8K → 2K tokens · No cache reuse
  Output throughput · includes prefill · 1 request
  215tok/s
  Best measured · 4 concurrent requests
  517tok/s
  Observed GPU occupancy
  86.3GiB
  First token
  298 ms
  Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 concurrency levels
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  vllm · vLLM 0.29.0
  Protocol
  wbench-fixed-http-v3
  Dataset
  migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=isolated:repeats=3
  Workload
  8192in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 214.6 9.63 s 298 ms 86.28 303.8 1 86d36e4f ↗
  C4 516.94 16.14 s 872 ms 86.28 339.5 1 86d36e4f ↗
- 1 other test configurations on this GPU
  - AWQ · 8K → 2K tokens · Prefix cache reuse
    Output throughput · includes prefill · 1 request
    223tok/s
    Best measured · 4 concurrent requests
    742tok/s
    Observed GPU occupancy
    86.3GiB
    First token
    28 ms
    Use this recipe · 4 concurrent requests ↗Evidence report and run availability ↗Same test on other GPUs
    Test details · 2 concurrency levels
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    vllm · vLLM 0.29.0
    Protocol
    wbench-fixed-http-v3
    Dataset
    migration-analysis-v1:sha256=68590c248e43f32c04dfd8062464fd1e4856917795fe307225e5c6112a54fe94:cache=prefix-reuse:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 222.74 9.19 s 28 ms 86.28 288.8 1 adbfef8a ↗
    C4 741.85 11.05 s 35 ms 86.28 335 1 adbfef8a ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	214.6	9.63 s	298 ms	86.28	303.8	1	86d36e4f ↗
C4	516.94	16.14 s	872 ms	86.28	339.5	1	86d36e4f ↗

Concurrency	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
C1	222.74	9.19 s	28 ms	86.28	288.8	1	adbfef8a ↗
C4	741.85	11.05 s	35 ms	86.28	335	1	adbfef8a ↗

embeddinggemma-2

google · 2 tested configurations

nf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 189vectors/sec
Best measured · Batch 16: 2,572vectors/sec
Observed GPU occupancy: 1.3GiB
P95 latency: 6 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	189.23	6 ms	—	1.33	145.5	1	c673556b ↗
B4	713.32	6 ms	—	1.33	142.8	1	c673556b ↗
B16	2,571.53	7 ms	—	1.33	162.7	1	c673556b ↗

1 other test configurations on this GPU

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 207vectors/sec
Best measured · Batch 16: 2,939vectors/sec
Observed GPU occupancy: 1.5GiB
P95 latency: 5 ms

Use this recipe · Batch 16 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	206.79	5 ms	—	1.48	119.3	1	196678d3 ↗
B4	764.95	6 ms	—	1.48	141.4	1	196678d3 ↗
B16	2,938.63	6 ms	—	1.48	161.8	1	196678d3 ↗

IndexTTS-2.5
IndexTeam · 1 tested configuration
- Native · Not recorded → Audio · tts_prompts · Test duration 9 s
  Throughput · Batch 1
  4.2audio-sec/sec
  Observed GPU occupancy
  6.5GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  indextts · PyTorch 2.8.0+cu128
  Protocol
  audio-native-v1
  Dataset
  tts_prompts
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 4.24 — — 6.48 221.7 1 a46ec2c4 ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	4.24	—	—	6.48	221.7	1	a46ec2c4 ↗

granite-speech-5.0-470m-turboctc

ibm-granite · 2 tested configurations

Native · Audio → Text · asr_prompts · Test duration 16 s

Throughput · Batch 1: 2,662audio-sec/sec
Best measured · Batch 32: 9,194audio-sec/sec
Observed GPU occupancy: 1.6GiB
P95 latency: Not recorded

Evidence report and run availability ↗Same test on other GPUs

Test details · 11 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: transformers-ctc · PyTorch 2.13.0
Protocol: audio-native-v1
Dataset: asr_prompts
Workload: asr

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	2,662.33	—	—	1.64	122.1	1	3be7f67e ↗
B2	4,844.87	—	—	1.67	134.9	1	3be7f67e ↗
B4	6,237.58	—	—	1.75	148	1	3be7f67e ↗
B8	7,907.41	—	—	1.9	165.1	1	3be7f67e ↗
B16	9,083.29	—	—	2.33	194.8	1	3be7f67e ↗
B32	9,193.57	—	—	3.25	243.5	1	3be7f67e ↗
B64	8,622.98	—	—	5.09	345.5	1	3be7f67e ↗
B128	8,355.43	—	—	10.1	451	1	3be7f67e ↗
B256	8,343	—	—	20.09	477.6	1	3be7f67e ↗
B512	8,391.94	—	—	40.63	492.3	1	3be7f67e ↗
B1024	8,387.34	—	—	80.89	501.3	1	3be7f67e ↗

1 other test configurations on this GPU

Native · Audio → Text · asr_prompts

Throughput · Batch 1: 3,143audio-sec/sec
Best measured · Batch 16: 8,581audio-sec/sec
Observed GPU occupancy: 1.6GiB
P95 latency: Not recorded

Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: transformers-ctc · PyTorch 2.13.0
Protocol: audio-native-v1
Dataset: asr_prompts
Workload: asr

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	3,143.16	—	—	1.64	95.8	1	d77a0cfd ↗
B8	8,162.53	—	—	1.64	105.2	1	d77a0cfd ↗
B16	8,581.45	—	—	1.92	111.9	1	d77a0cfd ↗

Qwen3-TTS-12Hz-0.6B-CustomVoice
Qwen · 1 tested configuration
- Native · Not recorded → Audio · tts_prompts · Test duration 43 s
  Throughput · Batch 1
  1.6audio-sec/sec
  Best measured · Batch 4
  5.2audio-sec/sec
  Observed GPU occupancy
  3.3GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  qwen-tts · PyTorch 2.14.1
  Protocol
  audio-native-v1
  Dataset
  tts_prompts
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 1.6 — — 3.34 154.4 1 ebb272f8 ↗
  B4 5.22 — — 4.92 161 1 ebb272f8 ↗
magpie_tts_multilingual_357m
nvidia · 1 tested configuration
- Native · Not recorded → Audio · tts_prompts · Test duration 19 s
  Throughput · Batch 1
  3.6audio-sec/sec
  Observed GPU occupancy
  4GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  nemo-magpie · PyTorch 2.13.0
  Protocol
  audio-native-v1
  Dataset
  tts_prompts
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 3.57 — — 4.01 178.8 1 666efb0f ↗
Qwen3.8-Flash-Next-NVFP4-NVIDIA
Qwen/Qwen3.8-Flash-Next · 6 tested configurations
- Q4_K_XL · 8K → 2K tokensCPU + GPU hybrid
  Output throughput · includes prefill · 1 request
  125tok/s
  Observed GPU occupancy
  79.7GiB
  First token
  1.8 s
  Evidence report and run availability ↗Same test on other GPUs
  Test details
  Total output tokens divided by the full request window, including prefill and queueing.
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  strata
  Protocol
  wbench-fixed-http-v1
  Dataset
  synthetic-tokenizer-v1:repeats=3
  Workload
  8192in-2048out
  Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  C1 125.06 23.75 s 1.8 s 79.71 279.5 1 18c8631f ↗
- 5 other test configurations on this GPU
  - NVFP4 · 8K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    151tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    1.1 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 151.39 15.46 s 1.1 s 71.22 292.2 1 60a45fd1 ↗
  - NVFP4 · 8K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    135tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    1.3 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    8192in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 134.63 19.87 s 1.3 s 71.22 280 1 f2b52c15 ↗
  - NVFP4 · 32K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    114tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    4.8 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    32768in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 114.25 20.98 s 4.8 s 71.22 316.1 1 4342000e ↗
  - Q4_K_XL · 32K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    106tok/s
    Observed GPU occupancy
    79.7GiB
    First token
    6.7 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    32768in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 106.11 23.97 s 6.7 s 79.71 308 1 66369e97 ↗
  - NVFP4 · 32K → 2K tokensCPU + GPU hybrid
    Output throughput · includes prefill · 1 request
    124tok/s
    Observed GPU occupancy
    71.2GiB
    First token
    4.1 s
    Evidence report and run availability ↗Same test on other GPUs
    Test details
    Total output tokens divided by the full request window, including prefill and queueing.
    Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
    Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
    Engine
    strata
    Protocol
    wbench-fixed-http-v1
    Dataset
    synthetic-tokenizer-v1:repeats=3
    Workload
    32768in-2048out
    Concurrency Mean throughput (tok/s) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
    C1 124.44 18.26 s 4.1 s 71.22 319.5 1 4a41258c ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	1.6	—	—	3.34	154.4	1	ebb272f8 ↗
B4	5.22	—	—	4.92	161	1	ebb272f8 ↗

Qwen3-Embedding-0.6B

Qwen · 3 tested configurations

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s

Throughput · Batch 1: 232vectors/sec
Best measured · Batch 32: 3,763vectors/sec
Observed GPU occupancy: 2.2GiB
P95 latency: 5 ms

Use this recipe · Batch 32 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	232.18	5 ms	—	2.22	155	1	65594583 ↗
B8	1,601.21	5 ms	—	2.22	191.2	1	65594583 ↗
B32	3,763.07	9 ms	—	2.22	208.7	1	65594583 ↗

2 other test configurations on this GPU

fp16 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s

Throughput · Batch 1: 218vectors/sec
Best measured · Batch 32: 3,617vectors/sec
Observed GPU occupancy: 2.2GiB
P95 latency: 5 ms

Use this recipe · Batch 32 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	217.92	5 ms	—	2.22	156.5	1	a7d85bcf ↗
B8	1,494.89	5 ms	—	2.22	185.4	1	a7d85bcf ↗
B32	3,616.74	9 ms	—	2.22	206.8	1	a7d85bcf ↗

nf4 · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s

Throughput · Batch 1: 196vectors/sec
Best measured · Batch 32: 3,091vectors/sec
Observed GPU occupancy: 1.6GiB
P95 latency: 5 ms

Use this recipe · Batch 32 ↗Evidence report and run availability ↗Same test on other GPUs

Test details · 3 batch sizes

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

Engine: sentence-transformers · PyTorch 2.12.1+cu130
Protocol: native-forward-v1
Dataset: embedding-mteb-tasks-v1
Workload: embedding

Batch size	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	195.8	5 ms	—	1.61	174.8	1	b9501833 ↗
B8	1,156.9	7 ms	—	1.61	196	1	b9501833 ↗
B32	3,091.08	10 ms	—	1.61	207.4	1	b9501833 ↗

Kokoro-82M
hexgrad · 1 tested configuration
- fp32 · Text → Audio · kokoro-english-v1 · Test duration 6 s
  Throughput · Batch 1
  120audio-sec/sec
  Best measured · Batch 4
  125audio-sec/sec
  Observed GPU occupancy
  2GiB
  P95 latency
  29 ms
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  kokoro · PyTorch 2.13.0+cu130
  Protocol
  audio-native-v1
  Dataset
  kokoro-english-v1
  Workload
  tts
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 120.3 29 ms — 2 152.1 2 061d047a ↗
  B4 125.19 124 ms — 2.03 169.7 2 061d047a ↗
whisper-tiny
openai · 1 tested configuration
- fp16 · Audio → Text · asr_prompts · Test duration 2 s
  Throughput · Batch 1
  233audio-sec/sec
  Best measured · Batch 4
  364audio-sec/sec
  Observed GPU occupancy
  1.4GiB
  P95 latency
  Not recorded
  Evidence report and run availability ↗Same test on other GPUs
  Test details · 2 batch sizes
  Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.
  Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.
  Engine
  faster-whisper · PyTorch 2.14.1
  Protocol
  audio-native-v1
  Dataset
  asr_prompts
  Workload
  asr
  Batch size Mean throughput (audio-sec/sec) P95 TTFT Observed occupancy (GiB) GPU power (W) Reports combined View report
  B1 232.81 — — 1.42 113.5 2 ba5a45ac ↗
  B4 364.19 — — 1.43 115.8 2 ba5a45ac ↗

Sortable tables by exact test condition · 231 rows

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
StartLux-Decision-v1-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	startlux-nativePyTorch 2.11.0+cu130	50.25	20 ms	—	3.17	161.3	1	df141700 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
StartLux-Decision-v1-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	startlux-nativePyTorch 2.11.0+cu130	50.28	20 ms	—	2.29	165	1	8c1985ea ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0+cu130	50.83	20 ms	—	2.36	169	2	18b06f63 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	51.51	20 ms	—	2.6	164.2	1	7f2e07c0 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
StartLux-Decision-v1.1-0.8B-Instant	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	startlux-transformersPyTorch 2.11.0+cu130	18.69	54 ms	—	2.73	134.5	2	a4a69815 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
StartLux-Decision-v1.1-0.8B-Instant	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	startlux-transformersPyTorch 2.11.0	18.83	54 ms	—	2.54	125.5	1	c342a9ca ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	50.29	21 ms	—	4.9	216.2	1	e4bcd6a3 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0+cu130	47.71	338 ms	—	19.55	499.5	2	82d98b36 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 6 s · Batch 17 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0+cu130	346.72	50 ms	—	3.07	378	1	6ccb367c ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	31.83	32 ms	—	18.95	388.5	1	10c64aff ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	39.27	93 ms	—	18.95	449.5	1	10c64aff ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	5.41	8.21 s	—	19.28	233.4	1	10c64aff ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 33 s · Batch 64 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-flash	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	15.43	9.66 s	—	21.89	324.3	1	10c64aff ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-omni	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	7.45	134 ms	—	60.88	232.8	1	16597f91 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-omni	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	7.47	537 ms	—	60.89	249.7	1	16597f91 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-omni	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	7.48	2.14 s	—	60.89	252.8	1	16597f91 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 44 s · Batch 64 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef-omni	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	7.49	8.56 s	—	60.89	257.2	1	16597f91 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	12.32	82 ms	—	52.4	469.1	1	243b7f64 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	1.4	8.09 s	—	52.4	225.7	1	243b7f64 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	4.33	9.1 s	—	52.98	308.5	1	243b7f64 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 46 s · Batch 64 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
clef	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	clef-nativePyTorch 2.11.0	9.39	12.46 s	—	57.95	435.9	1	243b7f64 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	52.8	19 ms	—	2.36	172.8	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 2 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	95.72	21 ms	—	2.38	186.5	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	177.64	23 ms	—	2.43	242	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 8 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	306.42	26 ms	—	2.57	344.2	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 12 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	322.68	37 ms	—	37.46	371.9	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 13 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	336	39 ms	—	37.46	380.8	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 14 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	344.61	41 ms	—	37.46	392.6	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 15 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	345.75	44 ms	—	37.46	389.5	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	345.28	47 ms	—	2.85	383.9	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 17 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	349.83	49 ms	—	37.46	397.8	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 18 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	327.34	55 ms	—	37.46	383.4	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 20 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	333.21	60 ms	—	37.46	402.6	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 24 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	331.98	73 ms	—	37.46	401.2	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 28 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	327.01	86 ms	—	37.46	399.2	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 32 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	331.67	97 ms	—	3.41	397.7	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 48 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	326.46	148 ms	—	37.46	405.9	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 64 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	318.22	202 ms	—	4.51	403.2	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 128 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	298.77	430 ms	—	6.71	405.8	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 256 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	298.58	859 ms	—	11.11	407.3	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 512 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	299.67	1.71 s	—	19.9	411.2	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 396 s · Batch 1024 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	297.46	3.64 s	—	37.46	410.6	1	5eef633e ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 1 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	52.04	20 ms	—	2.36	162.6	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 2 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	96.21	21 ms	—	2.38	213.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 4 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	177.08	23 ms	—	2.43	243.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 8 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	299.06	27 ms	—	2.57	341.2	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 12 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	306.24	40 ms	—	36.98	349.6	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 14 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	322.36	43 ms	—	36.98	365.9	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 15 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	328.14	46 ms	—	36.98	369.8	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 16 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	333.92	48 ms	—	3.19	368.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 17 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	331.9	52 ms	—	36.98	379.5	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 18 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	310.84	58 ms	—	36.98	365.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 20 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	316.13	64 ms	—	36.98	375.1	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 24 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	312.87	77 ms	—	36.98	377.5	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 28 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	309.61	91 ms	—	36.98	379.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 32 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	319.44	101 ms	—	3.44	376.3	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 40 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	307.24	131 ms	—	36.98	379.6	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 48 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	306.75	159 ms	—	36.98	380.4	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 64 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	304.06	213 ms	—	4.2	383.7	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 128 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	285.17	453 ms	—	6.23	387.8	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 256 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	284.61	904 ms	—	10.63	390.3	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 512 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	282.22	2.04 s	—	19.42	390.1	1	156910e7 ↗

Decision · Text → Structured data · typed-invoices-v1 · Test duration 399 s · Batch 1024 · decisions/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (decisions/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Intern-Decision-0.8B	RTX PRO 6000 Blackwell Server Edition · 96 GB	bf16	intern-decision-nativePyTorch 2.11.0	286.53	3.58 s	—	36.98	396.2	1	156910e7 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	217.92	5 ms	—	2.22	156.5	1	a7d85bcf ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	195.8	5 ms	—	1.61	174.8	1	b9501833 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 8 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	1,494.89	5 ms	—	2.22	185.4	1	a7d85bcf ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 8 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	1,156.9	7 ms	—	1.61	196	1	b9501833 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 32 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	3,616.74	9 ms	—	2.22	206.8	1	a7d85bcf ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 15 s · Batch 32 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	3,091.08	10 ms	—	1.61	207.4	1	b9501833 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	189.23	6 ms	—	1.33	145.5	1	c673556b ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	206.79	5 ms	—	1.48	119.3	1	196678d3 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 1 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	232.18	5 ms	—	2.22	155	1	65594583 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	713.32	6 ms	—	1.33	142.8	1	c673556b ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 4 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	764.95	6 ms	—	1.48	141.4	1	196678d3 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 8 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	1,601.21	5 ms	—	2.22	191.2	1	65594583 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	nf4	sentence-transformersPyTorch 2.12.1+cu130	2,571.53	7 ms	—	1.33	162.7	1	c673556b ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 16 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
embeddinggemma-2	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	2,938.63	6 ms	—	1.48	161.8	1	196678d3 ↗

Embedding · Text → Vectors · embedding-mteb-tasks-v1 · Test duration 90 s · Batch 32 · vectors/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (vectors/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-Embedding-0.6B	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	sentence-transformersPyTorch 2.12.1+cu130	3,763.07	9 ms	—	2.22	208.7	1	65594583 ↗

Speech recognition · Audio → Text · asr_prompts · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	3,143.16	—	—	1.64	95.8	1	d77a0cfd ↗

Speech recognition · Audio → Text · asr_prompts · Batch 8 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,162.53	—	—	1.64	105.2	1	d77a0cfd ↗

Speech recognition · Audio → Text · asr_prompts · Batch 16 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,581.45	—	—	1.92	111.9	1	d77a0cfd ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 2 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	faster-whisperPyTorch 2.14.1	232.81	—	—	1.42	113.5	2	ba5a45ac ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 2 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
whisper-tiny	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp16	faster-whisperPyTorch 2.14.1	364.19	—	—	1.43	115.8	2	ba5a45ac ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	2,662.33	—	—	1.64	122.1	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 2 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	4,844.87	—	—	1.67	134.9	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	6,237.58	—	—	1.75	148	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 8 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	7,907.41	—	—	1.9	165.1	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 16 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	9,083.29	—	—	2.33	194.8	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 32 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	9,193.57	—	—	3.25	243.5	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 64 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,622.98	—	—	5.09	345.5	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 128 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,355.43	—	—	10.1	451	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 256 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,343	—	—	20.09	477.6	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 512 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,391.94	—	—	40.63	492.3	1	3be7f67e ↗

Speech recognition · Audio → Text · asr_prompts · Test duration 16 s · Batch 1024 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
granite-speech-5.0-470m-turboctc	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	transformers-ctcPyTorch 2.13.0	8,387.34	—	—	80.89	501.3	1	3be7f67e ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 9 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
IndexTTS-2.5	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	indexttsPyTorch 2.8.0+cu128	4.24	—	—	6.48	221.7	1	a46ec2c4 ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 19 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
magpie_tts_multilingual_357m	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	nemo-magpiePyTorch 2.13.0	3.57	—	—	4.01	178.8	1	666efb0f ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 43 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-TTS-12Hz-0.6B-CustomVoice	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	qwen-ttsPyTorch 2.14.1	1.6	—	—	3.34	154.4	1	ebb272f8 ↗

Speech synthesis · Not recorded → Audio · tts_prompts · Test duration 43 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-TTS-12Hz-0.6B-CustomVoice	RTX PRO 6000 Blackwell Server Edition · 96 GB	Native	qwen-ttsPyTorch 2.14.1	5.22	—	—	4.92	161	1	ebb272f8 ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 1 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp32	kokoroPyTorch 2.13.0+cu130	120.3	29 ms	—	2	152.1	2	061d047a ↗

Speech synthesis · Text → Audio · kokoro-english-v1 · Test duration 6 s · Batch 4 · audio-sec/sec · 1 rows
Model	Hardware	Format	Engine	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Kokoro-82M	RTX PRO 6000 Blackwell Server Edition · 96 GB	fp32	kokoroPyTorch 2.13.0+cu130	125.19	124 ms	—	2.03	169.7	2	061d047a ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	Q4_K_XL	strata	125.06	23.75 s	1.8 s	79.71	279.5	1	18c8631f ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	strata	151.39	15.46 s	1.1 s	71.22	292.2	1	60a45fd1 ↗

Text generation · 8K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	strata	134.63	19.87 s	1.3 s	71.22	280	1	f2b52c15 ↗

Text generation · 8K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-4B-AWQ	RTX PRO 6000 Blackwell Server Edition · 96 GB	AWQ	vllmvLLM 0.29.0	214.6	9.63 s	298 ms	86.28	303.8	1	86d36e4f ↗

Text generation · 8K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-4B-AWQ	RTX PRO 6000 Blackwell Server Edition · 96 GB	AWQ	vllmvLLM 0.29.0	516.94	16.14 s	872 ms	86.28	339.5	1	86d36e4f ↗

Text generation · 8K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-4B-AWQ	RTX PRO 6000 Blackwell Server Edition · 96 GB	AWQ	vllmvLLM 0.29.0	222.74	9.19 s	28 ms	86.28	288.8	1	adbfef8a ↗

Text generation · 8K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3-4B-AWQ	RTX PRO 6000 Blackwell Server Edition · 96 GB	AWQ	vllmvLLM 0.29.0	741.85	11.05 s	35 ms	86.28	335	1	adbfef8a ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	strata	114.25	20.98 s	4.8 s	71.22	316.1	1	4342000e ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	Q4_K_XL	strata	106.11	23.97 s	6.7 s	79.71	308	1	66369e97 ↗

Text generation · 32K → 2K tokens · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-Flash-Next-NVFP4-NVIDIA	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	strata	124.44	18.26 s	4.1 s	71.22	319.5	1	4a41258c ↗

Text generation · 32K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	153.44	13.39 s	753 ms	85.25	343.4	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	52.78	38.8 s	3.3 s	87.37	337.3	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	55.59	36.91 s	3.2 s	85.98	341	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	208.65	19.64 s	1.1 s	85.25	360.4	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	87.2	48.71 s	4.9 s	87.37	343.4	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	91.37	46.48 s	4.6 s	85.98	342.9	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	259.19	33.21 s	1.6 s	85.25	398.9	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	142.14	64.91 s	7 s	87.37	375.3	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	149.21	62.63 s	6.5 s	85.98	372.9	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	187.69	79.85 s	8.1 s	87.37	402.8	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	196.73	76.24 s	7.8 s	85.98	398.9	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	202.31	85.21 s	8.7 s	87.37	411.8	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	211.87	81.41 s	8.3 s	85.98	408.7	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	295.37	59.5 s	2.1 s	85.25	422	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	215.05	94.52 s	9.3 s	87.37	421.4	1	aee8b33e ↗

Text generation · 32K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	224.61	90.5 s	8.9 s	85.98	414.8	1	dca51ce5 ↗

Text generation · 32K → 2K tokens · No cache reuse · 12 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	311.33	86.14 s	2.7 s	85.25	432.4	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · No cache reuse · 16 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	319.17	112.4 s	3.3 s	85.25	438	1	d2914dd8 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	161.57	12.72 s	71 ms	85.25	327.8	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	56.96	36.17 s	222 ms	87.37	322.6	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	57.09	35.88 s	231 ms	86.63	319	1	37f29252 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	56.4	36.51 s	244 ms	86.35	319.1	1	791b887d ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	60.1	34.34 s	217 ms	85.98	323	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	59.68	34.59 s	230 ms	86.82	323.3	1	83d37469 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	60.47	33.87 s	228 ms	86.86	320.3	1	cbf1431c ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	269.43	15.21 s	86 ms	85.25	349.1	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	104.14	39.34 s	355 ms	87.37	305.9	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	109.3	37.48 s	349 ms	85.98	310.8	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	460.41	17.83 s	106 ms	85.25	427.3	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	198.42	41.3 s	704 ms	87.37	319.3	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	197.8	41.44 s	650 ms	86.63	330.2	1	37f29252 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	196.45	41.73 s	654 ms	86.35	315.5	1	791b887d ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	209.23	39.18 s	686 ms	85.98	324.8	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	207.31	39.54 s	664 ms	86.82	320	1	83d37469 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	207.86	39.44 s	636 ms	86.86	333.6	1	cbf1431c ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	291.12	42.23 s	1 s	87.37	342.1	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	306.94	40.06 s	992 ms	85.98	336	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	333.32	43.04 s	1.1 s	87.37	349.1	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	351.24	40.83 s	1.1 s	85.98	342.6	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	644.45	25.52 s	127 ms	85.25	486.9	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	374.04	43.82 s	1.3 s	87.37	353.2	1	bced0642 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	372.88	43.98 s	1.3 s	86.63	367.3	1	37f29252 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	370.68	44.24 s	1.3 s	86.36	348.1	1	791b887d ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	392.93	41.71 s	1.3 s	85.98	348.2	1	9cdf45b7 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	394.04	41.6 s	1.2 s	86.82	346.1	1	83d37469 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	394.52	41.6 s	1.3 s	86.86	361.6	1	cbf1431c ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 16 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	771.04	42.77 s	184 ms	85.25	533.9	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 17 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	812.79	43.14 s	195 ms	85.25	550.4	1	75198c63 ↗

Text generation · 32K → 2K tokens · Prefix cache reuse · 18 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	856.64	43.41 s	196 ms	85.25	568.9	1	75198c63 ↗

Text generation · 64K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	101.26	20.26 s	2.3 s	85.25	369.1	1	c5c9cf6f ↗

Text generation · 64K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	44.86	45.67 s	8.6 s	87.37	360.5	1	d2ff32ec ↗

Text generation · 64K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	47.15	43.45 s	8.4 s	85.98	364.4	1	e5a201fa ↗

Text generation · 64K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	123.11	33.41 s	3.6 s	85.25	404.8	1	c5c9cf6f ↗

Text generation · 64K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	67.59	63.21 s	12.7 s	87.37	379.1	1	d2ff32ec ↗

Text generation · 64K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	70.39	60.71 s	12.3 s	85.98	381.3	1	e5a201fa ↗

Text generation · 64K → 2K tokens · No cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	84.95	84.07 s	14.6 s	87.37	398.5	1	d2ff32ec ↗

Text generation · 64K → 2K tokens · No cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	88.36	80.85 s	14.2 s	85.98	403.6	1	e5a201fa ↗

Text generation · 64K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	140.19	64.26 s	4.5 s	85.25	430	1	c5c9cf6f ↗

Text generation · 64K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	96.77	105.62 s	16.3 s	87.37	415.8	1	d2ff32ec ↗

Text generation · 64K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	100.66	101.65 s	15.8 s	85.98	419.2	1	e5a201fa ↗

Text generation · 64K → 2K tokens · No cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	142.08	97.79 s	5.3 s	85.25	430.4	1	c5c9cf6f ↗

Text generation · 64K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	150.73	122.37 s	6.3 s	85.25	446.7	1	c5c9cf6f ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	113.51	18.06 s	135 ms	85.25	341.3	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	54.72	37.56 s	329 ms	87.37	322.4	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	54.04	38.07 s	361 ms	86.36	320.7	1	417f69be ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	54.54	37.55 s	359 ms	86.63	324.9	1	7d1a7c8e ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	57.45	35.8 s	326 ms	85.98	326.1	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	57.08	36.15 s	358 ms	86.82	326	1	275a096d ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	57.67	35.51 s	357 ms	86.86	325.8	1	fe8141d2 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	153.3	26.73 s	161 ms	85.25	369.5	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	99.95	40.99 s	522 ms	87.37	315.9	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	104.7	39.14 s	515 ms	85.98	321	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	281.86	29.1 s	202 ms	85.25	483.8	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	189.07	43.34 s	971 ms	87.37	337.8	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	187.12	43.81 s	879 ms	86.36	333.2	1	417f69be ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	188.48	43.48 s	878 ms	86.63	350.2	1	7d1a7c8e ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	198.72	41.24 s	954 ms	85.98	344.2	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	196.91	41.63 s	870 ms	86.83	338.7	1	275a096d ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	197.42	41.52 s	870 ms	86.86	354.3	1	fe8141d2 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	293.74	41.93 s	214 ms	85.25	459.2	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	264.7	46.44 s	1.3 s	87.37	364	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 6 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	277.65	44.28 s	1.3 s	85.98	358.8	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	340.1	42.27 s	227 ms	85.25	496.1	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	303.08	47.32 s	1.4 s	87.37	373.4	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 7 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	317.41	45.18 s	1.4 s	85.98	369.2	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	387.93	42.41 s	237 ms	85.25	534.9	1	5fee1315 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	339.61	48.27 s	1.7 s	87.37	381.2	1	d3f6b770 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	336.95	48.66 s	1.6 s	86.36	379.7	1	417f69be ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	338.65	48.43 s	1.6 s	86.63	396.9	1	7d1a7c8e ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	354.95	46.19 s	1.6 s	85.98	377.1	1	67a5e6d3 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	355.89	46.08 s	1.6 s	86.83	375.7	1	275a096d ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	356.25	46.05 s	1.6 s	86.86	392.4	1	fe8141d2 ↗

Text generation · 64K → 2K tokens · Prefix cache reuse · 9 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	433.07	42.78 s	246 ms	85.25	575.8	1	5fee1315 ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	56.09	36.63 s	8 s	85.25	418.9	1	2c661c7e ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	31.54	64.93 s	24.9 s	87.37	402.7	1	73fcb7c1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	31.45	65.12 s	24.9 s	86.63	415.7	1	448b0777 ↗

Text generation · 128K → 2K tokens · No cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	32.72	62.59 s	24.3 s	85.99	406.5	1	28f51cf1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	62.52	67.49 s	12 s	85.25	444.3	1	2c661c7e ↗

Text generation · 128K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	41.27	111.14 s	34 s	87.37	426.3	1	73fcb7c1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	42.54	106.24 s	33.5 s	85.99	428.9	1	28f51cf1 ↗

Text generation · 128K → 2K tokens · No cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	63.59	109.31 s	13.5 s	85.25	446.8	1	2c661c7e ↗

Text generation · 128K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	66.75	144.63 s	15 s	85.25	461.6	1	2c661c7e ↗

Text generation · 128K → 2K tokens · No cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	50.67	220.6 s	45.8 s	86.63	469.4	1	448b0777 ↗

Text generation · 128K → 2K tokens · No cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	57.74	421.65 s	61.9 s	86.63	494.5	1	448b0777 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	71.05	28.83 s	268 ms	85.25	356.6	1	1ff0c98a ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	49.98	40.98 s	506 ms	86.36	322.7	1	68d1f2c6 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	52.97	38.72 s	435 ms	85.99	333.9	1	78fc908c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	52.9	38.72 s	508 ms	86.86	338	1	7cea351c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 1 request · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	52.59	38.95 s	509 ms	86.83	328.8	1	d14962c1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	89.43	45.82 s	323 ms	85.25	399.6	1	1ff0c98a ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 2 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	97.25	42.14 s	646 ms	85.99	340.8	1	78fc908c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	135.43	45.41 s	379 ms	85.25	470.9	1	1ff0c98a ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 3 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	138.66	44.36 s	822 ms	85.99	358.8	1	78fc908c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
WeVisOne-VL-2B	RTX PRO 6000 Blackwell Server Edition · 96 GB	BF16	vllmvLLM 0.28.0	165.29	49.61 s	420 ms	85.25	524	1	1ff0c98a ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	171.33	47.85 s	1 s	86.36	365.5	1	68d1f2c6 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	181.16	45.25 s	1.1 s	85.99	375.3	1	78fc908c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	179.81	45.65 s	1 s	86.86	387.5	1	7cea351c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 4 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	179.42	45.72 s	1.1 s	86.83	371.9	1	d14962c1 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	287.31	57.12 s	1.9 s	86.36	425.4	1	68d1f2c6 ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	301.16	54.48 s	1.9 s	86.86	443.7	1	7cea351c ↗

Text generation · 128K → 2K tokens · Prefix cache reuse · 8 concurrent requests · tok/s · 1 rows
Model	Hardware	Format	Engine	Mean throughput (tok/s)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
Qwen3.8-27B-NVFP4	RTX PRO 6000 Blackwell Server Edition · 96 GB	NVFP4	vllmvLLM 0.28.0	300.98	54.52 s	1.9 s	86.83	426	1	d14962c1 ↗

Models with reports but no qualified results · 40

Their original reports remain available on each model page. They did not qualify for comparison; each report states why.

Models with reports but no qualified results · 40 rows
Model	Test task
DeepSeek-R1-Distill-Qwen-1.5Bdeepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B	Text generation
Llama-3.1-8B-Instruct-NVFP4nvidia/Llama-3.1-8B-Instruct-NVFP4	Text generation
NVIDIA-Nemotron-Nano-9B-v2-NVFP4nvidia/NVIDIA-Nemotron-Nano-9B-v2-NVFP4	Text generation
NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QADnvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-NVFP4-QAD	Text generation
Qwen2.5-0.5Bqwen2.5:0.5b	Text generation
qwen2.5-0.5b-instruct-q4_k_m.ggufQwen/Qwen2.5-0.5B-Instruct-GGUF/qwen2.5-0.5b-instruct-q4_k_m.gguf	Text generation
Qwen2.5-1.5Bqwen2.5:1.5b	Text generation
qwen2.5-1.5b-instruct-q4_k_m.ggufQwen/Qwen2.5-1.5B-Instruct-GGUF/qwen2.5-1.5b-instruct-q4_k_m.gguf	Text generation
Qwen2.5-3Bqwen2.5:3b	Text generation
Qwen2.5-3B-InstructQwen/Qwen2.5-3B-Instruct	Text generation
qwen2.5-3b-instruct-q4_k_m.ggufQwen/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_k_m.gguf	Text generation
Qwen2.5-7Bqwen2.5:7b	Text generation
Qwen2.5-72B-AWQQwen/Qwen2.5-72B-Instruct-AWQ	Text generation
Qwen3-0.6BQwen/Qwen3-0.6B	Text generation
Qwen3-0.6B-FP8Qwen/Qwen3-0.6B-FP8	Text generation
Qwen3-0.6B-GPTQ-Int8Qwen/Qwen3-0.6B-GPTQ-Int8	Text generation
Qwen3-0.6B-Q8Qwen/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q8_0.gguf	Text generation
Qwen3-1.7BQwen/Qwen3-1.7B	Text generation
Qwen3-1.7B-FP8Qwen/Qwen3-1.7B-FP8	Text generation
Qwen3-1.7B-Q8Qwen/Qwen3-1.7B-GGUF/Qwen3-1.7B-Q8_0.gguf	Text generation
Qwen3-4BQwen/Qwen3-4B	Text generation
Qwen3-4B-FP8Qwen/Qwen3-4B-FP8	Text generation
Qwen3-4B-Q4Qwen/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf	Text generation
Qwen3-4B-Q8_0Qwen/Qwen3-4B-GGUF/Qwen3-4B-Q8_0.gguf	Text generation
Qwen3-8BQwen/Qwen3-8B	Text generation
Qwen3-8B-AWQQwen/Qwen3-8B-AWQ	Text generation
Qwen3-8B-NVFP4RedHatAI/Qwen3-8B-NVFP4	Text generation
Qwen3-8B-Q4Qwen/Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf	Text generation
Qwen3-14B-AWQQwen/Qwen3-14B-AWQ	Text generation
Qwen3-14B-Q4Qwen/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.gguf	Text generation
Qwen3-32B-FP8Qwen/Qwen3-32B-FP8	Text generation
Qwen3-32B-Q4Qwen/Qwen3-32B-GGUF/Qwen3-32B-Q4_K_M.gguf	Text generation
Qwen3-VL-2B-Instruct-FP8Qwen/Qwen3-VL-2B-Instruct-FP8	Text generation
Qwen3-VL-4B-Instruct-FP8Qwen/Qwen3-VL-4B-Instruct-FP8	Text generation
Qwen3-VL-30B-A3B-FP8Qwen/Qwen3-VL-30B-A3B-Instruct-FP8	Text generation
Qwen3-VL-32B-FP8Qwen/Qwen3-VL-32B-Instruct-FP8	Text generation
Qwen3.5-0.8BQwen/Qwen3.5-0.8B	Text generation
Qwen3.6-35B-A3B-AWQ-4bitcyankiwi/Qwen3.6-35B-A3B-AWQ-4bit	Text generation
Qwen3.6-35B-A3B-FP8Qwen/Qwen3.6-35B-A3B-FP8	Text generation
Qwen3.8-27B-NVFP4hf:nvidia/Qwen3.8-27B-NVFP4	Text generation

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	120.3	29 ms	—	2	152.1	2	061d047a ↗
B4	125.19	124 ms	—	2.03	169.7	2	061d047a ↗

Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
B1	232.81	—	—	1.42	113.5	2	ba5a45ac ↗
B4	364.19	—	—	1.43	115.8	2	ba5a45ac ↗