Run an AI model on your computer

1. granite-speech-5.0-470m-turboctc

   ibm-granite · 2 tested configurations · 1 GPU

   • RTX PRO 6000 Blackwell Server Edition · 96 GBNative · Audio → Text · asr_prompts · Test duration 16 s

     Throughput · Batch 1

     2,662audio-sec/sec

     Best measured · Batch 32

     9,194audio-sec/sec

     Observed GPU occupancy

     1.6GiB

     P95 latency

     Not recorded

     Evidence report and run availability ↗Same test on other GPUs
     Test details · 11 batch sizes

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     transformers-ctc · PyTorch 2.13.0

     Protocol

     audio-native-v1

     Dataset

     asr_prompts

     Workload

     asr

     Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     B1	2,662.33	—	—	1.64	122.1	1	3be7f67e ↗
     B2	4,844.87	—	—	1.67	134.9	1	3be7f67e ↗
     B4	6,237.58	—	—	1.75	148	1	3be7f67e ↗
     B8	7,907.41	—	—	1.9	165.1	1	3be7f67e ↗
     B16	9,083.29	—	—	2.33	194.8	1	3be7f67e ↗
     B32	9,193.57	—	—	3.25	243.5	1	3be7f67e ↗
     B64	8,622.98	—	—	5.09	345.5	1	3be7f67e ↗
     B128	8,355.43	—	—	10.1	451	1	3be7f67e ↗
     B256	8,343	—	—	20.09	477.6	1	3be7f67e ↗
     B512	8,391.94	—	—	40.63	492.3	1	3be7f67e ↗
     B1024	8,387.34	—	—	80.89	501.3	1	3be7f67e ↗

   • 1 other test configurations on this GPU

     • Native · Audio → Text · asr_prompts

       Throughput · Batch 1

       3,143audio-sec/sec

       Best measured · Batch 16

       8,581audio-sec/sec

       Observed GPU occupancy

       1.6GiB

       P95 latency

       Not recorded

       Evidence report and run availability ↗Same test on other GPUs
       Test details · 3 batch sizes

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       transformers-ctc · PyTorch 2.13.0

       Protocol

       audio-native-v1

       Dataset

       asr_prompts

       Workload

       asr

       Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       B1	3,143.16	—	—	1.64	95.8	1	d77a0cfd ↗
       B8	8,162.53	—	—	1.64	105.2	1	d77a0cfd ↗
       B16	8,581.45	—	—	1.92	111.9	1	d77a0cfd ↗

2. whisper-tiny

   openai · 7 tested configurations · 4 GPUs

   • RTX 3080 Ti · 12 GBfp16 · Audio → Text · asr_prompts · Test duration 2 s

     Throughput · Batch 1

     65.7audio-sec/sec

     Best measured · Batch 4

     91.8audio-sec/sec

     Observed GPU occupancy

     0.4GiB

     P95 latency

     Not recorded

     Evidence report and run availability ↗Same test on other GPUs
     Test details · 2 batch sizes

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     faster-whisper · PyTorch 2.12.1+cu130

     Protocol

     audio-native-v1

     Dataset

     asr_prompts

     Workload

     asr

     Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     B1	65.65	—	—	0.41	134.9	1	3f8b97e6 ↗
     B4	91.84	—	—	0.4	143.6	1	3f8b97e6 ↗

   • RTX PRO 6000 Blackwell Server Edition · 96 GBfp16 · Audio → Text · asr_prompts · Test duration 2 s

     Throughput · Batch 1

     233audio-sec/sec

     Best measured · Batch 4

     364audio-sec/sec

     Observed GPU occupancy

     1.4GiB

     P95 latency

     Not recorded

     Evidence report and run availability ↗Same test on other GPUs
     Test details · 2 batch sizes

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     faster-whisper · PyTorch 2.14.1

     Protocol

     audio-native-v1

     Dataset

     asr_prompts

     Workload

     asr

     Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     B1	232.81	—	—	1.42	113.5	2	ba5a45ac ↗
     B4	364.19	—	—	1.43	115.8	2	ba5a45ac ↗

   • V100-PCIE-32GB · 32 GBfp16 · Audio → Text · asr_prompts · Test duration 1 s

     Throughput · Batch 1

     105audio-sec/sec

     Best measured · Batch 4

     156audio-sec/sec

     Observed GPU occupancy

     0.5GiB

     P95 latency

     Not recorded

     Evidence report and run availability ↗Same test on other GPUs
     Test details · 2 batch sizes

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     faster-whisper · PyTorch 2.12.1+cu126

     Protocol

     audio-native-v1

     Dataset

     asr_prompts

     Workload

     asr

     Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     B1	104.96	—	—	0.51	51.5	1	64035c7b ↗
     B4	155.93	—	—	0.52	50.1	1	64035c7b ↗

   • 1 other test configurations on this GPU

     • fp16 · Audio → Text · asr_prompts · Test duration 5 s

       Throughput · Batch 1

       90.9audio-sec/sec

       Best measured · Batch 4

       132audio-sec/sec

       Observed GPU occupancy

       0.5GiB

       P95 latency

       Not recorded

       Evidence report and run availability ↗Same test on other GPUs
       Test details · 2 batch sizes

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       faster-whisper · PyTorch 2.12.1+cu126

       Protocol

       audio-native-v1

       Dataset

       asr_prompts

       Workload

       asr

       Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       B1	90.94	—	—	0.51	51.5	10	e4ef00c1 ↗
       B4	132.16	—	—	0.52	51.2	10	e4ef00c1 ↗

   Show 1 more GPUs

   • vGPU-32GB · 32 GBfp16 · Audio → Text · asr_prompts

     Throughput · Batch 1

     73.7audio-sec/sec

     Observed GPU occupancy

     0.4GiB

     P95 latency

     Not recorded

     Evidence report and run availability ↗Same test on other GPUs
     Test details · 2 batch sizes

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     faster-whisper · PyTorch 2.13.0+cu130

     Protocol

     audio-native-v1

     Dataset

     asr_prompts

     Workload

     asr

     Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     B1	73.68	—	—	0.38	46.8	1	7d182de2 ↗
     B4	68.2	—	—	0.38	49.2	1	7d182de2 ↗

   • 2 other test configurations on this GPU

     • fp16 · Audio → Text · asr_prompts · Test duration 4 s

       Throughput · Batch 1

       122audio-sec/sec

       Best measured · Batch 4

       168audio-sec/sec

       Observed GPU occupancy

       0.4GiB

       P95 latency

       Not recorded

       Evidence report and run availability ↗Same test on other GPUs
       Test details · 2 batch sizes

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       faster-whisper

       Protocol

       audio-native-v1

       Dataset

       asr_prompts

       Workload

       asr

       Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       B1	121.9	—	—	0.42	73.6	1	7aaa1f0d ↗
       B4	168.25	—	—	0.42	77.4	1	7aaa1f0d ↗

     • fp16 · Audio → Text · asr_prompts · Test duration 1 s

       Throughput · Batch 1

       137audio-sec/sec

       Best measured · Batch 4

       205audio-sec/sec

       Observed GPU occupancy

       0.4GiB

       P95 latency

       Not recorded

       Evidence report and run availability ↗Same test on other GPUs
       Test details · 2 batch sizes

       Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

       Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

       Engine

       faster-whisper · PyTorch 2.13.0+cu130

       Protocol

       audio-native-v1

       Dataset

       asr_prompts

       Workload

       asr

       Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
       B1	137.01	—	—	0.38	54.3	1	4b83f25d ↗
       B4	205.27	—	—	0.38	66.3	1	4b83f25d ↗

3. SenseVoiceSmall

   FunAudioLLM · 1 tested configuration · 1 GPU

   • RTX 3080 Ti · 12 GBfp16 · Audio → Text · asr_prompts

     Throughput · Batch 1

     360audio-sec/sec

     Best measured · Batch 2

     365audio-sec/sec

     Observed GPU occupancy

     1.3GiB

     P95 latency

     Not recorded

     Evidence report and run availability ↗Same test on other GPUs
     Test details · 2 batch sizes

     Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.

     Reports combined counts reports, not inference requests. Request successes and failures are listed in each report.

     Engine

     funasr · PyTorch 2.12.1+cu130

     Protocol

     audio-native-v1

     Dataset

     asr_prompts

     Workload

     asr

     Batch size	Mean throughput (audio-sec/sec)	P95	TTFT	Observed occupancy (GiB)	GPU power (W)	Reports combined	View report
     B1	359.69	—	—	1.27	101.7	1	c6b5f31d ↗
     B2	364.87	—	—	1.27	101.6	1	c6b5f31d ↗

Models shown: 3 / 3

Recipe availability is checked on each report against the published recipe executor.

Occupancy includes engine buffers and reserved KV cache. It is not the minimum VRAM required by the model.