GPU Acceleration Guide¶

Complete guide to using GPU acceleration with SLM Packager across different hardware platforms.

Overview¶

SLM Packager supports GPU acceleration on: - Apple Silicon (M1/M2/M3) via MPS - Zero setup! ⚡ - NVIDIA GPUs via CUDA - Requires setup - Apple Silicon (GGUF models) via Metal - Requires rebuild

Quick Reference¶

Platform	Runtime	Device	Setup Required	Speed Boost
Mac (M1/M2/M3)	transformers	`mps`	❌ None!	~2-3x
Mac (M1/M2/M3)	llama.cpp	Metal	✅ Rebuild	~2-4x
NVIDIA GPU	llama.cpp	`cuda`	✅ Rebuild	~3-5x
NVIDIA GPU	transformers	`cuda`	❌ Auto	~3-5x
NVIDIA GPU	onnx	`cuda`	✅ Package	~3-5x

Apple Silicon (MPS) - Recommended for Mac Users 🍎¶

Overview¶

Metal Performance Shaders (MPS) provides GPU acceleration for PyTorch models on Apple Silicon with zero setup required.

Benefits: - ✅ No installation or configuration needed - ✅ Works immediately on M1/M2/M3 Macs - ✅ 2-3x speedup vs CPU - ✅ Same memory usage as CPU

Real Performance (M2 Pro)¶

GPT-2 (124M parameters)
├─ CPU:  1.3 tokens/sec  | 2.1GB RAM
└─ MPS:  2.4 tokens/sec  | 2.1GB RAM  ⚡ 2.14x faster!

TinyLlama (1.1B parameters)  
├─ CPU: 12 tokens/sec  | 1.8GB RAM
└─ MPS: 28 tokens/sec  | 1.8GB RAM  ⚡ 2.3x faster!

Usage¶

Option 1: Create New Config¶

slm init \
  --name gpt2 \
  --path gpt2 \
  --format pytorch \
  --runtime transformers \
  --device mps \
  --output gpt2-gpu.yaml

slm run gpt2-gpu.yaml --prompt "Explain quantum computing"

Option 2: Edit Existing Config¶

model:
  name: "gpt2"
  path: "gpt2"
  format: "transformers"

runtime:
  type: "transformers"
  device: "mps"  # ← Change from "cpu" to "mps"
  threads: 4
  context_size: 1024

params:
  temperature: 0.8
  max_tokens: 100

Requirements¶

macOS: 12.3 (Monterey) or later
Hardware: Apple Silicon (M1, M2, M3, and Pro/Max/Ultra variants)
PyTorch: 1.12 or later (automatically installed)

Troubleshooting¶

Error: "MPS backend not available"

Check your setup:

import torch
print(f"PyTorch version: {torch.__version__}")
print(f"MPS available: {torch.backends.mps.is_available()}")

Requirements: - PyTorch ≥ 1.12 - macOS ≥ 12.3 - Apple Silicon (not Intel)

Solution if missing:

pip install --upgrade torch

NVIDIA CUDA - For Maximum Performance 🚀¶

Overview¶

CUDA provides the fastest inference on NVIDIA GPUs with 3-5x speedup vs CPU.

Setup Methods¶

Method 1: llama.cpp with CUDA (GGUF models)¶

Best for: Quantized models, maximum efficiency

Installation:

# Reinstall llama-cpp-python with CUDA support
CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python --no-cache-dir

Configuration:

model:
  name: "tinyllama"
  path: "./models/tinyllama.Q4_K_M.gguf"
  format: "gguf"

runtime:
  type: "llama_cpp"
  device: "cuda"  # Enable CUDA
  gpu_layers: 32  # Offload 32 layers to GPU (adjust based on VRAM)
  threads: 4
  context_size: 2048

GPU Layers Guide: - Small models (1-3B): gpu_layers: 32 (all layers) - Medium models (7B): gpu_layers: 32-40 - Large models (13B+): Adjust based on VRAM (each layer ≈ 100-200MB)

Method 2: transformers with CUDA (PyTorch models)¶

Best for: Latest models, fine-tuning workflows

Installation:

# Install PyTorch with CUDA (if not already installed)
pip install torch --index-url https://download.pytorch.org/whl/cu118

# Verify CUDA is available
python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}')"

Configuration:

model:
  name: "gpt2"
  path: "gpt2"
  format: "transformers"

runtime:
  type: "transformers"
  device: "cuda"  # Auto device mapping
  threads: 0      # Not used with CUDA

Method 3: ONNX with CUDA¶

Best for: Optimized inference, cross-platform deployment

Installation:

# Install ONNX Runtime with CUDA
pip install onnxruntime-gpu

Configuration:

model:
  name: "gpt2"
  path: "./models/gpt2-onnx"
  format: "onnx"

runtime:
  type: "onnx"
  device: "cuda"
  threads: 4

CUDA Troubleshooting¶

"CUDA not available" with PyTorch

Check CUDA version:
```
nvidia-smi
```

Install matching PyTorch:

# For CUDA 11.8
pip install torch --index-url https://download.pytorch.org/whl/cu118

# For CUDA 12.1
pip install torch --index-url https://download.pytorch.org/whl/cu121

Out of Memory (OOM) errors

Solutions: 1. Reduce gpu_layers for llama.cpp 2. Use smaller batch size 3. Use quantized models (Q4_K_M

) 4. Clear GPU cache: torch.cuda.empty_cache()

Performance not improving

Check: 1. GPU is actually being used: nvidia-smi (should show GPU usage) 2. Enough layers offloaded: gpu_layers should be > 0 3. Model fits in VRAM: Check nvidia-smi memory usage

Metal (Apple Silicon + GGUF) - Maximum Speed on Mac 🏎️¶

Overview¶

Metal provides the fastest inference on Apple Silicon when using GGUF models with llama.cpp.

Performance: Up to 4x faster than CPU!

Installation¶

# Rebuild llama-cpp-python with Metal support
CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python --no-cache-dir

Configuration¶

model:
  name: "tinyllama"
  path: "./models/tinyllama.Q4_K_M.gguf"
  format: "gguf"

runtime:
  type: "llama_cpp"
  device: "cpu"  # Metal auto-detected
  gpu_layers: 32  # Offload to GPU via Metal
  threads: 4

Verification¶

Check if Metal is available:

python -c "import llama_cpp; print('Metal support:', hasattr(llama_cpp.llama_cpp, 'LLAMA_SUPPORTS_GPU_OFFLOAD'))"

Troubleshooting¶

Metal not working

Rebuild with Metal:

pip uninstall llama-cpp-python -y
CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python --no-cache-dir --force-reinstall

Check build output for "Metal: ON"

Slower than expected

Increase gpu_layers (try 32 or all layers)
Ensure model is GGUF format
Use Q4_K_M quantization for best speed/quality balance

Performance Comparison Table¶

Real-world benchmarks on different hardware:

GPT-2 (124M parameters)¶

Platform	Runtime	Config	Tokens/sec	Memory
M2 Pro	transformers CPU	`device: cpu`	1.3	2.1GB
M2 Pro	transformers MPS	`device: mps`	2.4	2.1GB
M2 Pro	onnx CPU	`device: cpu`	13.8	600MB
RTX 3080	transformers CUDA	`device: cuda`	~40-60	1GB VRAM
RTX 3080	llama.cpp CUDA	`gpu_layers: 32`	~80-120	400MB VRAM

TinyLlama (1.1B parameters)¶

Platform	Runtime	Config	Tokens/sec	Memory
M2 Pro	llama.cpp CPU	Q4_K_M	15-20	800MB
M2 Pro	llama.cpp Metal	`gpu_layers: 32`	40-60	800MB
M2 Pro	transformers MPS	`device: mps`	28	1.8GB
RTX 3080	llama.cpp CUDA	`gpu_layers: 32`	100-150	600MB VRAM

Benchmarks collected December 2025. Performance varies by hardware, model, and config.

Best Practices¶

Choose the Right Runtime¶

For Apple Silicon (M1/M2/M3): 1. MPS with transformers - Easiest, zero setup, all models 2. Metal with llama.cpp - Fastest, requires GGUF models

For NVIDIA GPUs: 1. CUDA with llama.cpp - Maximum efficiency, quantized models 2. CUDA with transformers - Latest models, fine-tuning

Optimize GPU Usage¶

llama.cpp (GGUF): - Set gpu_layers to offload computation - Start with 32 and adjust based on VRAM - Monitor with nvidia-smi or Activity Monitor

transformers (PyTorch): - Use device: cuda or device: mps - Model automatically placed on GPU - Use FP16 for faster inference (automatic)

ONNX: - Use CUDA execution provider for NVIDIA - Enable with device: cuda

Memory Management¶

Prevent OOM: - Use quantized models (Q4_K_M for GGUF) - Reduce context_size if needed - Lower gpu_layers to use less VRAM - Close other GPU applications

Clear GPU cache:

# PyTorch (transformers, ONNX)
import torch
if torch.cuda.is_available():
    torch.cuda.empty_cache()
if torch.backends.mps.is_available():
    torch.mps.empty_cache()

Quick Decision Guide¶

I have an Apple Silicon Mac (M1/M2/M3): - ✅ Use MPS with transformers (zero setup!) - ✅ Or rebuild llama.cpp with Metal for GGUF models - ❌ Don't use CUDA (not available)

I have an NVIDIA GPU: - ✅ Use CUDA with llama.cpp for quantized models - ✅ Use CUDA with transformers for latest models - ❌ Don't use MPS (not available)

I only have CPU: - ✅ Use GGUF models with llama.cpp (fastest CPU inference) - ✅ Use ONNX for optimized models - ✅ Increase threads to match CPU cores

I want maximum speed: - Apple Silicon: Metal + GGUF (llama.cpp) - NVIDIA: CUDA + GGUF (llama.cpp)
- Any platform: Use quantized models (Q4_K_M)

Additional Resources¶

GGUF Guide - llama.cpp with Metal/CUDA setup
ONNX Guide - ONNX runtime with GPU
Model Formats - Choosing the right format

Questions? Open an issue on GitHub or check the troubleshooting sections above.

Happy accelerating! 🚀