Discover how to troubleshoot and resolve slow performance issues in Ollama setups, particularly when models are not utilizing GPUs effectively. Slow LLM performance often occurs when insufficient VRAM forces computations to fallback on CPUs. This guide will help you identify causes, optimize configurations, and improve performance.
Understanding the Problem: Ollama Slow and CPU Usage
When running large language models (LLMs) like Ollama locally, users might experience significant performance slowdowns, with token generation rates dropping to 100 tokens per second or less.
Root causes:
- GPU is not being utilized effectively, resulting in CPU fallback.
- Model sizes exceeding available VRAM trigger silent "spillovers" into system RAM.
This fallback to the CPU is the primary bottleneck as it significantly increases inference latency.
VRAM Requirements for Large Language Models
Efficient LLM performance heavily depends on ensuring sufficient GPU VRAM for your model. Here's a quick reference table:
comparison
Model Size
- Small Models (3B parameters): ~4 GB VRAM
- Medium Models (13B parameters): ~12 GB VRAM
- Large Models (70B parameters): ~48 GB VRAM
Impact of VRAM Shortage
When VRAM is insufficient, the model will offload computations to the CPU, causing token generation rates to plummet. This scenario is particularly damaging for tasks requiring high throughput or low latency.
Fixing Local LLM Not Using GPU
Follow these steps to troubleshoot and resolve GPU utilization issues in your Ollama setup.
steps
Verify GPU Availability: Ensure your system has a compatible GPU meeting the VRAM requirements for your model. Run the following command to check:
bashnvidia-smiConfirm that your GPU is detected and has sufficient memory available.
Ensure Proper Llama CPP or Llama Server Setup: Double-check that the LLM software is configured to use the GPU. For Llama CPP, use the
--use-gpuflag when starting the model inference server.Increase Concurrency and Parallelism: Implement multiple Llama Server instances or adjust parameters such as concurrency for better resource utilization:
bashpython run_llama_tests.py --concurrency 128 --parallelism 64Use Llama Throughput Lab: Test different configurations to find the optimal setup for your hardware. This tool allows sweeps across GPUs, instances, and parameters, automating the optimization process.
Common Pitfalls and Solutions
When troubleshooting, be aware of common errors that may lead to suboptimal performance.
Using Llama Throughput Lab for Optimization
Llama Throughput Lab is an excellent tool for optimizing GPU and model configurations. It systematically tests multiple parameters, helping identify the most efficient setup for your hardware.
steps
Download and Install the Tool: Clone the repository and install its dependencies:
bashgit clone https://github.com/example/llama-throughput-lab.git cd llama-throughput-lab pip install -r requirements.txtConduct Parameter Sweeps: Run a "Full Sweep" to test combinations of concurrency, parallelism, and instances:
bashpython run_llama_tests.py --test full_sweepAnalyze Results: Review throughput metrics in the
/resultsfolder. Look for configurations that deliver maximum tokens per second:bashcat results/full_sweep_summary.txtApply Recommended Configurations: Re-run your server with the optimized settings.
Takeaway
Optimizing your Ollama setup for GPU usage can significantly improve performance and throughput. To achieve this:
- Confirm your GPU meets the VRAM requirements for your chosen model.
- Avoid CPU fallback by monitoring and adjusting VRAM and concurrency settings.
- Use tools like Llama Throughput Lab to fine-tune your setup.
With these adjustments, slow performance related to "local LLM not using GPU" can be efficiently addressed.
FAQ
Why is my Ollama setup so slow?
The most common reason is insufficient GPU VRAM, which forces the model to fallback on CPU for computations. This dramatically reduces token generation speeds.
How much VRAM does a 70B parameter model require?
A 70B parameter model typically requires at least 48 GB of VRAM. Ensure your GPU supports this requirement to avoid CPU fallback.
What tools can I use to optimize GPU performance in Ollama?
Llama Throughput Lab is an open-source tool that tests various configurations (e.g., concurrency, parallelism) to maximize GPU utilization and token throughput.
Can I run Ollama on a system without a dedicated GPU?
Yes, but performance will be significantly reduced, as the CPU is slower for LLM inference. For production use, a GPU is highly recommended.
Official reference: Ollama documentation.