1# Token generation performance troubleshooting
 2
 3## Verifying that the model is running on the GPU with CUDA
 4Make sure you compiled llama with the correct env variables according to [this guide](/docs/build.md#cuda), so that llama accepts the `-ngl N` (or `--n-gpu-layers N`) flag. When running llama, you may configure `N` to be very large, and llama will offload the maximum possible number of layers to the GPU, even if it's less than the number you configured. For example:
 5```shell
 6./llama-cli -m "path/to/model.gguf" -ngl 200000 -p "Please sir, may I have some "
 7```
 8
 9When running llama, before it starts the inference work, it will output diagnostic information that shows whether cuBLAS is offloading work to the GPU. Look for these lines:
10```shell
11llama_model_load_internal: [cublas] offloading 60 layers to GPU
12llama_model_load_internal: [cublas] offloading output layer to GPU
13llama_model_load_internal: [cublas] total VRAM used: 17223 MB
14... rest of inference
15```
16
17If you see these lines, then the GPU is being used.
18
19## Verifying that the CPU is not oversaturated
20llama accepts a `-t N` (or `--threads N`) parameter. It's extremely important that this parameter is not too large. If your token generation is extremely slow, try setting this number to 1. If this significantly improves your token generation speed, then your CPU is being oversaturated and you need to explicitly set this parameter to the number of the physical CPU cores on your machine (even if you utilize a GPU). If in doubt, start with 1 and double the amount until you hit a performance bottleneck, then scale the number down.
21
22# Example of runtime flags effect on inference speed benchmark
23These runs were tested on the following machine:
24GPU: A6000 (48GB VRAM)
25CPU: 7 physical cores
26RAM: 32GB
27
28Model: `TheBloke_Wizard-Vicuna-30B-Uncensored-GGML/Wizard-Vicuna-30B-Uncensored.q4_0.gguf` (30B parameters, 4bit quantization, GGML)
29
30Run command: `./llama-cli -m "path/to/model.gguf" -p "An extremely detailed description of the 10 best ethnic dishes will follow, with recipes: " -n 1000 [additional benchmark flags]`
31
32Result:
33
34| command | tokens/second (higher is better) |
35| - | - |
36| -ngl 2000000 | N/A (less than 0.1) |
37| -t 7 | 1.7 |
38| -t 1 -ngl 2000000 | 5.5 |
39| -t 7 -ngl 2000000 | 8.7 |
40| -t 4 -ngl 2000000 | 9.1 |