1# llama.cpp/tools/imatrix
 2
 3Compute an importance matrix for a model and given text dataset. Can be used during quantization to enhance the quality of the quantized models.
 4More information is available in <https://github.com/ggml-org/llama.cpp/pull/4861>.
 5
 6## Usage
 7
 8```
 9./llama-imatrix \
10    -m model.gguf -f some-text.txt [-o imatrix.gguf] [--output-format {gguf,dat}] [--no-ppl] \
11    [--process-output] [--chunk 123] [--save-frequency 0] [--output-frequency 10] \
12    [--in-file imatrix-prev-0.gguf --in-file imatrix-prev-1.gguf ...] [--parse-special] \
13    [--show-statistics] [...]
14```
15
16Here `-m | --model` with a model name and `-f | --file` with a file containing calibration data (such as e.g. `wiki.train.raw`) are mandatory.
17The parameters in square brackets are optional and have the following meaning:
18
19* `-h | --help` shows usage information and exits.
20* `-lv | --verbosity` specifies the verbosity level. If set to `0`, no output other than the perplexity of the processed chunks will be generated. If set to `1`, each time the results are saved a message is written to `stderr`. If `>=2`, a message is output each time data is collected for any tensor. Default verbosity level is `1`.
21* `-o | --output-file` specifies the name of the file where the computed data will be stored. If missing `imatrix.gguf` is used.
22* `-ofreq | --output-frequency` specifies how often the so far computed result is saved to disk. Default is 10 (i.e., every 10 chunks)
23* `--output-format` specifies the output format of the generated imatrix file. Either "gguf", or "dat" (the legacy format). Defaults to "gguf".
24* `--save-frequency` specifies how often to save a copy of the imatrix in a separate file. Default is 0 (i.e., never)
25* `--process-output` specifies if data will be collected for the `output.weight` tensor. Typically, it is better not to utilize the importance matrix when quantizing `output.weight`, so this is set to `false` by default.
26* `--in-file` one or more existing imatrix files to load and combine. Useful for merging files from multiple runs/datasets.
27* `--parse-special` enables parsing of special tokens (e.g., `<|im_start|>` in some models). Useful for models with custom tokenizers.
28* `--chunk | --from-chunk` to skip the first `n` chunks of tokens from the input data. Useful for resuming or skipping initial low-quality data.
29* `--chunks` maximum number of chunks to process. Default is -1 for all available chunks.
30* `--no-ppl` disables the calculation of perplexity for the processed chunks. Useful if you want to speed up the processing and do not care about perplexity.
31* `--show-statistics` displays imatrix file's statistics.
32
33For faster computation, make sure to use GPU offloading via the `-ngl | --n-gpu-layers` argument.
34
35Recent versions of `llama-imatrix` store data in GGUF format by default. For the legacy format, use an extension other than `.gguf` when saving the output file. More information is available in <https://github.com/ggml-org/llama.cpp/pull/9400>.
36
37## Examples
38
39```bash
40# generate importance matrix using default filename (imatrix.gguf), offloading 99 layers to GPU
41./llama-imatrix -m ggml-model-f16.gguf -f calibration-data.txt -ngl 99
42
43# use the imatrix to perform a Q4_K_M quantization
44./llama-quantize --imatrix imatrix.gguf ggml-model-f16.gguf ./ggml-model-q4_k_m.gguf q4_k_m
45```
46
47```bash
48# generate and save the imatrix using legacy format
49./llama-imatrix -m ggml-model-f16.gguf -f calibration-data.txt --output-format dat -o imatrix-legcy-format.dat -ngl 99
50```
51
52```bash
53# convert legacy (binary) imatrix format to new (GGUF) format
54./llama-imatrix --in-file imatrix-legacy-format.dat -o imatrix-new-format.gguf
55```
56
57```bash
58# convert new (GGUF) imatrix format to legacy (binary) format
59./llama-imatrix --in-file imatrix-new-format.gguf --output-format dat -o imatrix-legacy-format.dat
60```
61
62```bash
63# combine existing imatrices
64./llama-imatrix --in-file imatrix-prev-0.gguf --in-file imatrix-prev-1.gguf -o imatrix-combined.gguf
65```
66
67```bash
68# skip first 5 chunks, save intermediates every 20 chunks and snapshots every 50, parsing special tokens
69./llama-imatrix -m ggml-model-f16.gguf -f calibration-data.txt --chunk 5 --output-frequency 20 --save-frequency 50 --parse-special
70```
71
72```bash
73# analyse imatrix file and display summary statistics instead of running inference
74./llama-imatrix --in-file imatrix.gguf --show-statistics
75```
76
77`--show-statistics` will display the following statistics:
78
79#### Per tensor
80
81* Σ(Act²): sum of all squared activations (the importance scores)
82* Min & Max: minimum and maximum squared activations values
83* μ & σ: Squared activations' mean and standard deviation
84* % Active: proportion of elements whose average squared activation exceeds a small threshold (1e-5). Helpful to determine how alive/dormant the tensor is during inference
85* N: number of squared activations
86* Entropy: entropy of the squared activation distribution, in bits (standard Shannon entropy measurement) $S = -\sum_{i=1}^N p_i \log_2 p_i$
87* E (norm): Normalized entropy. $E(norm)=\frac{-\sum_{i=1}^N p_i \log_2 p_i}{log_2 N}$. These two metrics can be used to determine how well a prompt "exercises" the model's capabilities
88* ZD Score: z-score distribution as described in _3.1 Layer Importance Scores_ of [Layer-Wise Quantization](https://arxiv.org/abs/2406.17415)
89* CosSim: cosine similarity with respect to the previous layer's tensor. Useful to determine how similar the squared activations of the current layer are to the previous layer's squared activations.
90
91#### Per layer
92
93Weighted averages of Σ(Act²), ZD Score and CosSim are also calculated.
94
95#### Important note on the computed Statistics
96
97When using these statistics, please note that they are computed on the squared activations, **not on the actual (raw) activations**.
98Whilst the results are still useful, they're less realiable than using the raw values, and in the case of the cosine similarity, could be misleading if the tensor contains opposite vectors.