1# Debugging Tests Tips
  2
  3## How to run & execute or debug a specific test without anything else to keep the feedback loop short?
  4
  5There is a script called debug-test.sh in the scripts folder whose parameter takes a REGEX and an optional test number.
  6
  7For example, running the following command will output an interactive list from which you can select a test. It takes this form:
  8
  9`debug-test.sh [OPTION]... <test_regex> <test_number>`
 10
 11It will then build & run in the debugger for you.
 12
 13To just execute a test and get back a PASS or FAIL message run:
 14
 15```bash
 16./scripts/debug-test.sh test-tokenizer
 17```
 18
 19To test in GDB use the `-g` flag to enable gdb test mode.
 20
 21```bash
 22./scripts/debug-test.sh -g test-tokenizer
 23
 24# Once in the debugger, i.e. at the chevrons prompt, setting a breakpoint could be as follows:
 25>>> b main
 26```
 27
 28To speed up the testing loop, if you know your test number you can just run it similar to below:
 29
 30```bash
 31./scripts/debug-test.sh test 23
 32```
 33
 34For further reference use `debug-test.sh -h` to print help.
 35
 36&nbsp;
 37
 38### How does the script work?
 39If you want to be able to use the concepts contained in the script separately, the important ones are briefly outlined below.
 40
 41#### Step 1: Reset and Setup folder context
 42
 43From base of this repository, let's create `build-ci-debug` as our build context.
 44
 45```bash
 46rm -rf build-ci-debug && mkdir build-ci-debug && cd build-ci-debug
 47```
 48
 49#### Step 2: Setup Build Environment and Compile Test Binaries
 50
 51Setup and trigger a build under debug mode. You may adapt the arguments as needed, but in this case these are sane defaults.
 52
 53```bash
 54cmake -DCMAKE_BUILD_TYPE=Debug -DLLAMA_CUDA=1 -DLLAMA_FATAL_WARNINGS=ON ..
 55make -j
 56```
 57
 58#### Step 3: Find all tests available that matches REGEX
 59
 60The output of this command will give you the command & arguments needed to run GDB.
 61
 62* `-R test-tokenizer` : looks for all the test files named `test-tokenizer*` (R=Regex)
 63* `-N` : "show-only" disables test execution & shows test commands that you can feed to GDB.
 64* `-V` : Verbose Mode
 65
 66```bash
 67ctest -R "test-tokenizer" -V -N
 68```
 69
 70This may return output similar to below (focusing on key lines to pay attention to):
 71
 72```bash
 73...
 741: Test command: ~/llama.cpp/build-ci-debug/bin/test-tokenizer-0 "~/llama.cpp/tests/../models/ggml-vocab-llama-spm.gguf"
 751: Working Directory: .
 76Labels: main
 77  Test  #1: test-tokenizer-0-llama-spm
 78...
 794: Test command: ~/llama.cpp/build-ci-debug/bin/test-tokenizer-0 "~/llama.cpp/tests/../models/ggml-vocab-falcon.gguf"
 804: Working Directory: .
 81Labels: main
 82  Test  #4: test-tokenizer-0-falcon
 83...
 84```
 85
 86#### Step 4: Identify Test Command for Debugging
 87
 88So for test #1 above we can tell these two pieces of relevant information:
 89* Test Binary: `~/llama.cpp/build-ci-debug/bin/test-tokenizer-0`
 90* Test GGUF Model: `~/llama.cpp/tests/../models/ggml-vocab-llama-spm.gguf`
 91
 92#### Step 5: Run GDB on test command
 93
 94Based on the ctest 'test command' report above we can then run a gdb session via this command below:
 95
 96```bash
 97gdb --args ${Test Binary} ${Test GGUF Model}
 98```
 99
100Example:
101
102```bash
103gdb --args ~/llama.cpp/build-ci-debug/bin/test-tokenizer-0 "~/llama.cpp/tests/../models/ggml-vocab-llama-spm.gguf"
104```