|
diff --git a/README.md b/README.md
|
| 1 |
# llmnpc |
1 |
# llmnpc |
| 2 |
|
2 |
|
| 3 |
A command-line LLM inference tool powered by |
3 |
Command-line LLM inference and simple context retrieval powered by |
| 4 |
[llama.cpp](https://github.com/ggerganov/llama.cpp) for testing how/if NPC's |
4 |
[llama.cpp](https://github.com/ggerganov/llama.cpp) to test viability of using |
| 5 |
could use LLM's. |
5 |
LLM's to drive NPC behaviour. |
| 6 |
|
6 |
|
| 7 |
## Building |
7 |
## Building |
| 8 |
|
8 |
|
| ... |
| 16 |
|
16 |
|
| 17 |
1. Build llama.cpp libraries: |
17 |
1. Build llama.cpp libraries: |
| 18 |
```bash |
18 |
```bash |
| 19 |
make llamacpp |
19 |
make build/llama.cpp |
| 20 |
``` |
20 |
``` |
| 21 |
|
21 |
|
| 22 |
2. Download models |
22 |
2. Download models: |
| 23 |
```bash |
23 |
```bash |
| 24 |
make fetchmodels |
24 |
make run/fetch-models |
| 25 |
``` |
25 |
``` |
| 26 |
|
26 |
|
| 27 |
3. Build the prompt binary: |
27 |
3. Build binaries: |
| 28 |
```bash |
28 |
```bash |
| 29 |
make prompt |
29 |
make build/prompt |
|
|
30 |
make build/context |
| 30 |
``` |
31 |
``` |
| 31 |
|
32 |
|
| 32 |
## Usage |
33 |
## Usage |
| 33 |
|
34 |
|
|
|
35 |
### Build a vector context database |
|
|
36 |
|
|
|
37 |
`context` reads a text file (one document per line), embeds each line, and |
|
|
38 |
produces a binary vector database file. |
|
|
39 |
|
| 34 |
```bash |
40 |
```bash |
| 35 |
./prompt -p "Your prompt here" |
41 |
./context -i context.txt -o context.vdb |
| 36 |
./prompt -m flan-t5-small -p "What is machine learning?" |
42 |
./context -m flan-t5-small -i context.txt -o context.vdb |
| 37 |
``` |
43 |
``` |
| 38 |
|
44 |
|
| 39 |
### Options |
45 |
### Run a prompt with retrieved context |
|
|
46 |
|
|
|
47 |
`prompt` reads the context text file, embeds the query, selects the top 3 |
|
|
48 |
matching lines by cosine similarity, and builds a prompt from those lines. |
|
|
49 |
|
|
|
50 |
```bash |
|
|
51 |
./prompt -p "What is machine learning?" -c context.txt |
|
|
52 |
./prompt -m flan-t5-small -p "What is machine learning?" -c context.txt |
|
|
53 |
``` |
|
|
54 |
|
|
|
55 |
### context options |
|
|
56 |
|
|
|
57 |
| Flag | Description | |
|
|
58 |
|------|-------------| |
|
|
59 |
| `-m, --model` | Embedding model to use (default: first model in config) | |
|
|
60 |
| `-i, --in` | Input context text file (required) | |
|
|
61 |
| `-o, --out` | Output vector database file (required) | |
|
|
62 |
| `-l, --list` | List available models | |
|
|
63 |
| `-v, --verbose` | Enable llama.cpp logging | |
|
|
64 |
| `-h, --help` | Show help message | |
|
|
65 |
|
|
|
66 |
### prompt options |
| 40 |
|
67 |
|
| 41 |
| Flag | Description | |
68 |
| Flag | Description | |
| 42 |
|------|-------------| |
69 |
|------|-------------| |
| 43 |
| `-m, --model` | Model to use (default: first model in config) | |
70 |
| `-m, --model` | Model to use (default: first model in config) | |
| 44 |
| `-p, --prompt` | Prompt text (required) | |
71 |
| `-p, --prompt` | Prompt text (required) | |
|
|
72 |
| `-c, --context` | Context text file (required) | |
|
|
73 |
| `-v, --verbose` | Enable llama.cpp logging | |
| 45 |
| `-h, --help` | Show help message | |
74 |
| `-h, --help` | Show help message | |
| 46 |
|
75 |
|
| 47 |
## Models |
76 |
## Models |
| 48 |
|
77 |
|
| 49 |
Configure models in `models.h`. The default model is `flan-t5-small`, expecting a GGUF file at `models/flan-t5-small.F16.gguf`. |
78 |
Configure models in `models.h`. The default model is the first entry in the |
|
|
79 |
`models` array; each entry points at a local GGUF file under `models/`. |
|
|
80 |
|
|
|
81 |
## Vector database format |
|
|
82 |
|
|
|
83 |
`context` produces a binary file with a fixed header and a contiguous list of |
|
|
84 |
documents. The header includes a magic value, version, embedding size, maximum |
|
|
85 |
text length, and document count. Each document stores the original text (fixed |
|
|
86 |
size `VDB_MAX_TEXT`) and its embedding (`VDB_EMBED_SIZE`). |
| 50 |
|
87 |
|
| 51 |
## Docker |
88 |
## Docker |
| 52 |
|
89 |
|
| 53 |
```bash |
90 |
```bash |
| 54 |
make docker |
91 |
make run/docker |
| 55 |
``` |
92 |
``` |
| 56 |
|
93 |
|
| 57 |
This builds a Docker image and drops you into a shell with the prompt binary and models available at `/app/`. |
94 |
Builds a Docker image and runs an interactive shell with the binaries and |
|
|
95 |
models under `/app/`. |
| 58 |
|
96 |
|
| 59 |
## Cleaning |
97 |
## Cleaning |
| 60 |
|
98 |
|
| 61 |
```bash |
99 |
```bash |
| 62 |
make clean |
100 |
make run/clean |
| 63 |
``` |
101 |
``` |
| 64 |
|
102 |
|
| 65 |
## Reading material |
103 |
## Reading material |
| ... |