blob: 9d8fe1b393b346df5bc271166b316d9ae6984f1a (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
|
# llmnpc
A command-line LLM inference tool powered by
[llama.cpp](https://github.com/ggerganov/llama.cpp) for testing how/if NPC's
could use LLM's.
## Building
### Prerequisites
- C compiler (gcc/clang)
- CMake
- Docker (optional, for containerized use of binaries)
### Build Steps
1. Build llama.cpp libraries:
```bash
make llamacpp
```
2. Download models
```bash
make fetchmodels
```
3. Build the prompt binary:
```bash
make prompt
```
## Usage
```bash
./prompt -p "Your prompt here"
./prompt -m flan-t5-small -p "What is machine learning?"
```
### Options
| Flag | Description |
|------|-------------|
| `-m, --model` | Model to use (default: first model in config) |
| `-p, --prompt` | Prompt text (required) |
| `-h, --help` | Show help message |
## Models
Configure models in `models.h`. The default model is `flan-t5-small`, expecting a GGUF file at `models/flan-t5-small.F16.gguf`.
## Docker
```bash
make docker
```
This builds a Docker image and drops you into a shell with the prompt binary and models available at `/app/`.
## Cleaning
```bash
make clean
```
## Reading material
- https://www.tinyllm.org/
|
