1# Function Calling
2
3[chat.h](../common/chat.h) (https://github.com/ggml-org/llama.cpp/pull/9639) adds support for [OpenAI-style function calling](https://platform.openai.com/docs/guides/function-calling) and is used in:
4- `llama-server` when started w/ `--jinja` flag
5
6## Universal support w/ Native & Generic handlers
7
8Function calling is supported for all models (see https://github.com/ggml-org/llama.cpp/pull/9639):
9
10- Native tool call formats supported:
11 - Llama 3.1 / 3.3 (including builtin tools support - tool names for `wolfram_alpha`, `web_search` / `brave_search`, `code_interpreter`), Llama 3.2
12 - Functionary v3.1 / v3.2
13 - Hermes 2/3, Qwen 2.5
14 - Qwen 2.5 Coder
15 - Mistral Nemo
16 - Firefunction v2
17 - Command R7B
18 - DeepSeek R1 (WIP / seems reluctant to call any tools?)
19
20- Generic tool call is supported when the template isn't recognized by native format handlers (you'll see `Chat format: Generic` in the logs).
21 - Use `--chat-template-file` to override the template when appropriate (see examples below)
22 - Generic support may consume more tokens and be less efficient than a model's native format.
23
24- Multiple/parallel tool calling is supported on some models but disabled by default, enable it by passing `"parallel_tool_calls": true` in the completion endpoint payload.
25
26<details>
27<summary>Show some common templates and which format handler they use</summary>
28
29| Template | Format |
30|----------|--------|
31| Almawave-Velvet-14B.jinja | Hermes 2 Pro |
32| AtlaAI-Selene-1-Mini-Llama-3.1-8B.jinja | Llama 3.x |
33| CohereForAI-aya-expanse-8b.jinja | Generic |
34| CohereForAI-c4ai-command-r-plus-default.jinja | Generic |
35| CohereForAI-c4ai-command-r-plus-rag.jinja | Generic |
36| CohereForAI-c4ai-command-r-plus-tool_use.jinja | Generic |
37| CohereForAI-c4ai-command-r7b-12-2024-default.jinja | Command R7B (extract reasoning) |
38| CohereForAI-c4ai-command-r7b-12-2024-rag.jinja | Command R7B (extract reasoning) |
39| CohereForAI-c4ai-command-r7b-12-2024-tool_use.jinja | Command R7B (extract reasoning) |
40| CohereForAI-c4ai-command-r7b-12-2024.jinja | Generic |
41| DavieLion-Llama-3.2-1B-SPIN-iter3.jinja | Generic |
42| Delta-Vector-Rei-12B.jinja | Mistral Nemo |
43| EpistemeAI-Mistral-Nemo-Instruct-12B-Philosophy-Math.jinja | Mistral Nemo |
44| FlofloB-83k_continued_pretraining_Qwen2.5-0.5B-Instruct_Unsloth_merged_16bit.jinja | Hermes 2 Pro |
45| FlofloB-test_continued_pretraining_Phi-3-mini-4k-instruct_Unsloth_merged_16bit.jinja | Generic |
46| HelpingAI-HAI-SER.jinja | Generic |
47| HuggingFaceTB-SmolLM2-1.7B-Instruct.jinja | Generic |
48| HuggingFaceTB-SmolLM2-135M-Instruct.jinja | Generic |
49| HuggingFaceTB-SmolLM2-360M-Instruct.jinja | Generic |
50| INSAIT-Institute-BgGPT-Gemma-2-27B-IT-v1.0.jinja | Generic |
51| Ihor-Text2Graph-R1-Qwen2.5-0.5b.jinja | Hermes 2 Pro |
52| Infinigence-Megrez-3B-Instruct.jinja | Generic |
53| Josephgflowers-TinyLlama_v1.1_math_code-world-test-1.jinja | Generic |
54| LGAI-EXAONE-EXAONE-3.5-2.4B-Instruct.jinja | Generic |
55| LGAI-EXAONE-EXAONE-3.5-7.8B-Instruct.jinja | Generic |
56| LatitudeGames-Wayfarer-12B.jinja | Generic |
57| Magpie-Align-Llama-3-8B-Magpie-Align-v0.1.jinja | Generic |
58| Magpie-Align-Llama-3.1-8B-Magpie-Align-v0.1.jinja | Generic |
59| MaziyarPanahi-calme-3.2-instruct-78b.jinja | Generic |
60| MiniMaxAI-MiniMax-Text-01.jinja | Generic |
61| MiniMaxAI-MiniMax-VL-01.jinja | Generic |
62| NaniDAO-deepseek-r1-qwen-2.5-32B-ablated.jinja | DeepSeek R1 (extract reasoning) |
63| NexaAIDev-Octopus-v2.jinja | Generic |
64| NousResearch-Hermes-2-Pro-Llama-3-8B-default.jinja | Generic |
65| NousResearch-Hermes-2-Pro-Llama-3-8B-tool_use.jinja | Hermes 2 Pro |
66| NousResearch-Hermes-2-Pro-Mistral-7B-default.jinja | Generic |
67| NousResearch-Hermes-2-Pro-Mistral-7B-tool_use.jinja | Hermes 2 Pro |
68| NousResearch-Hermes-3-Llama-3.1-70B-default.jinja | Generic |
69| NousResearch-Hermes-3-Llama-3.1-70B-tool_use.jinja | Hermes 2 Pro |
70| NovaSky-AI-Sky-T1-32B-Flash.jinja | Hermes 2 Pro |
71| NovaSky-AI-Sky-T1-32B-Preview.jinja | Hermes 2 Pro |
72| OnlyCheeini-greesychat-turbo.jinja | Generic |
73| Orenguteng-Llama-3.1-8B-Lexi-Uncensored-V2.jinja | Llama 3.x |
74| OrionStarAI-Orion-14B-Chat.jinja | Generic |
75| PowerInfer-SmallThinker-3B-Preview.jinja | Generic |
76| PrimeIntellect-INTELLECT-1-Instruct.jinja | Generic |
77| Qwen-QVQ-72B-Preview.jinja | Generic |
78| Qwen-QwQ-32B-Preview.jinja | Hermes 2 Pro |
79| Qwen-Qwen1.5-7B-Chat.jinja | Generic |
80| Qwen-Qwen2-7B-Instruct.jinja | Generic |
81| Qwen-Qwen2-VL-72B-Instruct.jinja | Generic |
82| Qwen-Qwen2-VL-7B-Instruct.jinja | Generic |
83| Qwen-Qwen2.5-0.5B.jinja | Hermes 2 Pro |
84| Qwen-Qwen2.5-1.5B-Instruct.jinja | Hermes 2 Pro |
85| Qwen-Qwen2.5-14B-Instruct-1M.jinja | Hermes 2 Pro |
86| Qwen-Qwen2.5-14B.jinja | Hermes 2 Pro |
87| Qwen-Qwen2.5-32B-Instruct.jinja | Hermes 2 Pro |
88| Qwen-Qwen2.5-32B.jinja | Hermes 2 Pro |
89| Qwen-Qwen2.5-3B-Instruct.jinja | Hermes 2 Pro |
90| Qwen-Qwen2.5-72B-Instruct.jinja | Hermes 2 Pro |
91| Qwen-Qwen2.5-7B-Instruct-1M.jinja | Hermes 2 Pro |
92| Qwen-Qwen2.5-7B-Instruct.jinja | Hermes 2 Pro |
93| Qwen-Qwen2.5-7B.jinja | Hermes 2 Pro |
94| Qwen-Qwen2.5-Coder-32B-Instruct.jinja | Hermes 2 Pro |
95| Qwen-Qwen2.5-Coder-7B-Instruct.jinja | Hermes 2 Pro |
96| Qwen-Qwen2.5-Math-1.5B.jinja | Hermes 2 Pro |
97| Qwen-Qwen2.5-Math-7B-Instruct.jinja | Hermes 2 Pro |
98| Qwen-Qwen2.5-VL-3B-Instruct.jinja | Hermes 2 Pro |
99| Qwen-Qwen2.5-VL-72B-Instruct.jinja | Hermes 2 Pro |
100| Qwen-Qwen2.5-VL-7B-Instruct.jinja | Hermes 2 Pro |
101| RWKV-Red-Team-ARWKV-7B-Preview-0.1.jinja | Hermes 2 Pro |
102| SakanaAI-TinySwallow-1.5B-Instruct.jinja | Hermes 2 Pro |
103| SakanaAI-TinySwallow-1.5B.jinja | Hermes 2 Pro |
104| Sao10K-70B-L3.3-Cirrus-x1.jinja | Llama 3.x |
105| SentientAGI-Dobby-Mini-Leashed-Llama-3.1-8B.jinja | Llama 3.x |
106| SentientAGI-Dobby-Mini-Unhinged-Llama-3.1-8B.jinja | Llama 3.x |
107| Steelskull-L3.3-Damascus-R1.jinja | Llama 3.x |
108| Steelskull-L3.3-MS-Nevoria-70b.jinja | Llama 3.x |
109| Steelskull-L3.3-Nevoria-R1-70b.jinja | Llama 3.x |
110| THUDM-glm-4-9b-chat.jinja | Generic |
111| THUDM-glm-edge-1.5b-chat.jinja | Generic |
112| Tarek07-Progenitor-V1.1-LLaMa-70B.jinja | Llama 3.x |
113| TheBloke-FusionNet_34Bx2_MoE-AWQ.jinja | Generic |
114| TinyLlama-TinyLlama-1.1B-Chat-v1.0.jinja | Generic |
115| UCLA-AGI-Mistral7B-PairRM-SPPO-Iter3.jinja | Generic |
116| ValiantLabs-Llama3.1-8B-Enigma.jinja | Llama 3.x |
117| abacusai-Fewshot-Metamath-OrcaVicuna-Mistral.jinja | Generic |
118| ai21labs-AI21-Jamba-1.5-Large.jinja | Generic |
119| allenai-Llama-3.1-Tulu-3-405B-SFT.jinja | Generic |
120| allenai-Llama-3.1-Tulu-3-405B.jinja | Generic |
121| allenai-Llama-3.1-Tulu-3-8B.jinja | Generic |
122| arcee-ai-Virtuoso-Lite.jinja | Hermes 2 Pro |
123| arcee-ai-Virtuoso-Medium-v2.jinja | Hermes 2 Pro |
124| arcee-ai-Virtuoso-Small-v2.jinja | Hermes 2 Pro |
125| avemio-GRAG-NEMO-12B-ORPO-HESSIAN-AI.jinja | Generic |
126| bespokelabs-Bespoke-Stratos-7B.jinja | Hermes 2 Pro |
127| bfuzzy1-acheron-m1a-llama.jinja | Generic |
128| bofenghuang-vigogne-2-70b-chat.jinja | Generic |
129| bytedance-research-UI-TARS-72B-DPO.jinja | Generic |
130| bytedance-research-UI-TARS-7B-DPO.jinja | Generic |
131| bytedance-research-UI-TARS-7B-SFT.jinja | Generic |
132| carsenk-phi3.5_mini_exp_825_uncensored.jinja | Generic |
133| cyberagent-DeepSeek-R1-Distill-Qwen-14B-Japanese.jinja | DeepSeek R1 (extract reasoning) |
134| cyberagent-DeepSeek-R1-Distill-Qwen-32B-Japanese.jinja | DeepSeek R1 (extract reasoning) |
135| databricks-dbrx-instruct.jinja | Generic |
136| deepseek-ai-DeepSeek-Coder-V2-Instruct.jinja | Generic |
137| deepseek-ai-DeepSeek-Coder-V2-Lite-Base.jinja | Generic |
138| deepseek-ai-DeepSeek-Coder-V2-Lite-Instruct.jinja | Generic |
139| deepseek-ai-DeepSeek-R1-Distill-Llama-70B.jinja | DeepSeek R1 (extract reasoning) |
140| deepseek-ai-DeepSeek-R1-Distill-Llama-8B.jinja | DeepSeek R1 (extract reasoning) |
141| deepseek-ai-DeepSeek-R1-Distill-Qwen-1.5B.jinja | DeepSeek R1 (extract reasoning) |
142| deepseek-ai-DeepSeek-R1-Distill-Qwen-14B.jinja | DeepSeek R1 (extract reasoning) |
143| deepseek-ai-DeepSeek-R1-Distill-Qwen-32B.jinja | DeepSeek R1 (extract reasoning) |
144| deepseek-ai-DeepSeek-R1-Distill-Qwen-7B.jinja | DeepSeek R1 (extract reasoning) |
145| deepseek-ai-DeepSeek-R1-Zero.jinja | DeepSeek R1 (extract reasoning) |
146| deepseek-ai-DeepSeek-R1.jinja | DeepSeek R1 (extract reasoning) |
147| deepseek-ai-DeepSeek-V2-Lite.jinja | Generic |
148| deepseek-ai-DeepSeek-V2.5.jinja | DeepSeek R1 (extract reasoning) |
149| deepseek-ai-DeepSeek-V3.jinja | DeepSeek R1 (extract reasoning) |
150| deepseek-ai-deepseek-coder-33b-instruct.jinja | Generic |
151| deepseek-ai-deepseek-coder-6.7b-instruct.jinja | Generic |
152| deepseek-ai-deepseek-coder-7b-instruct-v1.5.jinja | Generic |
153| deepseek-ai-deepseek-llm-67b-chat.jinja | Generic |
154| deepseek-ai-deepseek-llm-7b-chat.jinja | Generic |
155| dicta-il-dictalm2.0-instruct.jinja | Generic |
156| ehristoforu-Falcon3-8B-Franken-Basestruct.jinja | Hermes 2 Pro |
157| fireworks-ai-llama-3-firefunction-v2.jinja | FireFunction v2 |
158| godlikehhd-alpaca_data_sampled_ifd_new_5200.jinja | Hermes 2 Pro |
159| godlikehhd-alpaca_data_score_max_0.7_2600.jinja | Hermes 2 Pro |
160| google-gemma-2-27b-it.jinja | Generic |
161| google-gemma-2-2b-it.jinja | Generic |
162| google-gemma-2-2b-jpn-it.jinja | Generic |
163| google-gemma-7b-it.jinja | Generic |
164| huihui-ai-DeepSeek-R1-Distill-Llama-70B-abliterated.jinja | DeepSeek R1 (extract reasoning) |
165| huihui-ai-DeepSeek-R1-Distill-Llama-8B-abliterated.jinja | DeepSeek R1 (extract reasoning) |
166| huihui-ai-DeepSeek-R1-Distill-Qwen-14B-abliterated-v2.jinja | DeepSeek R1 (extract reasoning) |
167| huihui-ai-DeepSeek-R1-Distill-Qwen-32B-abliterated.jinja | DeepSeek R1 (extract reasoning) |
168| huihui-ai-DeepSeek-R1-Distill-Qwen-7B-abliterated-v2.jinja | DeepSeek R1 (extract reasoning) |
169| huihui-ai-Qwen2.5-14B-Instruct-1M-abliterated.jinja | Hermes 2 Pro |
170| ibm-granite-granite-3.1-8b-instruct.jinja | Generic |
171| indischepartij-MiniCPM-3B-OpenHermes-2.5-v2.jinja | Generic |
172| inflatebot-MN-12B-Mag-Mell-R1.jinja | Generic |
173| jinaai-ReaderLM-v2.jinja | Generic |
174| kms7530-chemeng_qwen-math-7b_24_1_100_1_nonmath.jinja | Hermes 2 Pro |
175| knifeayumu-Cydonia-v1.3-Magnum-v4-22B.jinja | Mistral Nemo |
176| langgptai-qwen1.5-7b-chat-sa-v0.1.jinja | Generic |
177| lightblue-DeepSeek-R1-Distill-Qwen-7B-Japanese.jinja | DeepSeek R1 (extract reasoning) |
178| mattshumer-Reflection-Llama-3.1-70B.jinja | Generic |
179| meetkai-functionary-medium-v3.1.jinja | Functionary v3.1 Llama 3.1 |
180| meetkai-functionary-medium-v3.2.jinja | Functionary v3.2 |
181| meta-llama-Llama-2-7b-chat-hf.jinja | Generic |
182| meta-llama-Llama-3.1-8B-Instruct.jinja | Llama 3.x |
183| meta-llama-Llama-3.2-11B-Vision-Instruct.jinja | Llama 3.x |
184| meta-llama-Llama-3.2-1B-Instruct.jinja | Llama 3.x |
185| meta-llama-Llama-3.2-3B-Instruct.jinja | Llama 3.x |
186| meta-llama-Llama-3.3-70B-Instruct.jinja | Llama 3.x |
187| meta-llama-Meta-Llama-3-8B-Instruct.jinja | Generic |
188| meta-llama-Meta-Llama-3.1-8B-Instruct.jinja | Llama 3.x |
189| microsoft-Phi-3-medium-4k-instruct.jinja | Generic |
190| microsoft-Phi-3-mini-4k-instruct.jinja | Generic |
191| microsoft-Phi-3-small-8k-instruct.jinja | Generic |
192| microsoft-Phi-3.5-mini-instruct.jinja | Generic |
193| microsoft-Phi-3.5-vision-instruct.jinja | Generic |
194| microsoft-phi-4.jinja | Generic |
195| migtissera-Tess-3-Mistral-Nemo-12B.jinja | Generic |
196| ministral-Ministral-3b-instruct.jinja | Generic |
197| mistralai-Codestral-22B-v0.1.jinja | Generic |
198| mistralai-Mistral-7B-Instruct-v0.1.jinja | Generic |
199| mistralai-Mistral-7B-Instruct-v0.2.jinja | Generic |
200| mistralai-Mistral-7B-Instruct-v0.3.jinja | Mistral Nemo |
201| mistralai-Mistral-Large-Instruct-2407.jinja | Mistral Nemo |
202| mistralai-Mistral-Large-Instruct-2411.jinja | Generic |
203| mistralai-Mistral-Nemo-Instruct-2407.jinja | Mistral Nemo |
204| mistralai-Mistral-Small-24B-Instruct-2501.jinja | Generic |
205| mistralai-Mixtral-8x7B-Instruct-v0.1.jinja | Generic |
206| mkurman-Qwen2.5-14B-DeepSeek-R1-1M.jinja | Hermes 2 Pro |
207| mlabonne-AlphaMonarch-7B.jinja | Generic |
208| mlx-community-Josiefied-Qwen2.5-0.5B-Instruct-abliterated-v1-float32.jinja | Hermes 2 Pro |
209| mlx-community-Qwen2.5-VL-7B-Instruct-8bit.jinja | Hermes 2 Pro |
210| mobiuslabsgmbh-DeepSeek-R1-ReDistill-Qwen-1.5B-v1.1.jinja | DeepSeek R1 (extract reasoning) |
211| netcat420-MFANNv0.20.jinja | Generic |
212| netcat420-MFANNv0.24.jinja | Generic |
213| netease-youdao-Confucius-o1-14B.jinja | Hermes 2 Pro |
214| nvidia-AceMath-7B-RM.jinja | Hermes 2 Pro |
215| nvidia-Eagle2-1B.jinja | Hermes 2 Pro |
216| nvidia-Eagle2-9B.jinja | Hermes 2 Pro |
217| nvidia-Llama-3.1-Nemotron-70B-Instruct-HF.jinja | Llama 3.x |
218| onnx-community-DeepSeek-R1-Distill-Qwen-1.5B-ONNX.jinja | DeepSeek R1 (extract reasoning) |
219| open-thoughts-OpenThinker-7B.jinja | Hermes 2 Pro |
220| openchat-openchat-3.5-0106.jinja | Generic |
221| pankajmathur-orca_mini_v6_8b.jinja | Generic |
222| princeton-nlp-Mistral-7B-Base-SFT-RDPO.jinja | Generic |
223| princeton-nlp-Mistral-7B-Instruct-DPO.jinja | Generic |
224| princeton-nlp-Mistral-7B-Instruct-RDPO.jinja | Generic |
225| prithivMLmods-Bellatrix-Tiny-1.5B-R1.jinja | Hermes 2 Pro |
226| prithivMLmods-Bellatrix-Tiny-1B-R1.jinja | Llama 3.x |
227| prithivMLmods-Bellatrix-Tiny-1B-v3.jinja | Generic |
228| prithivMLmods-Bellatrix-Tiny-3B-R1.jinja | Llama 3.x |
229| prithivMLmods-Blaze-14B-xElite.jinja | Generic |
230| prithivMLmods-Calcium-Opus-14B-Elite2-R1.jinja | Hermes 2 Pro |
231| prithivMLmods-Calme-Ties-78B.jinja | Generic |
232| prithivMLmods-Calme-Ties2-78B.jinja | Generic |
233| prithivMLmods-Calme-Ties3-78B.jinja | Generic |
234| prithivMLmods-ChemQwen2-vL.jinja | Generic |
235| prithivMLmods-GWQ2b.jinja | Generic |
236| prithivMLmods-LatexMind-2B-Codec.jinja | Generic |
237| prithivMLmods-Llama-3.2-6B-AlgoCode.jinja | Llama 3.x |
238| prithivMLmods-Megatron-Opus-14B-Exp.jinja | Hermes 2 Pro |
239| prithivMLmods-Megatron-Opus-14B-Stock.jinja | Hermes 2 Pro |
240| prithivMLmods-Megatron-Opus-7B-Exp.jinja | Hermes 2 Pro |
241| prithivMLmods-Omni-Reasoner-Merged.jinja | Hermes 2 Pro |
242| prithivMLmods-Omni-Reasoner4-Merged.jinja | Hermes 2 Pro |
243| prithivMLmods-Primal-Opus-14B-Optimus-v1.jinja | Hermes 2 Pro |
244| prithivMLmods-QwQ-Math-IO-500M.jinja | Hermes 2 Pro |
245| prithivMLmods-Qwen-7B-Distill-Reasoner.jinja | DeepSeek R1 (extract reasoning) |
246| prithivMLmods-Qwen2.5-1.5B-DeepSeek-R1-Instruct.jinja | Hermes 2 Pro |
247| prithivMLmods-Qwen2.5-14B-DeepSeek-R1-1M.jinja | Hermes 2 Pro |
248| prithivMLmods-Qwen2.5-32B-DeepSeek-R1-Instruct.jinja | Hermes 2 Pro |
249| prithivMLmods-Qwen2.5-7B-DeepSeek-R1-1M.jinja | Hermes 2 Pro |
250| prithivMLmods-Triangulum-v2-10B.jinja | Hermes 2 Pro |
251| qingy2024-Falcon3-2x10B-MoE-Instruct.jinja | Hermes 2 Pro |
252| rubenroy-Zurich-14B-GCv2-5m.jinja | Hermes 2 Pro |
253| rubenroy-Zurich-7B-GCv2-5m.jinja | Hermes 2 Pro |
254| silma-ai-SILMA-Kashif-2B-Instruct-v1.0.jinja | Generic |
255| simplescaling-s1-32B.jinja | Hermes 2 Pro |
256| sometimesanotion-Lamarck-14B-v0.7.jinja | Hermes 2 Pro |
257| sonthenguyen-zephyr-sft-bnb-4bit-DPO-mtbr-180steps.jinja | Generic |
258| sthenno-tempesthenno-icy-0130.jinja | Generic |
259| sumink-qwft.jinja | Hermes 2 Pro |
260| teknium-OpenHermes-2.5-Mistral-7B.jinja | Generic |
261| thirdeyeai-elevate360m.jinja | Generic |
262| tiiuae-Falcon3-10B-Instruct.jinja | Hermes 2 Pro |
263| unsloth-DeepSeek-R1-Distill-Llama-8B-unsloth-bnb-4bit.jinja | DeepSeek R1 (extract reasoning) |
264| unsloth-DeepSeek-R1-Distill-Llama-8B.jinja | DeepSeek R1 (extract reasoning) |
265| unsloth-DeepSeek-R1.jinja | DeepSeek R1 (extract reasoning) |
266| unsloth-Mistral-Small-24B-Instruct-2501-unsloth-bnb-4bit.jinja | Generic |
267| upstage-solar-pro-preview-instruct.jinja | Generic |
268| whyhow-ai-PatientSeek.jinja | Generic |
269| xwen-team-Xwen-72B-Chat.jinja | Hermes 2 Pro |
270| xwen-team-Xwen-7B-Chat.jinja | Hermes 2 Pro |
271
272This table can be generated with:
273
274<!-- TODO @ngxson : we should update this, since minja dependency has been removed -->
275
276```bash
277./build/bin/test-chat ../minja/build/tests/*.jinja 2>/dev/null
278```
279
280</details>
281
282# Usage - need tool-aware Jinja template
283
284First, start a server with any model, but make sure it has a tools-enabled template: you can verify this by inspecting the `chat_template` or `chat_template_tool_use` properties in `http://localhost:8080/props`).
285
286Here are some models known to work (w/ chat template override when needed):
287
288```shell
289# Native support:
290
291llama-server --jinja -fa -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M
292llama-server --jinja -fa -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K_L
293llama-server --jinja -fa -hf bartowski/Llama-3.3-70B-Instruct-GGUF:Q4_K_M
294
295# Native support for DeepSeek R1 works best w/ our template override (official template is buggy, although we do work around it)
296
297llama-server --jinja -fa -hf bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF:Q6_K_L \
298 --chat-template-file models/templates/llama-cpp-deepseek-r1.jinja
299
300llama-server --jinja -fa -hf bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF:Q4_K_M \
301 --chat-template-file models/templates/llama-cpp-deepseek-r1.jinja
302
303# Native support requires the right template for these GGUFs:
304
305llama-server --jinja -fa -hf bartowski/functionary-small-v3.2-GGUF:Q4_K_M
306 --chat-template-file models/templates/meetkai-functionary-medium-v3.2.jinja
307
308llama-server --jinja -fa -hf bartowski/Hermes-2-Pro-Llama-3-8B-GGUF:Q4_K_M \
309 --chat-template-file models/templates/NousResearch-Hermes-2-Pro-Llama-3-8B-tool_use.jinja
310
311llama-server --jinja -fa -hf bartowski/Hermes-3-Llama-3.1-8B-GGUF:Q4_K_M \
312 --chat-template-file models/templates/NousResearch-Hermes-3-Llama-3.1-8B-tool_use.jinja
313
314llama-server --jinja -fa -hf bartowski/firefunction-v2-GGUF -hff firefunction-v2-IQ1_M.gguf \
315 --chat-template-file models/templates/fireworks-ai-llama-3-firefunction-v2.jinja
316
317llama-server --jinja -fa -hf bartowski/c4ai-command-r7b-12-2024-GGUF:Q6_K_L \
318 --chat-template-file models/templates/CohereForAI-c4ai-command-r7b-12-2024-tool_use.jinja
319
320# Generic format support
321llama-server --jinja -fa -hf bartowski/phi-4-GGUF:Q4_0
322llama-server --jinja -fa -hf bartowski/gemma-2-2b-it-GGUF:Q8_0
323llama-server --jinja -fa -hf bartowski/c4ai-command-r-v01-GGUF:Q2_K
324```
325
326To get the official template from original HuggingFace repos, you can use [scripts/get_chat_template.py](../scripts/get_chat_template.py) (see examples invocations in [models/templates/README.md](../models/templates/README.md))
327
328> [!TIP]
329> If there is no official `tool_use` Jinja template, you may want to set `--chat-template chatml` to use a default that works with many models (YMMV!), or write your own (e.g. we provide a custom [llama-cpp-deepseek-r1.jinja](../models/templates/llama-cpp-deepseek-r1.jinja) for DeepSeek R1 distills)
330
331> [!CAUTION]
332> Beware of extreme KV quantizations (e.g. `-ctk q4_0`), they can substantially degrade the model's tool calling performance.
333
334Test in CLI (or with any library / software that can use OpenAI-compatible API backends):
335
336```bash
337curl http://localhost:8080/v1/chat/completions -d '{
338 "model": "gpt-3.5-turbo",
339 "tools": [
340 {
341 "type":"function",
342 "function":{
343 "name":"python",
344 "description":"Runs code in an ipython interpreter and returns the result of the execution after 60 seconds.",
345 "parameters":{
346 "type":"object",
347 "properties":{
348 "code":{
349 "type":"string",
350 "description":"The code to run in the ipython interpreter."
351 }
352 },
353 "required":["code"]
354 }
355 }
356 }
357 ],
358 "messages": [
359 {
360 "role": "user",
361 "content": "Print a hello world message with python."
362 }
363 ]
364}'
365
366
367curl http://localhost:8080/v1/chat/completions -d '{
368 "model": "gpt-3.5-turbo",
369 "messages": [
370 {"role": "system", "content": "You are a chatbot that uses tools/functions. Dont overthink things."},
371 {"role": "user", "content": "What is the weather in Istanbul?"}
372 ],
373 "tools": [{
374 "type":"function",
375 "function":{
376 "name":"get_current_weather",
377 "description":"Get the current weather in a given location",
378 "parameters":{
379 "type":"object",
380 "properties":{
381 "location":{
382 "type":"string",
383 "description":"The city and country/state, e.g. `San Francisco, CA`, or `Paris, France`"
384 }
385 },
386 "required":["location"]
387 }
388 }
389 }]
390}'
391```
392
393<details>
394<summary>Show output</summary>
395
396```json
397{
398"choices": [
399 {
400 "finish_reason": "tool",
401 "index": 0,
402 "message": {
403 "content": null,
404 "tool_calls": [
405 {
406 "name": "python",
407 "arguments": "{\"code\":\" \\nprint(\\\"Hello, World!\\\")\"}"
408 }
409 ],
410 "role": "assistant"
411 }
412 }
413],
414"created": 1727287211,
415"model": "gpt-3.5-turbo",
416"object": "chat.completion",
417"usage": {
418 "completion_tokens": 16,
419 "prompt_tokens": 44,
420 "total_tokens": 60
421},
422"id": "chatcmpl-Htbgh9feMmGM0LEH2hmQvwsCxq3c6Ni8"
423}
424```
425
426</details>