simple-tokenize.h « peg-parser « tests « llama.cpp - llmnpc - Testing how LLM's fair as NPC's

summary refs log tree commit diff

path: root/llama.cpp/tests/peg-parser/simple-tokenize.h

blob: 1772432c5aa4ecad1efd4a51c9cee6b740a27b6f (plain)

#pragma once

#include <string>
#include <vector>

std::vector<std::string> simple_tokenize(const std::string &);