1ied 4 ½ months
2__ggml_vocab_test__
3Äpfel
4__ggml_vocab_test__
5
6__ggml_vocab_test__
7
8__ggml_vocab_test__
9
10__ggml_vocab_test__
11
12__ggml_vocab_test__
13
14__ggml_vocab_test__
15
16
17__ggml_vocab_test__
18
19
20
21__ggml_vocab_test__
22
23
24
25
26__ggml_vocab_test__
27
28
29__ggml_vocab_test__
30Hello world
31__ggml_vocab_test__
32 Hello world
33__ggml_vocab_test__
34Hello World
35__ggml_vocab_test__
36 Hello World
37__ggml_vocab_test__
38 Hello World!
39__ggml_vocab_test__
40Hello, world!
41__ggml_vocab_test__
42 Hello, world!
43__ggml_vocab_test__
44 this is 🦙.cpp
45__ggml_vocab_test__
46w048 7tuijk dsdfhu
47__ggml_vocab_test__
48нещо на Български
49__ggml_vocab_test__
50កាន់តែពិសេសអាចខលចេញ
51__ggml_vocab_test__
52🚀 (normal) 😶🌫️ (multiple emojis concatenated) ✅ (only emoji that has its own token)
53__ggml_vocab_test__
54Hello
55__ggml_vocab_test__
56 Hello
57__ggml_vocab_test__
58 Hello
59__ggml_vocab_test__
60 Hello
61__ggml_vocab_test__
62 Hello
63__ggml_vocab_test__
64 Hello
65 Hello
66__ggml_vocab_test__
67 (
68__ggml_vocab_test__
69
70 =
71__ggml_vocab_test__
72' era
73__ggml_vocab_test__
74Hello, y'all! How are you 😁 ?我想在apple工作1314151天～
75__ggml_vocab_test__
76!!!!!!
77__ggml_vocab_test__
783
79__ggml_vocab_test__
8033
81__ggml_vocab_test__
82333
83__ggml_vocab_test__
843333
85__ggml_vocab_test__
8633333
87__ggml_vocab_test__
88333333
89__ggml_vocab_test__
903333333
91__ggml_vocab_test__
9233333333
93__ggml_vocab_test__
94333333333
95__ggml_vocab_test__
96Cửa Việt
97__ggml_vocab_test__
98 discards
99__ggml_vocab_test__
100
101
102
103
104
105
106
107
108
109
110
111🚀 (normal) 😶🌫️ (multiple emojis concatenated) ✅ 🦙🦙 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច😁 ?我想在apple工作1314151天～ ------======= нещо на Български ''''''```````""""......!!!!!!?????? I've been 'told he's there, 'RE you sure? 'M not sure I'll make it, 'D you like some tea? We'Ve a'lL
112__ggml_vocab_test__