disabled for now, consider adding tests for all CPU variants instead

Remove GGML_CUDA_CUB_3DOT2 flag once CCCL 3.2 is bundled within CTK and that CTK version is used in this project

Remove GGML_CUDA_CUB_3DOT2 flag once CCCL 3.2 is bundled within CTK and that CTK version is used in this project

add ssl support ; we will also need to modify win-build-sycl.bat to accept user-specified args

test on devices"

simplify the following workflows using a matrix

run lighter CI on PRs and the full CI only on master (if needed)

Remove GGML_CUDA_CUB_3DOT2 flag once CCCL 3.2 is bundled within CTK and that CTK version is used in this project

analyze performance impact, see https://spidermonkey.dev/blog/2025/01/15/is-memory64-actually-worth-using

abbreviations usage)_

add guidelines with examples and apply them to the codebase)_

Remove GGML_CUDA_CUB_3DOT2 flag once CCCL 3.2 is bundled within CTK and that CTK version is used in this project

this hangs for some reason ...

use list(APPEND LLAMA_COMMON_EXTRA_LIBS ...)

detect this based on current console

maybe convert enum llama_example to string

support arg with 2 values

Delete this when json_partial adds top-level support for null/true/false

Note that form.allow_toolcall_in_think is not tested yet. If anyone confirms it works, this comment can be removed.

Tool calling

rename to params

these can become expensive for long messages - how to optimize?

this is ugly, refactor it somehow

do we need to merge, or replacing is fine?

merge properly instead of overwriting (matching old behavior)

improve this later

if (has_raw_python)

support that mix in handlers below.

refactor this to "bool enable_thinking"

refactor this to "bool parse_reasoning"

windows + arm64 + mingw64

windows + arm64 + mingw64

move to common/sampling

fix naming

support threadpool)

repace embd_norm with an enum

maybe support multiline history entries?

maybe retry only on certain codes

use actual GET status?

cache the manifest response so that it appears in the model list

get GGUF size, not manifest size

handle lstrip/rstrip for comments? (not important for now)

tests can also be expressed like this: if x is eq 3

probably allow print value_none as "None" string? currently this breaks some templates

make sure this is the same behavior as Python's strftime

Support non-specified delimiter (split on consecutive (no leading or trailing) whitespace)

Support non-specified delimiter (split on consecutive (no leading or trailing) whitespace)

sorting is currently always case sensitive

sorting is currently always case sensitive

not sure if this is the right behavior

avoid circular references

avoid circular references

C++20 <=> operator

handle more unclosed top-level primitive if the stack was empty but we got an error (e.g. "tru", "\"", etc...)

use json_fwd.hpp when possible

support minimum, maximum, exclusiveMinimum, exclusiveMaximum at least for zero

Implement more comprehensive grammar generation for raw strings.

maybe throw an error instead?

maybe implement to_env() if needed

deduplicate with llama-impl.h

measure grammar performance

simplify

compute this from the vocab

measure grammar performance

track performance of most recent calls

optimize or pass from outside?

implement

noop

why do we squeeze here?

use Q4_K and Q6_K

Handle "sliding_attention" similarly when models start implementing it

should these be marked as UNUSED instead? (maybe not)

how to determine special FIM tokens automatically?

remove this once everyone has migrated to newer version of llama.cpp

multiply by the scale directly instead of inverting it twice

this is a hack, should be fixed

these special tokens should be exported only for the CodeGemma family

implement self.prediction_coefs.weight.clamp_(...)

does this really matter?

mimo v2 does not indicate the number of next-token-prediction layers, therefore we cannot do the same way as GLM4_MOE

Extend this if the prefix(es) need to be configurable

remove this once image support is implemented for Chameleon

remove once MXFP4 is supported more generally

remove this once everyone migrates to newer version of llama.cpp

uncomment U64, U32, and U16, ref: https://github.com/pytorch/pytorch/issues/58734

generate tokenizer tests for llama.cpp

this string has to exercise as much pre-tokenizer functionality as possible

add models here, base models preferred

add ellipsis in the type signature

make sure this is correct

support higher dimensional A shapes bigger than 1

compose the above two

match this with `llama_ftype`

rename to LLAMAFileType

move to `gguf.py`

verify this

reuse (probably move to gguf.py?)

Respect --vocab-dir?

Update to correct value when its been specified.

IBM XL */

Clang's output is still _much_ faster -- On an AMD Ryzen 3600,

support "uri", "email" string formats

support minimum, maximum, exclusiveMinimum, exclusiveMaximum at least for zero

print sampling/grammar timings for all clients

fix this

remove me

simplify this logic

simplify

print sampling/grammar timings for all drafts

mark all options as advanced when not GGML_STANDALONE

remove in the future

support for clang

convert to enum https://github.com/ggml-org/llama.cpp/pull/16187#discussion_r2388538726

temporary until model loading of ggml examples is refactored

when we start computing gradient, make a copy instead of view

when we start computing gradient, make a copy instead of view

when we start computing gradient, make a copy instead of view

this is very likely wrong for some cases! - needs more testing

needs to be adapted to ggml_flash_attn_ext

currently only lower, right, non-unitriangular variant is implemented

currently, only a few functions are in the base ggml API, while the rest are in the CPU backend

should not be set globally

probably these flags need to be tweaked on some architectures

better way to add external dependencies

backends cannot be safely unloaded without a function to destroy all the backend resources,

add a generic callback to the buffer interface

count the number of inputs instead of only checking when full

add public function to facilitate this, since applications do not have direct access to the backend interface

pass backend to the callback, then the user can decide if they want to synchronize

needs to be size*2 to account for leafs (do it in graph_split instead)

find the optimal value

performace is low.

check theta_scale_length and position_length.

acl_yarn_ramp_tensor use rope cache.

n_dims < ne0

ne0 != n_dims in mode2

If `ne12 > 1`, grouped multiplication and memory copying is used for efficiency.

each stream should have a memory pool.

add more device info later.

cann backend doesn't support quantized yet. Just leave the code

need handle tensor which has paddings.

refer to cann(#6017), it use thread's default stream.

Support 310p P2P copy

quantized type?

Support 310p P2P copy

this event is not effective with acl graph mode, change to use aclrtSynchronizeStream

support broadcast for ADD + RMS_NORM

Optimize here. Currently, we can only

support GGML_TYPE_BF16

Support rope_dim < ne00(dim)

add circular padding support for cann, see https://github.com/ggml-org/llama.cpp/pull/16985

support bias != 0.0f

support attention sinks [TAG_ATTN_SINKS]

support attention sinks [TAG_ATTN_SINKS]

padding to support

fix name to kvalues_iq4_nl

Separation to determine activation of VX/VXE/VXE2

not sure if correct (https://github.com/ggml-org/llama.cpp/pull/16315)

fix padding for vnni format

this is reference impl!

performance improvement: merge quant A

check if unrolling this is better

check if unrolling this is better

this does not check for OS support

can _mm256_mulhi_epu16 be faster even if 16-bits?

Use the 'traits' lookup table (for type conversion fns), instead of a mass of 'if' conditions with long templates

double-check these work correctly

move to ggml-threading

add support for explicit memory order

add support for explicit memory order

add support for explicit memory order

this is a bit of a hack, we should probably have a better way to handle this

extract to "extra_op"

this is a bit of a hack, we should probably have a better way to handle this

Windows etc.

get_rows can use additional threads, but the cost of launching additional threads

support > 64 CPUs

there seems to be no way to set lower prio on Apple platforms

this may not work on BSD, to be verified

this can become (n_tasks-1)

this can become (n_tasks-1)

this can become (n_tasks-1)

deep copy

move to ggml-base

this should check for __ARM_FEATURE_FP16_VECTOR_ARITHMETIC

support for transposed / permuted tensors

maybe this is not optimal?

support for transposed / permuted tensors

maybe this is not optimal?

templateify the implemenation and support for I64

support for transposed / permuted tensors

maybe this is not optimal?

smarter multi-theading

smarter multi-theading

smarter multi-theading

smarter multi-theading

optimize

optimize

optimize

optimize

add x parameter to ggml_vec_scale_f32 and remove this memcpy

handle transposed/permuted matrices

handle transposed/permuted matrices

is this supposed to be ceil instead of floor?

handle transposed/permuted matrices

optimize

on ARM, native f16 should be faster

transpose the output for smaller strides for big batches?

maybe unroll more?

what happens when (d_state % svcntw()) != 0?

optimize / multi-thread

optimize / multi-thread

Write SVE code and RVV code

handle transposed/permuted matrices

handle transposed/permuted matrices

add WASM SIMD

this branch seems wrong

generalise.

needs to be revisited

General batched mul mat for 4D tensors

is this optimal ?

is this optimal ?

Does this work?

is this optimal ?

is this optimal ?

Use the 'traits' lookup table (for type conversion fns), instead of a mass of 'if' conditions with long templates

optimize to process the remaining elements in groups using the smaller vector sizes from AVX2 and SSE

Write SVE code

Write SVE code

optimize performance

Remove once CCCL 3.2 has been released and bundled with CUDA Toolkit

might need to bail out if the HTP is stuck on something

handle errors

update profiling implementation, currently only works for opt_opsync mode

support broadcast for ne[2 and 3]

add support for non-contigiuos tensors

add support for non-contigiuos tensors

add support for sinks

add support for GGML_TYPE_F16 for src0

the current version might do incorrect reodering in cases where quantized src0

technically we don't need these and could use Q6_dmstart/wait/etc instead

might need to handle zero as a special case (see ggml-cpu code)

might need to handle zero as a special case (see ggml-cpu code)

might need to handle zero as a special case (see ggml-cpu code)

might need to handle zero as a special case (see ggml-cpu code)

might need to handle zero as a special case (see ggml-cpu code)

might need to handle zero as a special case (see ggml-cpu code)

use simd to speed up the remaining elements copy

do not use CUDA definitions for HIP

move to ggml.h? (won't be able to inline)

Consider allowing GGML_OP_NONE nodes in between

only add to the ggml-metal target?

for optimal performance, become function of the device and work size

this can be removed when the allocator starts filtering them earlier

make a simpler cpy_bytes kernel

relax this constraint in the future

helper function

determine the optimal parameters based on grid utilization

do not use CUDA definitions for MUSA

mudnn has not provided static libraries yet

initialize them for non SMALL_PATH path, or remove them.

add support

implement BF16, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, IQ4_NL support (https://github.com/ggml-org/llama.cpp/pull/14661)")

add circular padding support for opencl, see https://github.com/ggml-org/llama.cpp/pull/16985

if any unexpected results are seen, double check the offset -

use preallocated images instead of sub-buffer then image

find the optimal values for these

remove duplicate definitions of image description + format -- move to top

Unknown GPU");

add block_q4_0 variant.

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

Unknown GPU");

currently the output_size is always known, do we need support for commands with variable output size?

cache the alloc responses to avoid extra RPC calls?

obtain value from the server

call the remote backend and cache the results

adapt to hardwares

currently, it's not used for XMX really.

Downsample logic is separated from the kernel, a rewrite is desirable

Refactor and remove duplicates */

k-quants

do not crash if SYCL Buffer alloc fails

this is not thread safe

this is a hack to avoid having to implement a new buffer type

return device.maxBufferLength

check that src0->buffer->buft is a split buffer type, replace GGML_BACKEND_TYPE_GPU_SPLIT check

see https://github.com/ggml-org/llama.cpp/pull/13155

accuracy issues in MMQ

Refactor and cleanup of mul mat dispatching.

more efficient implementation

update for the new

support GGML_TYPE_BF16

keep a list of supported types to avoid breaking the backend when a new type is added

The configuration below needs more work to be supported with oneDNN

This specific configuration can fail with oneDNN and needs more debugging

add circular padding support for syscl, see https://github.com/ggml-org/llama.cpp/pull/16985

noncontigous inputs/outputs

currently not used

currently not used

We're no longer benefitting from the async compiles (shaders are

Use pointer or reference to avoid copy

staging_offset is not used

enable async and synchronize

error handling

handle multiple pipeline names

optional, needed?

optional, implement this

optional, think it coordinates with .init_tensor

for now, return maxBufferSize as both free and total memory

track need for these toggles: https://issues.chromium.org/issues/42251215

Maybe WebGPU needs a "fast" mode where you can request compilers skip adding checks like these,

support non-contiguous tensors, e.g. for MOE_EXPERT_REDUCE

implement support for quantized types

make thread-safe

Remove in the future as we are currently DLF16 -> FP32 then in the next op, FP32 -> DLF16 again. Inefficient.

Consider adding a ggml check.

If tensor = 4D, use ZDNN_NCHW by default.

If tensor = 2D, use ZDNN_NHWC by default.

required here for quantization functions

this should not be needed as long as we don't rely on aligned SIMD loads

support less-strict constraint

implement non F32 return

implement non F32 return

when implement backward, fix this:

implement antialias for modes other than bilinear

check if vT can be multiplied by (k*qT)

adapt to ggml_flash_attn_ext() changes");

check if vT can be multiplied by (k*qT)

maybe support other strides than 1?

support other variants

should probably be sum instead of mean

this branch isn't accessible anymore, maybe move this to ggml_build_forward_expand

use ggml-backend to obtain the tensor data

add GGMLFileType from ggml_ftype in ggml.h

need help with 64-bit types in Python

When/if _get_field_parts() support multi-dimensional arrays, this must do so too

add option to generate error on duplicate keys

Handle multi-dimensional arrays properly instead of flattening

cleaner way to get the first key

make this even more comprehensive

dict and set

maybe handle tensors in kwargs too

__array_function__

load adapter_config.json when possible, it usually contains the base model of the LoRA adapter

should word-based size labels always be removed instead?

should the basename version always be excluded?

these do not belong to block_mappings_cfg - move them to mappings_cfg

handle request errors (maybe with limited retries?)

internally store as the new format instead of converting to old

Verify that added tokens here _cannot_ overlap with the main vocab.

hf suffix which could be ignored but isn't

DPO in the name

should "base" be a 'finetune' or 'size_label'?

is a column-wise sum of squares appropriate?

show sample usage

remove, required until per token attributes are available from GGUF file

simplify (https://github.com/ggml-org/llama.cpp/pull/9294#pullrequestreview-2286561979)

consider SoA

rename this to "output"

update API to start accepting pointers to params structs (https://github.com/ggml-org/llama.cpp/discussions/9172)

rename to llama_get_pooling_type

rename to avoid confusion with llama_get_embeddings()

deprecate in favor of llama_get_logits_ith() (ref: https://github.com/ggml-org/llama.cpp/pull/14853#issuecomment-3113143522)

deprecate in favor of llama_get_embeddings_ith() (ref: https://github.com/ggml-org/llama.cpp/pull/14853#issuecomment-3113143522)

extend in the future

the check is failing for some reason:

add support for norm vector

a more general solution for non-CPU extra buft should be imlpemented in the future

pimpl

whole_seqs for embeddings?

support embeddings if needed in the future

this is more of a temporary solution until we have a better way to handle multiple positions per token/embd

start reading the actual value of mscale and handle the case where it is not 1.0f

move these checks to ggml_backend_sched

should we ignore ACCEL types too?

instead of the tensor names, use a map to keep track of which (FA) tensors belong to which layer

fa_device_mismatch logic is wrong for --no-kv-offload, but this is broken anyways

not sure if the following graph would be worster case for multi-stream KV caches:

if multiple single tokens are evaluated without a synchronization,

change the mctx->apply() to return information if a graph reserve is needed

use output_resolve_row()

use output_resolve_row()

not sure yet if we want to reserve here

should we reserve?

add new split mode where we pad the input sequences so that ubatch.equal_seqs == true

this clear of the buffer can easily be forgotten - need something better

this is a tmp solution until we have a proper way to support enc-dec models

hacky solution

avoid this workaround in the future

this clear of the buffer can easily be forgotten - need something better

is there something more efficient which also minimizes swaps?

hacky enc-dec support

also consider shrinking the buffer

not needed?

not sure if needed, might simplify in the future by removing this

fix in ggml_backend_sched

add more model-specific info which should prevent loading the session file if not identical

handle sampling buffers and samplers state ?

add more info which needs to be identical but which is not verified otherwise

handle sampling buffers and samplers state ?

handle this error");

better default

more flexible combinations of logical/physical batch size and context size

read/write lora adapters and cvec

tmp for handling cross-attention - need something better probably

remove, needed for tests atm

shared ptr

move the API below as member functions of llama_grammar

use ubatch->n_seqs instead of failing

need to move this to the unified cache and check there

need to move this to the unified cache and check there

need to move this to the unified cache and check there

use ubatch->n_seqs instead of failing

need to move this to the unified cache and check there

Hybrid input classes are a bit redundant.

need to move this to the unified cache and check there

need to move this to the unified cache and check there

Use scalar div instead when/if implemented

move to hparams?

add support for gated squared relu

needs more work to be correct, for now just use the tensor shape

if ubatch.equal_seqs() == true, we can split the three tensors below into ubatch.n_seqs_unq streams

remove

maybe separate the inner implementation into a separate function

Call llama_sampler_accept_ggml after all samplers have been applied.

tmp - need something better to pass the data from the encoder to the decoder

this needs more work to be correct, for now copy the embeddings data to host memory

needed by build_attn_mha, figure out a way to remove?

remove

move this implementation to llama_memory_recurrent.

better name

maybe support other convolution strides than 1

think of a better place for this function

pack the SWA params in a struct?

rename to llama_format ?

if we fail again, we should attempt different splitting strategies

add ggml helper function for this?

support multiple streams");

use ubatch->n_seqs instead of failing

we also need to save llama_kv_cell_ext when apply_ubatch() support loading it

we cannot yet restore llama_kv_cell_ext as the apply_ubatch() does not support it yet

add unit tests

non-sequential equal split can be done if using unified KV cache

will the recurrent cache be in an undefined context at this point?

non-sequential equal split can be done if using unified KV cache

will the recurrent cache be in an undefined context at this point?

non-sequential equal split can be done if using unified KV cache

optimize

would it be possible to resize the cache instead?

bake-in src refcounts in the cell metadata

llama_memory_recurrent should have a notion of max sequences

extract the cache state used for graph computation into llama_memory_recurrent_context_i

optimize for recurrent state needs

extract all the state like `head` and `n` here

consider moving to llama-impl.h if needed in more places

this is not very clever - figure out something better

make optional

implement split file support

implement LoRA support

Handle SWA metadata similarly when models start implementing it

become GGUF KV parameter

become GGUF KV parameter

become GGUF KV parameter

fix conversion scripts to correctly populate `n_swa` and `n_swa_pattern`

Jamba layers are a bit heterogenous, so naming this is hard.

when MTP is implemented, this should probably be updated if needed

add variants */

when MTP is implemented, this should probably be updated if needed

maybe add n_attn_temp_floor_scale as a separate KV?

move to a separate function

workaround for CPU backend buft having a NULL device

move reranking logic here and generalize

move this to new llm_arch_model_i interface

move this to new llm_arch_model_i interface

remove

avoid hardcoded tensor names - use the TN_* constants

explore better strategies

explore better strategies

use LLM_KV

use LLM_KV

avoid hardcoded tensor names - use the TN_* constants

use a symmetric type instead

temporary sanity check that the F16 -> MXFP4 is lossless

remove trigger_words support.

there are a lot of common parts between spm and bpe tokenizers, should be refactored and reused

reduce string copies by using cpts_offs array

should we set all of these to LLAMA_TOKEN_NULL?

remove, required until per token attributes are available from GGUF file

convert scripts should provide these tokens through the KV metadata LLM_KV_TOKENIZER_...

workaround for o200k_harmony and solar-open tokenizer: the "<|end|>" token should not be EOG

Extract attributes from GGUF file.

where do these characters come from?

do not use model.tok_embd directly, duplicate as model.output

this suppresses the output of image tokens, which is required to enable text-only outputs.

is causal == true correct? might need some changes

is causal == true correct? might need some changes

move this to right after the last KV layer

verify if this is the correct behavior in transformers implementation

skip computing output earlier for unused tokens

use semistructured matrices to implement state-space duality

skip computing output earlier for unused tokens

Only do the expert selection and weights once

can this ever be false?

if the model varies, these parameters need to be read from the model

is this correct?

remove in follow-up PR - move to .cpp files

reimplement this structure in endian-independent way

disabled on loongarch64 because the ggml-ci node lacks Python 3.8

disabled due to slowness

repair known memory leaks

Make a template or something

implement

add test with a non-contiguous view as input ; this case is needed for build_rope_2d in clip.cpp

this branch should become a separate test case parameter instead of hardcoding this for these head shapes

implement for all backends

or "other"

implement for all backends

add after WebGPU is fixed

better value for n_threads

biasing too much here makes the Vulkan sampling fail - should be investigated further

llama_chat_format_single will be deprecated, remove these tests later

extract to common helper (copied from test-grammar-integration.cpp)

The following line should fail, but currently it passes. `exclusiveMinimum` is not supported, as it would likely be too difficult to implement.

The following line should fail, but currently it passes. `uniqueItems` is not supported, as it would likely be too difficult to implement.

The following line should fail, but currently it passes. `uniqueItems` is not supported, as it would likely be too difficult to implement.

shold not include libllama sources

detect the true/false/null literal was complete

why is done twice?

((?:b)?a*+).* ??

show progress

reduce some copies here in the future

support remote files in the future (http, https, etc)

maybe support it later?

avoid using atexit() here by making `console` a singleton

one inconvenient of current chat template implementation is that we can't distinguish between user input and special tokens (prefix/postfix)

get rid of malloc if possible

get rid of this malloc if possible

customize padding token

enable Metal support when support for GGML_OP_SQRT is added

buf_size must be able to scale with params.n_batch

remove this when we can support merging subset of adapters. Ref: https://github.com/ggml-org/llama.cpp/pull/8607#discussion_r1686027777

add support for quantized lora

detect OS and use copy_file_range() here for better performance

extract into its own method; this is also used by the GGUF-based format

extract into its own method; this is also used by the legacy format

only get outputs when (params.process_output || params.compute_ppl)

there was a more efficient which relies on ggml_view and ggml_rope_ext_inplace, but the rope inplace does not work well with non-contiguous tensors ; we should fix that and revert back to the original implementation in https://github.com/ggml-org/llama.cpp/pull/13065

improve this later

support warmup size for custom token numbers

rename it to fc (fully connected layer)

q/k norm requires row size == n_embd, while here it's d_head

there was a more efficient which relies on ggml_view and ggml_rope_ext_inplace, but the rope inplace does not work well with non-contiguous tensors ; we should fix that and revert back to the original implementation in https://github.com/ggml-org/llama.cpp/pull/13065

verify the image_min_tokens

check kimivl preprocessor for exact values

this is a hack to support Yi-type llava

we don't support audio for Gemma 3N, but GGUF contains audio tensors

define the behavior for add_padding = false

this is only used by minicpmv, maybe remove it

remove this function

remove this function

should be enum, not string

Handle short audio differently or return error

probably unnecessary here? (or better doing it in g_cache?)

handle these checks better

maybe handle this better

support for --system-prompt with /clear command

maybe support batching, but this may come with memory cost

deprecate

deprecate

deprecate

deprecate

this could be made smaller; it's currently the worst-case size

don't evaluate the last token of each sequence

the last token of each of the sequences don't need to be evaluated

this could be made smaller; it's currently the worst-case size

end before the last token, no need to predict past the end of the sequences

this uses the wrong first logits when not skipping the choice word

don't evaluate the last token of each sequence

share with imatrix.cpp

list multiple datasets when there are more than one

use the base64::decode from base64.hpp)

add audio_url support by reusing handle_media()

test this properly */

The response format of this option is not yet OAI-compatible, but seems like no one really using it; We may need to fix it in the future

reuse llama_detokenize

optimize this block by reducing memory allocations and movement

make project name an input

current filename

configurable?)

server_tokens should be copyable - remove this:

move it to server-task.cpp

move it to server-task.cpp

move these to server-task.cpp

change to unique_ptrs for consistency:

move members that belong to the task (such as `generated_text`, `has_new_line`) to task_results_state

mtmd does not support prompt cache

improve logic

This will error out if a user requests two aloras, but only

speculative decoding requires multiple samples per batch - not supported yet

getting post/pre sampling logits is not yet supported with backend sampling

tmp until backend sampling is fully implemented

improve by not doing it more than once for each new line

optimize this with min-p optimization

simplify and improve

rework to have a single draft llama_context shared across all slots [TAG_SERVER_SPEC_REWORK]

maybe move branch to outside of this loop in the future

support memory-less logits computation

support can be added in the future when corresponding vision models get released

try to make this conditional on the context or the memory module, instead of the model type

try to terminate only the largest active slot/sequence and continue with the rest

update slot state based on llama_memory_seq_pos_min() and llama_memory_seq_pos_max()

handle ret == 2 (abort) when we start aborting

set it here instead of doing inside populate_token_probs

set result.probs

this log can become very long, put it behind a flag or think about a more compact format

this is inaccurate due to child tasks

get rid of this dynamic_cast

get rid of this dynamic_cast

this could maybe be multimodal.

maybe handle sink.write unsuccessful? for now, we rely on is_connection_closed()

move this to a virtual function once we have proper polymorphism support

remove this once we use HTTP client from download.h

maybe validate preset before rendering ?

allow refreshing cached model list

add support for this on web UI

add other fields, may require reading GGUF metadata

also add downloading state when the logic is added

deduplicate?

deduplicate?

implement

add more sanity checks for the input parameters

we may want to throw errors here, in case "el" is incorrect

for some reason we can't copy server_tokens, so we have to do this workaround

prevent including the whole server-common.h as we only use server_tokens

change this to more generic "response_format" to replace the "format_response_*" in server-common

implement

somehow reuse server_metrics in the future, instead of duplicating the fields

refactor in common/console

should not be a valid case

remove this once test_cache_vs_nocache_prompt is fixed

remove this once test_cache_vs_nocache_prompt is fixed

remove this once test_cache_vs_nocache_prompt is fixed

the result is not deterministic when using other slot than slot 0

remove this once test_cache_vs_nocache_prompt is fixed

find & add other lora adapters for this model

remove this once test_cache_vs_nocache_prompt is fixed

fix these (wrong results, either didn't respect decimal instruction or got wrong value)

Remove this polling once llama-server properly waits for the operation

start using log.h

remove me

potential opportunity to roll common stuff into common/console.cpp

reporting invalid_utf8 would be useful on non-Windows too.

this script is LLM-generated and probably very inefficient and should be rewritten

load from json

tokenization is slow for some reason - here is pre-tokenized input

not optimized at all

can be done once

all logits?

%s\n", __FILE__, __LINE__, message); \

Assert global.back.(width,height) == global.front.(width,height)

iswprint ch?

\r, \t, \v, \f, etc?

Reorder TB_CAP_* so more critical caps come first.

Harden against errors encountered mid-resize

iswprint ch?