llama.cpp AI LLM Engine — How It Works

Foreword

llama.cpp is a program (engine) for running large language models (LLM) on a computer or on mobile devices — on the processor (CPU) or the graphics card (GPU); I wrote the Offline AI Launcher application, which lets you use this same engine on Android.

Why this is needed:

• to get answers from neural network models (chat, text completion, code) without sending data to the cloud;
• the choice of C++ is needed for fast work with memory and hardware; the same code builds for Windows, Linux, macOS and Android.

Model "weights" are a huge set of numbers (millions or billions) on which all computations of the neural network are based: matrix multiplications, biases and so on. They are obtained during model training and saved to a file. During inference the engine only reads these numbers and applies them to the input data, without changing the weights themselves. Inference is the process of "querying" an already trained model: you provide text, and the model produces an answer step by step.

The engine works with the GGUF file format. GGUF is the format in which the saved model is stored:

• the "weights" (the numbers of the neural network);
• metadata (sizes, architecture type);
• the vocabulary (the "text ↔ numbers" correspondence for tokens).

In essence this is a single container file from which the engine reads everything it needs into memory. Quantization is supported (reducing the model size by storing numbers more coarsely) as well as hybrid computation (part on the CPU, part on the GPU). Models in GGUF format can be taken from Hugging Face.

The goal of this article is to walk step by step through how inference is arranged in llama.cpp: from loading the model to the appearance of the next token in the answer.

A token is a number corresponding to a piece of text (a word or part of a word); the model works only with numbers, and the vocabulary translates "text → tokens" and back.

The article is built as a scenario:

• first the preparation (initialization, model loading, context creation);
• then generation on request (tokenization, batch, decode, sampling).

Which models the engine works with:

• LLaMA (Meta);
• Qwen (Alibaba);
• Gemma (Google);
• Mistral;
• Phi and others.

The differences are in size, context length and details; in the code these are different hyperparameters and weight tensors. The overall scenario is one and the same.

Which projects use it:

• LM Studio;
• Ollama;
• GPT4All;
• Offline AI Launcher (running models on a smartphone);
• KoboldCpp;
• Text Generation WebUI.

Repository: github.com/ggml-org/llama.cpp. The code in the article is quoted from the master branch at commit 8887a48f, right after release b10736; in another version the details may differ.

How to read the article:

• the sections go in the order the scenario is executed;
• in each section, for every action there is a quote from the code and an explanation of what happens and why;
• the source files are indicated in the text;
• the article is aimed at an unprepared reader: all terms are explained in the glossary at the beginning of the article, all steps are supplied with code quotes and explanations.

Glossary of Terms

Batch — a set of tokens (or embeddings), positions and logit flags processed in one decode call. During Prefill the batch contains many prompt tokens; during Decode it contains one new token.

Backend — the computation "engine": CPU or graphics card (GPU). The scheduler distributes graph nodes across backends.

Decode — the generation stage for a single token: the batch holds one new token, the graph computes the logits for this position, and K and V are appended to the KV-cache.

Embedding — a vector of numbers into which a token is turned before being fed into the model layers; one row of the model's embedding matrix.

EOS (end of sequence) — a special "end of output" token; the application stops generation upon it.

GGUF — the model file format: header, metadata (key–value), tensor data. Quantization and mmap are supported.

Hyperparameters (hparams) — the numbers that define the model sizes: context length, number of layers, embedding size, number of attention heads and so on.

Inference — the process of getting an answer from the model: a prompt is supplied, and the model produces the next token step by step.

KV-cache — the cache of keys and values of the attention mechanism; it stores the already computed K and V for all previous positions so that they do not have to be recomputed at every step.

Logits — the model's "raw" scores for every token of the vocabulary before softmax; the sampler picks the next token from them.

Prefill — the prompt processing stage: the batch contains all (or many) prompt tokens, K and V are computed for them and written into the KV-cache.

Sampling — the choice of a single token from the logits (greedy, random with temperature/top_p and so on).

Tensor — a multidimensional array of numbers (model weights, embeddings, keys, values, logits and so on).

Token — an integer (ID) corresponding to a piece of text (a word or part of a word); the model works only with tokens.

Tokenizer — the vocabulary component that turns text into tokens (SPM, BPE and so on) and back.

Computation graph — a list of operations (matrices, additions, activations) and the links between them; the engine performs all the model computations according to it.

Ubatch — a part of the batch that is processed in one process_ubatch call. If the prompt is longer than n_ubatch, the batch is split into ubatches of n_ubatch tokens each; every ubatch is run through the model in turn.

What for:

• to limit peak memory consumption for a long prompt.

Scheduler (sched) — the GGML component that distributes the nodes of the computation graph across backends (CPU, GPU) and allocates buffers for the graph on those devices. When the graph is executed, the scheduler traverses the nodes in topological order and launches the operations on the selected devices.

Process Diagram: All Steps in Order

Below is the whole chain from starting the engine to the appearance of the next token in the answer. Each item is analyzed in detail further in the article, with code quotes and explanations.

Preparation (once at startup or when the model is changed):

• step 1: backend initialization (llama_backend_init)
• step 2: loading the model from a file (llama_model_load_from_file → the GGUF loader)
• step 3: determining the model type — the architecture the loader has already read from the GGUF key general.architecture decides which model class is created (llama_model_create)
• step 4: loading the hyperparameters — sizes, context (load_hparams)
• step 5: loading the vocabulary and the tokenizer (load_vocab)
• step 6: loading the weights into memory or onto the GPU (load_tensors)
• step 7: creating the inference context — KV-cache, scheduler (llama_init_from_model).

Generation (for each message and each new token in the answer):

• step 8: the prompt text arrives from the user
• step 9: tokenization — the text is turned into a sequence of tokens (llama_tokenize)
• step 10: forming the batch — the tokens are packed for one call (llama_batch_get_one or common_batch_add), and inside the call the batch is validated and laid out (balloc->init)
• step 11: llama_decode — the entry point of a generation step: the batch is checked and handed to the internal decode path (llama_context::decode)
• step 12: for a long prompt the batch is split into ubatches (memory->init_batch)
• step 13: each ubatch is run through the model: building the graph → execution on CPU/GPU (process_ubatch → build_graph → graph_compute)
• step 14: the logits are copied from the graph output into the context buffer — one row per position marked for output in the batch; in the generation loop that is only the last position
• step 15: sampling — one next token is chosen from the logits
• step 16: if the token ends generation (llama_vocab_is_eog — that covers EOS, EOT and the other end-of-generation tokens), the loop finishes; otherwise the token is converted to text and printed (llama_token_to_piece), a new batch is built from this single token and control goes back to step 11.

The General LLM Inference Process

The diagram above shows what happens after what. In terms of meaning, the process splits into two stages. The first is preparation: backend initialization, loading the model from GGUF (architecture, hyperparameters, vocabulary, weights) and creating the context. It is performed once at startup or when the model is changed. The second stage is generation: with every user message the text is turned into tokens, packed into a batch, run through the model (llama_decode), the next token is chosen from the logits, it is converted into text and printed; the loop repeats until the "end of output" token (EOS) or a limit. This loop repeats for every new message and for every new token in the answer. Below, each step of the diagram is analyzed separately: what exactly is called in the code, what happens and what may be non-obvious to an unprepared reader.

Repository Structure and Main Files

In the llama.cpp repository the main parts of the engine are spread across folders and files.

Why it is done this way:

• to separate responsibilities: file loading, the model, the vocabulary, the context and the batch are in different files;
• this makes it easier to search the code and debug it.

Below are listed the files that directly relate to model loading and inference; for each one it is stated what it contains and why it is needed.

Entry point and model loading:

• src/llama.cpp — this is where the functions llama_backend_init, llama_model_load_from_file, llama_model_load (static) live.

  What for: this is the "front door" into the engine: the application calls these functions to initialize the library and load the model; here too the loader is created and the model object is requested for the architecture read out of the file. The factory llama_model_create (it asks the loader via get_arch) is only called from here — it lives in src/llama-model.cpp. Then load_hparams, load_vocab and load_tensors are called in turn.

• src/llama-model-loader.cpp — the llama_model_loader class: opening the GGUF file, building the tensor index (weights_map), the get_weight and get_tensor_meta methods for looking a tensor up by name, and load_all_data for reading the tensor data itself.

  What for: the loader is needed so that, given a tensor name, one knows where its data lies in the file and how to read it or map it into memory (mmap); without it one cannot load the architecture, hyperparameters, vocabulary and weights step by step.

Model and vocabulary:

• src/llama-model.cpp — the llama_model class and its base implementation llama_model_base: the load_hparams, load_vocab and load_tensors methods, creating tensors and assigning buffers. The parts that differ from one architecture to another are split off into load_arch_hparams and load_arch_tensors, which each model family implements in its own file under src/models/.

  What for: the model object stores everything that was read from the file: the architecture type, the hyperparameters, the vocabulary and the weights themselves (tensors); the load_* methods fill in this data from the loader one after another. The architecture itself is decided earlier, when the model object is created: llama_model_create asks the loader for it (get_arch) and builds the class of the matching family.

• src/llama-vocab.cpp — the llama_vocab class, the implementation of llama_vocab::impl::load (loading the vocabulary from GGUF), tokenization and the reverse translation of tokens into text (tokenize, token_to_piece, detokenize).

  What for: the vocabulary is needed to turn text into numbers (tokens) on input and numbers back into text on output; without it the model can neither accept a prompt nor produce a readable answer.

Context and decode:

• src/llama-context.cpp — the llama_context class: creating the context (memory, scheduler, graph reservation), the decode method, process_ubatch, llama_get_logits_ith.

  What for: the context is the "working environment" of one generation session: it defines the context size, the KV-cache, the computation scheduler; the decode method runs the batch through the model and returns the logits.

• src/llama-graph.cpp — the building of the compute graph: the llm_graph_context class with build_inp_embd (the lookup of embeddings in the table), and the graph result llm_graph_result with its set_inputs and can_reuse methods.

  What for: the graph is what is actually computed on decode: this file assembles its input nodes, fills them with the ubatch data before the run and decides whether the previous graph can be reused instead of being built again.

• src/llama-batch.cpp — the llama_batch_allocr class and its init method (filling in positions and logit flags); the llama_batch structure itself is declared in include/llama.h.

  What for: the batch is a "package" of tokens for one decode call; the allocator checks the batch and, if fields are missing, fills them in (positions from memory, logits only for the last token) so that the calling code does not need to set everything manually.

• src/llama-kv-cache.cpp — allocation and update of the KV-cache and its init_batch (splitting into ubatches and reserving cache slots for them). The interface itself is declared separately, in src/llama-memory.h, because a model may use another kind of memory — a recurrent state or a hybrid — and each variant implements the same init_batch in its own file.

  What for: the KV-cache stores the already computed keys and values for all previous positions so that they do not have to be recomputed at every step; init_batch splits a large batch into ubatches of limited size so as not to overflow memory.

• src/llama-sampler.cpp — the samplers and the chain of them: llama_sampler_chain_init, llama_sampler_chain_add, llama_sampler_sample.

  What for: after decode the logits still have to be turned into a single token: the chain runs the samplers one after another and returns the chosen token.

Libraries and backends:

• ggml/ (a directory inside the repository — the library is vendored here, not pulled in as a submodule) — the computation graph (GGML), tensor types, the scheduler (ggml_backend_sched) and the CPU/GPU backends; the public headers are in ggml/include/, the backends in subdirectories of ggml/src/.

  What for: the graph describes which operations (matrix multiplications, activations and so on) to perform and in what order; the scheduler decides on which device (CPU or GPU) to compute each node of the graph.

• gguf (ggml/include/gguf.h and ggml/src/gguf.cpp — part of ggml, not a separate library) — reading and writing the GGUF format: the header, the metadata and the tensors.

  What for: the GGUF format defines how the header, the metadata and the tensor data lie in the file; these functions read them without manual byte parsing.

• include/llama.h — the API header for applications: declarations of llama_model_load_from_file, llama_context, llama_decode, llama_tokenize, llama_get_logits_ith, the samplers and so on.

  What for: the application includes this header and calls the declared functions without going into the internal files of the engine.

Main Types and Structures (For Reference)

To find your way around the code it is useful to know the main types. Below is what each type stores and what it is needed for.

Main structures:

• llama_model — the object of the loaded model. It stores: the hyperparameters (hparams), the vocabulary (vocab), the weight tensors, the list of devices (devices), the distribution of layers across CPU/GPU.

  What for: the model is everything that was read from the file and is needed for computations; one model object can be used for several contexts (several generation sessions).

• llama_context — the inference context. It contains: a reference to the model, the context parameters (cparams: n_ctx, n_batch, n_ubatch, n_threads and so on), the batch allocator (balloc), the KV-cache memory (memory), the scheduler (sched), the last built graph kept for reuse (gf_res_prev), the logits buffer.

  What for: the context is the "working environment" of one session: the context size, the cache of keys and values, the scheduler and the graphs for decode; on every request decode is called for exactly this context.

• llama_batch — an array of tokens (or embeddings), positions, sequence identifiers and logit flags.

  What for: one llama_decode call accepts one batch; it passes which tokens to process, at which positions and for which positions to return the logits (usually only for the last one).

• llama_model_loader — the GGUF loader. It contains: the metadata read out of the file (metadata), the tensor map (weights_map), the open files (files) and their memory mappings (mappings).

  What for: the loader lives only during model loading; through it the architecture, hyperparameters, vocabulary and tensors are read one after another; after loading it is not needed.

• ggml_context — the GGML graph context: the nodes and tensors of the graph are created in it.

  What for: the graph describes a sequence of operations (multiplications, activations and so on); all the graph nodes are created in one such context.

• ggml_backend_sched — the scheduler: the list of backends and the logic of distributing the graph nodes across devices.

  What for: the scheduler decides on which device (CPU or GPU) to execute each node of the graph, and allocates buffers for the graph on those devices.

What the main functions return:

• llama_model_load_from_file returns llama_model* or nullptr on error.

  What for: the application checks the pointer: if it is not nullptr, the model is loaded and the context can be created.

• llama_model_load — an internal helper inside src/llama.cpp, not part of the public header — returns a pair of a status and the model: 0 on success, -1 on error, -2 on cancellation through the progress callback.

  What for: on a negative status the model pointer that comes back is already null, so the calling code only logs the reason and returns nullptr to the application.

• llama_decode returns 0 on success. A positive value is a warning, not a fatal error: 1 means no free slot was found in the KV-cache for this batch (the batch has to be made smaller or the context larger), 2 means the call was aborted. -1 means the batch itself is invalid, anything below -1 is a fatal error.

  What for: from the return value the application understands whether decode succeeded and whether the logits can be read; on 1 it can retry with a smaller batch, and after 2 or a fatal error part of the batch has already landed in the context's memory, so how far it got has to be asked of the context separately.

• llama_tokenize returns the number of tokens written, but no more than the size of the buffer it was given. If the buffer is too small it returns a negative number, and its magnitude is how many tokens there would have been — so the usual pattern is to call it once with an empty buffer, allocate that many, and call it again.

  What for: to know how many elements of the token array are filled, and how big the array has to be.

• llama_get_logits_ith returns a pointer to an array of floats of size n_vocab — the logits of the i-th token of the last decode; for a non-negative i the argument is the token index inside the batch, which the context translates into an output row through output_ids, so a token whose logits were not requested gives NULL; a negative index counts among the output rows, so -1 is the last one. For an invalid index it returns NULL.

  What for: from this array the sampler chooses the next token (one number for each token of the vocabulary); in the loop it is almost always index -1, the position that has just been computed.

Step 1: Backend Initialization

Step 1 in the process diagram is backend initialization. Before loading the model and running inference, the application calls llama_backend_init() once. The call does three things: it starts the high-resolution timer that later measures loading and decode time, it creates and immediately frees an empty GGML context, and — the part that matters — it fills the backend registry if it is still empty, so that the engine has devices to compute on: the processor, and a graphics card if a backend for one is available. Without a registered backend the loading does not degrade quietly, it fails: the engine returns nullptr and writes that no backends are loaded. Below is the quote in full and in fragments (file src/llama.cpp).

// Step 1: the single entry point of engine initialization; called once at application startup
void llama_backend_init(void) {
    ggml_time_init();  // high-resolution timer; on Windows it must be initialized explicitly

    // needed to initialize f16 tables
    {
        struct ggml_init_params params = { 0, NULL, false };
        struct ggml_context * ctx = ggml_init(params);
        ggml_free(ctx);
    }

    // if nothing has registered yet — pull in the backends (CPU, CUDA, Metal, Vulkan ...);
    // this is what gives the engine devices to compute on
    if (!ggml_backend_reg_count()) {
        ggml_backend_load_all();
    }
}

What happens in the code step by step:

// Fragment 1: the timer is needed to measure the model loading time and the decode time later
ggml_time_init();

// Fragment 2: a temporary context with a zero-size memory pool, created and freed at once.
// The comment in the source calls this "initialize f16 tables", but in the current ggml the first
// ggml_init() only starts the timer — the block survives as a leftover and costs nothing
struct ggml_init_params params = { 0, NULL, false };
struct ggml_context * ctx = ggml_init(params);
ggml_free(ctx);

// Fragment 3: the meaningful part — if the registry is empty, load the backend libraries
// (CPU, CUDA, Metal, Vulkan and so on); without them there is nothing to compute on
if (!ggml_backend_reg_count()) {
    ggml_backend_load_all();
}

First the timer is switched on — later it measures the loading time and the decode time. Then a temporary GGML context with a zero-size pool is created and freed right away; the comment next to it in the source still says it initializes the f16 tables, but in the current ggml the first call to ggml_init only starts the timer, and the lookup tables for half precision are built later, when the CPU backend registers itself. The last block does the real work: if no backend has registered yet, ggml_backend_load_all looks for the backend libraries and registers the ones it finds. After this the engine knows which devices it can compute on and is ready to load the model and build the graph.

Step 2a: Loading the Model from a File (Entry Point)

Step 2 is loading the model from a file. It begins with the call to llama_model_load_from_file: the application passes the path to the .gguf and the parameters, and the engine returns a ready model object or nullptr on error.

What this function is for:

• it is the single entry point for loading a model from a file;
• the application passes the file path and the parameters, and the engine returns a ready model object or nullptr on error.

A quote from the code (file src/llama.cpp):

// Model loading entry point (step 2): the path to the .gguf and the loading parameters
struct llama_model * llama_model_load_from_file(
        const char * path_model,
        struct llama_model_params params) {
    // splits — the paths to the parts of a split model; empty here, the loader works them out itself
    std::vector<std::string> splits = {};
    // the three nullptrs and the FILE * are the other possible sources of a model
    // (GGUF metadata prepared in memory, and an already open file); only one source may be set
    return llama_model_load_from_file_impl(nullptr, nullptr, nullptr, path_model, splits, /*file*/ nullptr, params);
}

The function accepts the path to the model file (usually .gguf) and the loading parameters — the application says where to read the model from, how to read it, and how many layers to put on the graphics card. splits is always empty here: if the metadata says the model is split into several files, the loader derives the paths to the parts from the file name itself, which is why that name has to follow the pattern <name>-00001-of-00003.gguf. An explicit list of parts is passed through a separate entry point, llama_model_load_from_splits. Everything else — the check that a backend is registered, the default progress callback, the call to llama_model_load and the handling of its result — happens in llama_model_load_from_file_impl.

The loading parameters (llama_model_params) that matter for understanding:

• load_mode — how to read the file: mmap, mlock, direct input-output, or plain reading into a buffer.

  What for: mmap saves RAM and makes the start of loading faster, because the file is mapped instead of being copied; direct input-output gives a more predictable read speed on some disks and systems. The default is auto: the engine chooses mmap and falls back to ordinary reading if one of the devices cannot work with mapped memory. Inside the loader this single value is turned back into the two flags use_mmap and use_direct_io — that is the form it appears in further down.

• n_gpu_layers — how many layers to load onto the GPU (the rest onto the CPU).

  What for: so that part of the computations goes on the graphics card and part on the processor.

• progress_callback — the loading progress callback.

  What for: the application can show a progress bar or cancel the loading (return false).

• vocab_only — load only the vocabulary (without the weights).

  What for: when only the tokenizer is needed, without the heavy weights. The full list of parameters is in include/llama.h in the llama_model_params structure.

Step 2b: Preparing to Read the File

Step 2 (continued) — inside the entry point the preparation for reading the file is performed. In llama_model_load_from_file_impl the main preparation before reading the file is carried out.

Why it is done this way:

• before loading one needs to make sure that the model is being taken from exactly one source, that there is a backend for computations, and that progress display is set up;
• only after that the internal function llama_model_load is called, which reads the file step by step.

A quote of the beginning of the function (file src/llama.cpp):

static struct llama_model * llama_model_load_from_file_impl(
        struct gguf_context * metadata,
        llama_model_set_tensor_data_t set_tensor_data,
        void * set_tensor_data_ud,
        const std::string & path_model,
        std::vector<std::string> & splits,
        FILE * file,
        struct llama_model_params params) {
    // ... omitted: a guard that exactly one source is given — metadata, path_model or file
    ggml_time_init();  // timer for measuring the loading time

    // If we are loading not only the vocabulary — check that at least one backend is registered
    if (!params.vocab_only && ggml_backend_reg_count() == 0) {
        LLAMA_LOG_ERROR("%s: no backends are loaded. hint: use ggml_backend_load() or ggml_backend_load_all() to load a backend before calling this function\n", __func__);
        return nullptr;
    }

    unsigned cur_percentage = 0;
    // If the progress callback was not passed — substitute our own: print dots up to 100%
    if (params.progress_callback == NULL) {
        params.progress_callback_user_data = &cur_percentage;
        params.progress_callback = [](float progress, void * ctx) {
            unsigned * cur_percentage_p = (unsigned *) ctx;
            unsigned percentage = (unsigned) (100 * progress);
            while (percentage > *cur_percentage_p) {
                *cur_percentage_p = percentage;
                LLAMA_LOG_CONT(".");
                if (percentage >= 100) {
                    LLAMA_LOG_CONT("\n");
                }
            }
            return true;  // do not cancel the loading
        };
    }

The implementation is shared by all the public entry points, so it accepts every possible source of the model at once — ready-made GGUF metadata, a path (with the parts of a split model in splits), or an already open file — and it starts by checking that exactly one of them was actually passed.

What happens in this fragment and what for:

• ggml_time_init() — the timer is switched on.

  What for: so that later it is possible to measure how long the loading took.

• The check ggml_backend_reg_count() == 0 — whether there is at least one backend (CPU or GPU).

  What for: without a backend it will be impossible to perform the model computations; when loading only the vocabulary (vocab_only) a backend is not required.

• Setting up the progress callback — if the application did not pass its own, a default callback that prints dots is substituted.

  What for: the user sees that the loading is in progress; if desired one can pass one's own callback and show a progress bar or cancel the loading (return false).

• The model object itself is not created here. It appears one level down, inside llama_model_load, in the llama_model_create call — and only after the GGUF loader has been opened.

  What for: the class of the object depends on the architecture written in the file (LLaMA, Gemma, Qwen and so on), so the architecture has to be read out of GGUF first; the hyperparameters, the vocabulary and the tensors are then written into that object.

The rest of the function is just the call into the internal loader and the handling of its result (same file, the tail of llama_model_load_from_file_impl):

    // Internal loading: reads the file and builds the model (arch, hparams, vocab, tensors)
    const auto [status, model] = llama_model_load(metadata, set_tensor_data, set_tensor_data_ud, path_model, splits, file, params);
    GGML_ASSERT(status <= 0);
    if (status < 0) {
        if (status == -1) {
            LLAMA_LOG_ERROR("%s: failed to load model\n", __func__);
        } else if (status == -2) {
            LLAMA_LOG_INFO("%s: cancelled model load\n", __func__);
        }

        if (model) {
            llama_model_free(model);
        }
        return nullptr;
    }

    return model;
}

• The list of devices is built a level down as well, in llama_prepare_model_devices, which llama_model_load calls right after creating the model object; the result goes into model->devices.

  What for: it determines onto which accelerators the model layers will be distributed (see load_tensors). Only graphics devices get into this list — remote RPC servers first, then discrete graphics cards, and integrated ones only if no discrete card was found; the CPU is handled separately and is not part of it.

• llama_model_load — reads the file and fills in the model step by step: it creates the GGUF loader, determines the architecture and creates the model object for it, picks the devices, and then calls load_hparams, load_vocab and load_tensors in turn.

  What for: all the logic of reading GGUF and filling in the model is concentrated in one function.

• On a negative status the reason is logged (-1 — a loading error, -2 — cancellation by the progress callback) and nullptr is returned. The llama_model_free call in this branch is only a safety net: llama_model_load destroys the half-built model itself and never hands a pointer back on failure.

  What for: from the nullptr the application understands that the loading failed and does not use an incomplete model.

The progress callback is invoked inside load_tensors while reading every tensor: it is passed a number from 0.0 to 1.0 (the share of loaded data). If the callback returns false, the loading is interrupted and llama_model_load returns -2 (cancellation).

What for:

• the application can cancel a long loading or show a progress bar.

Steps 2 to 6 Together: Step-by-Step Loading of the Model from a File

Steps 2–6 are performed inside a single function llama_model_load: the loader is created (step 2), then the model object is created for the architecture the loader read (llama_model_create, step 3), and after it load_hparams (step 4), load_vocab (step 5) and load_tensors (step 6) are called in turn.

What the function llama_model_load is for: it performs the step-by-step loading of the model from the file: it creates the GGUF loader (opens the file and builds the tensor map), creates the model object of the right architecture, and then loads the hyperparameters, the vocabulary and the tensors one after another. The order matters: the architecture defines the set of keys in GGUF; the hyperparameters are read by these keys; the vocabulary is loaded taking the architecture into account; the tensors are created according to the known sizes and filled in from the file.

An abridged quote of the function (file src/llama.cpp, function llama_model_load):

// src/llama.cpp — llama_model_load(); blank lines and two try/catch wrappers removed for brevity
// Returns 0 on success, -1 on error, and -2 on cancellation via llama_progress_callback
static std::pair<int, llama_model *> llama_model_load(struct gguf_context * metadata, llama_model_set_tensor_data_t set_tensor_data, void * set_tensor_data_ud,
        const std::string & fname, std::vector<std::string> & splits, FILE * file, llama_model_params & params) {
    try {
        // Step 2: the loader opens GGUF, reads the header and the metadata, builds the tensor map (weights_map)
        llama_model_loader ml(metadata, set_tensor_data, set_tensor_data_ud, fname, splits, file, params.load_mode,
            params.check_tensors, params.no_alloc, params.load_mtp, params.kv_overrides, params.tensor_buft_overrides);
        ml.lazy.mode = params.lazy_mode;  // lazy tensor reading: some weights stay on disk and are read on demand
        ml.print_info();
        // Step 3: the architecture read by the loader selects the C++ class of the model
        std::unique_ptr<llama_model> model_ptr(llama_model_create(ml, params));
        // the list of devices (CPU, GPUs) this particular model will be laid out on
        bool ok = llama_prepare_model_devices(params, model_ptr.get());
        if (!ok) {
            return {-1, nullptr};
        }
        auto * model = dynamic_cast<llama_model_base *>(model_ptr.get());
        // ... elided: a null check that aborts if the model does not implement llama_model_base
        model->t_load_us = 0;
        time_meas tm(model->t_load_us);  // the loading time will be recalculated after the first eval, to include the page faults deferred by mmap
        model->t_start_us = tm.t_start_us;
        model->hparams.vocab_only = params.vocab_only;
        model->hparams.no_alloc   = params.no_alloc;
        // ... elided: each of the two calls below is wrapped in try/catch that rethrows with a prefixed message
        model->load_hparams(ml);  // Step 4: model sizes, context length, number of layers
        if (model->arch == LLM_ARCH_CLIP) {
            throw std::runtime_error("CLIP cannot be used as main model, use it with --mmproj instead");
        }
        model->load_vocab(ml);    // Step 5: the vocabulary and the tokenizer — text <-> tokens
        model->load_stats(ml);
        model->print_info();
        if (params.vocab_only) {
            LLAMA_LOG_INFO("%s: vocab only - skipping tensors\n", __func__);
            return {0, model_ptr.release()};
        }
        // Step 6: the model weights from the file into memory (or mmap) on CPU/GPU
        if (!model->load_tensors(ml)) {
            return {-2, nullptr};
        }
        return {0, model_ptr.release()};
    } catch (const std::exception & err) {
        LLAMA_LOG_ERROR("%s: error loading model: %s\n", __func__, err.what());
        return {-1, nullptr};
    }
}

Errors and cancellation of loading:

• on an error in any of the steps (creating the loader, llama_model_create, load_hparams, load_vocab, load_tensors) an exception is thrown; in the catch block the message is logged and -1 is returned;
• on cancellation via the progress callback (the callback returns false inside load_tensors) load_tensors returns false, no exception is thrown, but llama_model_load returns -2 (cancellation);
• on any failure the half-built model is destroyed right there: it is held in a unique_ptr which is handed over to the caller only on success, so llama_model_load_from_file_impl receives a null pointer, logs the reason and returns nullptr;
• this way the application can cancel a long loading via the callback and release resources correctly.

Step by step (what happens and what for):

• the loader llama_model_loader ml(...) is created — it opens the GGUF file, reads the metadata and builds weights_map.

  What for:

• without the loader one cannot read the architecture, hyperparameters, vocabulary and tensors from the file.

• ml.print_info() is called — it prints to the log the file format, the file type (quantization) and the file size with the bits-per-weight figure.

  What for:

• so that the user sees what kind of file is being opened.

• llama_model_create(ml, params) is called — the architecture the loader read (LLaMA, Gemma, Qwen and so on) selects the C++ class of the model, and the object is created.

  What for:

• the field names in GGUF and the set of tensors to create depend on the architecture; an unknown architecture is an error here.

• llama_prepare_model_devices is called — the list of devices (CPU and graphics cards) for this model is built and printed to the log.

  What for:

• it determines onto which devices the layers will later be laid out in load_tensors.

• only now the loading time is reset and the timer is started.

  What for:

• the figure measured here is not the final one: on the first eval llama_context recalculates the loading time from the saved t_start_us, so the page faults deferred by mmap — they only happen when the weights are actually touched — end up inside the count as well.

• model->load_hparams(ml) is called — we read the dimensions and parameters (context length, number of layers and the like).

  What for:

• to know the "shape" of the model and allocate memory for it.

• for CLIP an error will be thrown — it is used separately as a projector, not as the main model.

• model->load_vocab(ml) is called — we load the token vocabulary.

  What for:

• so that later text can be turned into numbers (tokens) and back during generation.

• load_stats(ml) copies the number of elements and the size in bytes from the loader into the model, and print_info() prints the architecture and all the hyperparameters to the log. If we are loading only the vocabulary (vocab_only == true), this is the exit point. Otherwise model->load_tensors(ml) is called — the model weights are read and laid out in memory. On success 0 is returned, on cancellation (the progress callback returned false) -2, on error -1.

The order of calls when loading the model (summary):

• llama_model_load_from_file → llama_model_load_from_file_impl;
• in impl: a check that the model source is given exactly once, the backend check, the default progress callback, and the call llama_model_load(metadata, set_tensor_data, set_tensor_data_ud, path_model, splits, file, params);
• inside llama_model_load: creating llama_model_loader, ml.print_info, llama_model_create, llama_prepare_model_devices, load_hparams, load_vocab, load_stats, print_info, and if needed load_tensors. All these steps are executed sequentially; on an error in any of them the loading is interrupted.

The order of the calls llama_model_create → load_hparams → load_vocab matters: the architecture defines the set of GGUF keys; the hyperparameters are read by these keys; the vocabulary is loaded taking the architecture into account (for example, the key names for the tokenizer). That is why vocab.load(ml, kv) is called exactly after load_hparams(ml) — by this moment both the architecture and the hyperparameters are already known, and the loader can correctly read the tokenizer type, the token lists and the merges.

Step 2c: The GGUF Loader — Opening the File and the Tensor Map

Step 2 (conclusion) (inside llama_model_load) — the GGUF loader is created: the file is opened, the header and the metadata are read, the tensor map is built.

What the GGUF loader is for: to open the model file, read the header and the metadata (without the weights themselves) and build a "map" — so that, given a tensor name, one knows where its data lies in the file and what size it is. Without this map one cannot later load the weights tensor by tensor. Tensors here are multidimensional arrays of numbers (matrices and vectors) that store the weights of the neural network; every model layer is several tensors, and the loader must know the name, size and offset in the file for each one.

Creating the loader is a call to the llama_model_loader constructor. The constructor itself lives in src/llama-model-loader.cpp; llama_model_load only calls it. Abridged, it looks like this:

// src/llama-model-loader.cpp - the constructor (step 2): opens the GGUF, reads header + metadata, builds the tensor map
llama_model_loader::llama_model_loader(
        struct gguf_context * meta,
        llama_model_set_tensor_data_t set_tensor_data,
        void * set_tensor_data_ud,
        const std::string & fname,
        std::vector<std::string> & splits,
        FILE * file,
        llama_load_mode load_mode,
        bool check_tensors,
        bool no_alloc,
        bool load_mtp,
        const llama_model_kv_override * param_overrides_p,
        const llama_model_tensor_buft_override * param_tensor_buft_overrides_p)
        : metadata(meta), set_tensor_data(set_tensor_data), set_tensor_data_ud(set_tensor_data_ud) {
    // ... the KV overrides, and load_mode turning into use_mmap / use_direct_io ...
    if (!fname.empty()) {
        struct ggml_context * ctx = NULL;
        struct gguf_init_params params = {
            /*.no_alloc = */ true,  // create empty tensors only, do not read the data blob
            /*.ctx      = */ &ctx,
        };
        metadata_ptr.reset(gguf_init_from_file(fname.c_str(), params));
        metadata = metadata_ptr.get();
        if (metadata == nullptr) {
            throw std::runtime_error(format("%s: failed to load model from %s", __func__, fname.c_str()));
        }
        // The architecture name (llama, qwen2 ...) decides every other GGUF key name
        get_key(llm_kv(LLM_KV_GENERAL_ARCHITECTURE), arch_name, false);
        llm_kv = LLM_KV(llm_arch_from_string(arch_name));
        files.emplace_back(new llama_file(fname.c_str(), "rb", use_direct_io));
        contexts.emplace_back(ctx);
        for (ggml_tensor * cur = ggml_get_first_tensor(ctx); cur; cur = ggml_get_next_tensor(ctx, cur)) {
            std::string tensor_name = std::string(cur->name);
            if (weights_map.find(tensor_name) != weights_map.end()) {
                throw std::runtime_error(format("invalid model: tensor '%s' is duplicated", ggml_get_name(cur)));
            }
            n_elements += ggml_nelements(cur);
            n_bytes    += ggml_nbytes(cur);
            weights_map.emplace(tensor_name, llama_tensor_weight(files.back().get(), 0, metadata, cur));
        }
        // ... the same loop again over the split files, when the model is sharded ...
    }
    // ... branches for an already open FILE * and for externally supplied metadata ...
    n_kv      = gguf_get_n_kv(metadata);
    n_tensors = weights_map.size();
    fver = (enum llama_fver) gguf_get_version(metadata);
    // ... guessing the file type and dumping the KV pairs to the log ...
}

In the constructor gguf_init_from_file reads the GGUF header and the metadata (without the weights themselves) — this is how the list of tensors and the key–value pairs (architecture, hyperparameters) are obtained without reading the heavy data into memory. The architecture name (general.architecture) is taken from the metadata — the names of the remaining fields in GGUF depend on it (different models have different keys). The file is opened for reading; later the tensor data will be read or mapped by the offsets from weights_map. From the list of tensors in GGUF, for each one the name, size and offset in the file are written into weights_map — later get_weight(name) finds the record in the map, and load_all_data reads (or, with mmap, points straight at) the bytes at that offset.

The llama_tensor_weight structure (an element of weights_map) holds three things: the index of the source file (a model can be split into several GGUF shards), the absolute offset of the tensor data in that file, and a pointer to the tensor in the GGUF context. The offset is computed once, when the record is created, as the start of the data section plus the tensor's offset inside it — and the same constructor checks that the whole tensor actually fits inside the file, so a truncated model is rejected here rather than during generation. The ml.get_key(...) method reads from the GGUF metadata the value for a key (the key name depends on the architecture — it is returned by kv(...)). This is how the loader obtains, for example, the architecture name, the tokenizer type, the hyperparameters.

After the loader is created, ml.print_info() is called in llama_model_load. A quote from the code:

// src/llama.cpp, inside llama_model_load, right after the loader is constructed:
ml.print_info();  // logs the GGUF version, the guessed file type and the total size of the weights

The gguf_init_from_file function (the GGUF library) opens the file and reads the header:

• the format version;
• the number of metadata keys;
• the number of tensors.

The metadata is read into the gguf_context (key–value pairs); the tensor data itself is not loaded at this step — only the names, types and offsets in the file. The constructor itself already logs the count of key–value pairs, the count of tensors, the file name and the GGUF version. The print_info method (src/llama-model-loader.cpp) adds three more lines:

• the GGUF format version;
• the file type — Q4_K - Medium and the like, guessed from the commonest tensor type unless the metadata states it outright;
• the total size of the weights, in MiB or GiB, together with the average number of bits per weight.

The GGUF Format (For Reference)

GGUF (GPT-Generated Unified Format) is a binary format for storing machine learning models.

What it is needed for:

• to store in one file both the model weights and the metadata (sizes, architecture type) and the vocabulary;
• from the header and the metadata the loader builds a "map" and then, on demand, reads the needed pieces of the file or maps them into memory (mmap).

Structure of the GGUF header:

• the magic number, the four bytes GGUF (format identification — by it one understands that this is GGUF);
• the format version (for compatibility when the format changes);
• the number of tensors (n_tensors);
• the number of metadata keys (n_kv).

The metadata is stored as an array of "key — value" pairs. For each pair the following are recorded:

• the key name (a string with its length written in front of it);
• the type of the value (string, number, boolean, array and so on);
• the value itself (architecture name, dimensions, RoPE parameters, tokenizer type, token lists and so on); for an array, its element type and element count come first, then the elements.

The tensors in the file come after the metadata. For each tensor the following are recorded:

• the name;
• the number of dimensions and the size along each of them (at most four; for example [n_embd, n_ff] for the feed-forward weights of one layer — every layer is a separate set of tensors, so the layer number is part of the name, blk.0.ffn_up.weight, and never a dimension);
• the element type (F32, F16, Q8_0, Q4_K and so on — from full precision to quantized formats);
• the offset at which the data begins, counted from the start of the tensor data block that follows the metadata.

Names like Q4_K_M describe a whole file, not a single tensor: they are a recipe that puts different tensors in different types.

Why the loader needs this: from this information it builds weights_map (the map "tensor name → where the data lies in the file") and, when the weights are actually needed, load_all_data reads or maps the corresponding region of the file.

The format version and the element types:

• the format version is set in the GGUF header.

  What for:

• when the format changes, the version tells the loader what it is dealing with. The current version is 3; a file with a newer or a no-longer-supported version is rejected outright, and the model does not load. Unknown metadata keys, by contrast, cost nothing — the loader asks only for the keys it knows about and never trips over the rest.

• from the tensor element type (F32, F16, Q8_0, Q4_K and so on) the loader knows how many bytes a block of elements takes — quantized types store elements in fixed-size blocks with a shared scale, not one by one — and how to interpret the data when copying into a buffer or when using mmap.

  What for:

• without this it is impossible to read or map the tensor data into memory correctly; different types have different sizes and different byte interpretation.

The tensor data types in GGUF define how to interpret the bytes: F32, F16, Q8_0, Q4_K and so on — from full precision to quantized formats. Quantization reduces the model size and speeds up computations at the cost of an approximate representation of the weights. When loading, the engine creates the tensors in the required format and copies or maps the data from the file into buffers on the CPU or the GPU.

Reading the GGUF metadata is done through the functions of the gguf library:

• gguf_get_n_kv — the number of keys;
• gguf_get_key — the key name by index;
• gguf_get_kv_type — the value type (string, number, array and so on);
• gguf_find_key — the index of a key by its name, or -1 if there is none;
• gguf_get_val_* — the value at that index, one function per type.

The model loader wraps this in the get_key(key, value) method taking the architecture into account: the key is converted into a field name in GGUF (for example, llama.embedding_length for LLaMA).

Step 3: Determining the Model Type (Architecture)

Step 3 is determining the model type (LLaMA, Gemma, Qwen and so on) from the architecture name in GGUF. The work is done by the factory function llama_model_create in src/llama-model.cpp. What this is needed for: the architecture decides which key names are read from the GGUF metadata, which tensors are created when the weights are loaded and how the computation graph is built; without it neither the hyperparameters nor the vocabulary can be read correctly. In llama.cpp this is not just a flag: every architecture is a separate C++ class, and the factory picks the one to instantiate. Quote (file src/llama-model.cpp, the function llama_model_mapping and the two llama_model_create overloads):

// src/llama-model.cpp — the architecture from GGUF decides which C++ class represents this model
static llama_model * llama_model_mapping(llm_arch arch, const llama_model_params & params) {
    switch (arch) {
        case LLM_ARCH_LLAMA:
            return new llama_model_llama(params);
        // ... one case per supported architecture ...
        default:
            throw std::runtime_error(std::string("unsupported model architecture: '") + llm_arch_name(arch) + "'");
    }
}

llama_model * llama_model_create(llm_arch arch, const llama_model_params & params) {
    llama_model * model = llama_model_mapping(arch, params);

    if (model != nullptr) {
        model->arch = arch;
        // ... check that the requested tensor split mode is implemented for this architecture ...
    }

    return model;
}

llama_model * llama_model_create(llama_model_loader & ml, const llama_model_params & params) {
    llm_arch arch = ml.get_arch();
    if (arch == LLM_ARCH_UNKNOWN) {
        throw std::runtime_error("unknown model architecture: '" + ml.get_arch_name() + "'");
    }

    return llama_model_create(arch, params);
}

What happens and what for:

• ml.get_arch() returns the llm_arch enum that the loader has already resolved from the general.architecture key of the GGUF metadata.

  What for: further on, this enum is used to choose the field names for the hyperparameters and the vocabulary (different architectures have different keys in GGUF).

• If the type is unknown (LLM_ARCH_UNKNOWN), an error is thrown.

  What for: the engine cannot work with an unknown architecture; the application will get an error message and will be able to report it to the user.

Where the architecture is actually read (file src/llama-model-loader.cpp): the loader's constructor does it once, immediately after opening the GGUF and before it indexes the tensors, so that everything else in the loader can already use the resolved value. get_arch() and get_arch_name() are then just accessors:

// src/llama-model-loader.cpp — the constructor resolves the architecture once, right after opening the GGUF
llama_model_loader::llama_model_loader(/* ... */) : metadata(meta), /* ... */ {
    // ... the GGUF file has just been opened into `metadata` ...

    get_key(llm_kv(LLM_KV_GENERAL_ARCHITECTURE), arch_name, false);  // the "general.architecture" string
    llm_kv = LLM_KV(llm_arch_from_string(arch_name));                // "llama" -> LLM_ARCH_LLAMA, "qwen2" -> LLM_ARCH_QWEN2

    // ... the tensor index (weights_map) is built from here on ...
}

// by the time anyone asks, the enum is already resolved — both accessors are trivial
std::string llama_model_loader::get_arch_name() const {
    return arch_name;
}

enum llm_arch llama_model_loader::get_arch() const {
    return llm_kv.arch;
}

What these calls do:

• get_key(llm_kv(LLM_KV_GENERAL_ARCHITECTURE), arch_name, false) — the value for the "general architecture" key (for example, llama, qwen2) is read from the GGUF metadata and written into arch_name.

  What for: without the architecture name one cannot choose the set of keys for the hyperparameters and the vocabulary.

• llm_arch_from_string(arch_name) — the string is turned into the llm_arch enum (llama → LLM_ARCH_LLAMA, qwen2 → LLM_ARCH_QWEN2).

  What for: by this enum the field names in GGUF for load_hparams and load_vocab are chosen further on, and so is the model class in llama_model_create. The read is optional (false), so a file without the architecture key yields LLM_ARCH_UNKNOWN, and the error is raised where the model is created, not here.

• Examples of enum values: LLM_ARCH_LLAMA, LLM_ARCH_GEMMA, LLM_ARCH_QWEN, LLM_ARCH_PHI3, LLM_ARCH_MISTRAL3, LLM_ARCH_CLIP and so on — the table currently holds about 150 named architectures. The GGUF key ids themselves (LLM_KV) are one shared set: the ones for the hyperparameters are templates such as %s.context_length, where the architecture name is substituted for %s, which is why the same key reads as llama.context_length for one model and as qwen3.context_length for another; the general ones, general.architecture and general.name among them, are plain literals with nothing to substitute.

  What for: without the right architecture the key names come out wrong, and the hyperparameters and the vocabulary cannot be read from GGUF. What is genuinely per-architecture is the handful of extra hyperparameters only that family has — those are read by its own load_arch_hparams under src/models/.

• The tensor names in GGUF depend on the architecture: for LLaMA they are blk.N.attn_q.weight, blk.N.attn_k.weight and so on; other models have their own prefixes and suffixes. Adding support for a new architecture means adding a value to the enum and its name to the name table, writing a new class under src/models/ that says which hyperparameters to read, which tensors to create and how to build the graph, and registering it in the factory; new GGUF keys are added only when the architecture needs a parameter none of the existing keys covers. The hyperparameters are used when creating the tensors in load_tensors (matrix sizes, the number of layers) and when creating the context (n_ctx, n_batch, the RoPE parameters and so on).

Step 4: Loading the Hyperparameters — Sizes and Context

Step 4 is loading the hyperparameters (model sizes, context length, number of layers and so on) from the GGUF metadata. The llama_model_base::load_hparams method fills in the hparams structure (the hyperparameters) from the GGUF metadata. What this is needed for: the hyperparameters define the "shape" of the model — the context length, the number of layers, the embedding size, the number of attention heads and the like; without them one cannot allocate memory for the tensors and build the computation graph. The keys shared by all architectures are read here; then the method calls load_arch_hparams, the hook each architecture implements in its own file under src/models/, which reads the parameters only that family has. A few fields are set after that hook returns, the RoPE type among them — it can only be worked out once the architecture has resolved its own parameters. A quote with the main keys (file src/llama-model.cpp, method llama_model_base::load_hparams):

// src/llama-model.cpp — llama_model_base::load_hparams(): the model's shape, read out of the GGUF metadata
void llama_model_base::load_hparams(llama_model_loader & ml) {
    const gguf_context * ctx = ml.metadata;

    // keep every non-array key-value pair as a string, for later reference
    for (int i = 0; i < gguf_get_n_kv(ctx); i++) {
        gguf_type type = gguf_get_kv_type(ctx, i);
        if (type == GGUF_TYPE_ARRAY) {
            continue;
        }
        const char * name = gguf_get_key(ctx, i);
        const std::string value = gguf_kv_to_str(ctx, i);
        gguf_kv.emplace(name, value);
    }

    ml.get_key(LLM_KV_GENERAL_NAME, name, false);

    // everything past this point is not vocab-related
    if (hparams.vocab_only || ml.get_arch() == LLM_ARCH_CLIP) {
        return;
    }

    // the architecture is the prefix of every key below: llama.context_length, qwen3.context_length, ...
    ml.get_key(LLM_KV_CONTEXT_LENGTH,   hparams.n_ctx_train);   // context length the model was trained on
    ml.get_key(LLM_KV_EMBEDDING_LENGTH, hparams.n_embd);        // embedding (hidden state) size
    ml.get_key(LLM_KV_BLOCK_COUNT,      hparams.n_layer_all);   // number of transformer layers
    GGML_ASSERT(hparams.n_layer_all > 0 && hparams.n_layer_all <= LLAMA_MAX_LAYERS);
    ml.get_key(LLM_KV_EXPERT_COUNT,     hparams.n_expert, false);  // MoE models only

    // ... the per-layer arrays are zero-filled here ...

    ml.get_key_or_arr(LLM_KV_FEED_FORWARD_LENGTH,  hparams.n_ff_arr,   hparams.n_layer(), false);  // FF size, per layer
    ml.get_key_or_arr(LLM_KV_ATTENTION_HEAD_COUNT, hparams.n_head_arr, hparams.n_layer(), false);  // attention heads, per layer

    // n_head_kv is optional, default to n_head
    hparams.n_head_kv_arr = hparams.n_head_arr;
    ml.get_key_or_arr(LLM_KV_ATTENTION_HEAD_COUNT_KV, hparams.n_head_kv_arr, hparams.n_layer(), false);  // KV heads (GQA)

    // rope_freq_base (optional)
    hparams.rope_freq_base_train = 10000.0f;
    ml.get_key(LLM_KV_ROPE_FREQ_BASE, hparams.rope_freq_base_train, false);

    // ... RoPE scaling, head sizes, sliding-window variants ...

    load_arch_hparams(ml);  // whatever only this architecture needs — src/models/<arch>.cpp

    // ... a few last fields are set after the hook, hparams.rope_type among them ...
}

What happens and what for:

• In the loop all key–value pairs from GGUF (except arrays) are read and saved into gguf_kv.

  What for:

• so that later, if needed, any field can be accessed by name.

• Then, using the keys that depend on the architecture, the hparams fields are filled in:

• n_ctx_train — the context length during training.

  What for:

• the size of the KV-cache and the maximum input length depend on it.

• n_embd — the embedding (hidden layer) size.

  What for:

• the sizes of the weight matrices depend on it.

• n_layer_all — the number of transformer blocks written in the file.

  What for:

• how many tensors to create in load_tensors depends on it; the number of layers actually built is hparams.n_layer(), which subtracts the extra multi-token-prediction blocks some models carry.

• n_ff_arr, n_head_arr, n_head_kv_arr — the feed-forward sizes and the number of attention heads per layer.

  What for:

• the sizes of the Q, K, V and feed-forward matrices depend on them.

• rope_freq_base_train and others — the RoPE (positional encoding) parameters.

  What for:

• they define how the positional encodings are applied in the graph. As a result hparams fully describes the sizes of the model — this is enough to allocate memory for the tensors and build the computation graph.

Some of the GGUF keys for the hyperparameters (depending on the architecture):

• llama.context_length — the context length during training;
• llama.embedding_length — the size of the hidden layer (the embedding);
• llama.block_count — the number of transformer layers;
• llama.attention.head_count — the number of attention heads;
• llama.attention.head_count_kv — the number of KV heads (for GQA);
• llama.feed_forward_length — the size of the intermediate feed-forward layer;
• llama.rope.freq_base — the base RoPE frequency.

For MoE models the keys for the number of experts and so on are added. The get_key_or_arr method reads either a single value or an array per layer (when the sizes differ across layers). After the hyperparameters are loaded, the model knows the "shape" of all the tensors — this is enough to allocate buffers and build the graph in load_tensors and when creating the context.

Step 5: Loading the Vocabulary and the Tokenizer

Step 5 is loading the vocabulary and the tokenizer: by the vocabulary, text is turned into tokens on input and back into text on output. The token vocabulary is a correspondence table between pieces of text and integers (tokens); by it text is split into tokens on input and assembled back into text on output. Loading the vocabulary is performed after the architecture has been determined and the hyperparameters read: in llama_model_load the architecture is settled when the model object is created (llama_model_create asks the loader for it via ml.get_arch()), then load_hparams(ml) is called, and only then load_vocab(ml). Inside load_vocab, vocab.load(ml, kv) is called. It is loaded in llama_model_base::load_vocab (src/llama-model.cpp):

void llama_model_base::load_vocab(llama_model_loader & ml) {
    const auto kv = LLM_KV(arch);

    vocab.load(ml, kv);
}

For the current architecture the set of vocabulary keys is taken, after which vocab.load(ml, kv) is called. The implementation is the llama_vocab::impl::load method in src/llama-vocab.cpp. The following are read from GGUF:

• the tokenizer type (SPM, BPE, WPM and so on);
• the token lists;
• the token types;
• the BPE merges (if present);
• the special tokens — everything needed to turn text into a sequence of numbers (tokens) and back.

After this the model is able to tokenize the prompt and translate the generated tokens back into text.

The vocab.load(ml, kv) call is performed inside llama_model_base::load_vocab after the set of keys kv has been obtained for the current architecture (via LLM_KV(arch)). The loader ml has already opened GGUF and read the metadata; the architecture is known from the loader (ml.get_arch()) and the hyperparameters have been read by load_hparams. The beginning of the llama_vocab::impl::load implementation (file src/llama-vocab.cpp):

void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) {
    struct gguf_context * ctx = ml.metadata;

    // determine vocab type
    {
        ml.get_key(LLM_KV_TOKENIZER_MODEL, tokenizer_model);
        ml.get_key(LLM_KV_TOKENIZER_PRE,   tokenizer_pre, false);
        // ... "no_vocab" / "none" -> LLAMA_VOCAB_TYPE_NONE and an early return ...
        if (tokenizer_model == "llama") {
            type = LLAMA_VOCAB_TYPE_SPM;
            // default special tokens
            special_bos_id  = 1;
            special_eos_id  = 2;
            special_unk_id  = 0;
        } else if (tokenizer_model == "gpt2" || tokenizer_model == "hybriddna" || tokenizer_model == "whitespace") {
            type = LLAMA_VOCAB_TYPE_BPE;
            // read bpe merges and populate bpe ranks
            const int merges_keyidx = gguf_find_key(ctx, kv(LLM_KV_TOKENIZER_MERGES).c_str());
            // ... a wrongly typed merges array is an error; a missing one is too, unless
            //     tokenizer_pre == "kimi-k2"; each entry "first second" -> bpe_ranks[{first, second}] = i ...
        }
        // ... "bert" -> WPM, "t5" -> UGM, "rwkv", "plamo2", "gemma4"; anything else is an error ...
    }
    const int token_idx = gguf_find_key(ctx, kv(LLM_KV_TOKENIZER_LIST).c_str());
    if (token_idx == -1) {
        throw std::runtime_error("cannot find tokenizer vocab in model file\n");
    }
    // ... the scores and token type arrays are located here and checked to cover every token ...
    const uint32_t n_tokens = gguf_get_arr_n(ctx, token_idx);
    id_to_token.resize(n_tokens);
    for (uint32_t i = 0; i < n_tokens; i++) {
        std::string word = gguf_get_arr_str(ctx, token_idx, i);
        // ... an empty word is replaced by a "[EMPTY_<i>]" placeholder ...
        token_to_id[word] = i;
        max_token_len = std::max(max_token_len, (int) word.size());
        auto & token_data = id_to_token[i];
        token_data.text = std::move(word);
        token_data.attr = LLAMA_TOKEN_ATTR_NORMAL;
        // ... score and attr are then taken from the GGUF arrays when the file has them ...
    }
    init_tokenizer(type);
    // ... the newline token, the special token ids and the add_bos / add_eos flags follow ...
}

In the code one can see: determining the vocabulary type by tokenizer_model (llama → SPM, gpt2 → BPE with reading of the merges), searching for the token array in GGUF, a loop over all tokens — filling in id_to_token and token_to_id, the call to init_tokenizer(type) to initialize the tokenizer (SPM, BPE and so on). Everything hidden behind the // ... lines is checking: the merges array and the token array must be present and of the right GGUF type, and the scores and token-type arrays must cover every token — otherwise loading is aborted with an exception. The one relaxation is made for the kimi-k2 pre-tokenizer: it tokenizes without the usual BPE merges, so a missing merges array is not an error there — the loader simply writes a line about it to the log. Further in the same function the special tokens (BOS, EOS, UNK and the like) are read and the add_bos, add_eos flags are set.

Special tokens:

• BOS (begin of sequence) — the beginning-of-sequence token, added before the prompt if needed.
• EOS (end of sequence) — the end-of-output token, upon which the application stops generation.
• UNK (unknown) — the token for unknown characters or those not present in the vocabulary.

Their IDs are stored in special_bos_id, special_eos_id, special_unk_id. In GGUF the vocabulary may have keys such as tokenizer.ggml.bos_token_id, tokenizer.ggml.eos_token_id and so on — they are read at the end of llama_vocab::impl::load and written into the corresponding fields. The add_bos and add_eos flags define whether these tokens should be added automatically during tokenization (it depends on the model and the chat format). After init_tokenizer(type) the vocabulary is ready for tokenization and for the reverse translation of tokens into text; these operations are used on every user request.

Step 6: Loading the Weights into Memory or onto the GPU

Step 6 — the model weights (tensors) are loaded from the file into memory or onto the GPU. The llama_model_base::load_tensors method (src/llama-model.cpp) creates the model tensors, assigns buffers to them — regions of memory on the CPU or the graphics card — and fills them with data from the GGUF file. Below is the beginning of the method, abridged.

// src/llama-model.cpp — llama_model_base::load_tensors()
bool llama_model_base::load_tensors(llama_model_loader & ml) {
    const auto & split_mode   = params.split_mode;
    const bool use_mlock      = params.load_mode == LLAMA_LOAD_MODE_MLOCK || params.load_mode == LLAMA_LOAD_MODE_MMAP_MLOCK;
    const auto & tensor_split = params.tensor_split;
    const int n_layer_all  = hparams.n_layer_all;
    const int n_gpu_layers = this->n_gpu_layers();   // a negative value in the params means "all layers"
    // ... with LLAMA_LOAD_MODE_AUTO, ml.use_mmap is switched off if some device cannot mmap
    // build a list of buffer types for the CPU and GPU devices
    pimpl->cpu_buft_list = make_cpu_buft_list(devices, params.use_extra_bufts, params.no_host);
    for (const auto & dev : devices) {
        buft_list_t buft_list = make_gpu_buft_list(dev.dev, split_mode, tensor_split);
        // add CPU buffer types as a fallback
        buft_list.insert(buft_list.end(), pimpl->cpu_buft_list.begin(), pimpl->cpu_buft_list.end());
        pimpl->gpu_buft_list.emplace(dev.dev, std::move(buft_list));
    }

    ggml_backend_dev_t cpu_dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU);
    // ... splits[] is filled from tensor_split, or from the free memory of every device, then normalised

    const int i_gpu_start = std::max(n_layer_all + 1 - n_gpu_layers, 0);
    const int act_gpu_layers = devices.empty() ? 0 : std::min(n_gpu_layers, n_layer_all + 1);
    auto get_layer_buft_list = [&](int il) -> llama_model::impl::layer_dev {
        // ... a per-layer LLAMA_LOG_DEBUG line omitted
        if (il < i_gpu_start || (il - i_gpu_start) >= act_gpu_layers) {
            return {cpu_dev, &pimpl->cpu_buft_list};
        }
        const int layer_gpu = std::upper_bound(splits.begin(), splits.begin() + n_devices(), float(il - i_gpu_start)/act_gpu_layers) - splits.begin();
        auto * dev = devices.at(layer_gpu).dev;
        return {dev, &pimpl->gpu_buft_list.at(dev)};
    };

    // there is very little benefit to offloading the input layer, so always keep it on the CPU
    pimpl->dev_input = { cpu_dev, &pimpl->cpu_buft_list };

    // assign the repeating layers to the devices according to the splits
    pimpl->dev_layer.resize(n_layer_all);
    for (int il = 0; il < n_layer_all; ++il) {
        pimpl->dev_layer[il] = get_layer_buft_list(il);
    }
    // assign the output layer
    pimpl->dev_output = get_layer_buft_list(n_layer_all);

What happens at the beginning of load_tensors and what for:

• lists of buffer types are formed for the CPU and for each graphics card

  What for: they determine onto which devices the model tensors will be placed — the input ones usually on the CPU, the layers according to the CPU/GPU split;

• from the settings (tensor_split) or from the free memory it is decided which layers to compute on the CPU and which on the GPU

  What for: to load the devices evenly and not to overflow the memory of one graphics card.

Note that mmap and mlock are no longer separate boolean parameters: llama_model_params carries a single load_mode field (auto, none, mmap, mlock, mmap+mlock, dio), and the command line sets it with -lm / --load-mode; the older --mmap, --no-mmap and --mlock switches map onto it.

The get_layer_buft_list(il) function returns, for a layer number, the device and the list of buffers: the input is always on the CPU, the repeating layers may be on the CPU or the GPU, the output according to the split. Further on, the tensors are created in a loop: the embeddings, the output, and for each layer the attention matrices (Q/K/V/O), the normalizations, the feed-forward. Creating a tensor at this stage only fixes its shape and picks the buffer it will live in — no bytes are moved yet. The data comes in a second pass, after every tensor has been declared and the backend buffers allocated: load_tensors then calls ml.load_all_data(...) once per buffer context, and it is inside that call that the choice between mmap and an ordinary read is made. As a result the model weights end up in memory (and on the graphics card if needed), and the model is ready to generate text.

The loader itself (src/llama-model-loader.cpp) exposes the two halves of this. create_tensor declares one tensor: it checks the shape against the GGUF metadata, chooses a buffer type according to the device split worked out above, and creates the tensor in the matching ggml context. load_all_data then walks every tensor of a context, looks its record up in weights_map by name to get the file index and the byte offset, and either points the tensor at the memory mapping or reads the bytes into its buffer. For large models mmap is used so as not to duplicate the data in RAM.

The tensor declaration pass, for the llama family — the load_arch_tensors method in src/models/llama.cpp:

// src/models/llama.cpp — llama_model_llama::load_arch_tensors(), called from load_tensors()
void llama_model_llama::load_arch_tensors(llama_model_loader &) {
    LLAMA_LOAD_LOCALS;   // brings n_embd, n_layer, n_ff, n_vocab and the rest into scope

    tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);

    // output
    output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
    output      = create_tensor(tn(LLM_TENSOR_OUTPUT,      "weight"), {n_embd, n_vocab}, TENSOR_NOT_REQUIRED);

    // if output is NULL, init from the input tok embed
    if (output == NULL) {
        output = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, TENSOR_DUPLICATED);
    }

    for (int i = 0; i < n_layer; ++i) {
        auto & layer = layers[i];

        layer.attn_norm = create_tensor(tn(LLM_TENSOR_ATTN_NORM, "weight", i), {n_embd}, 0);

        create_tensor_qkv(layer, i, n_embd, n_embd_head_k * n_head, n_embd_k_gqa, n_embd_v_gqa, 0);
        layer.wo = create_tensor(tn(LLM_TENSOR_ATTN_OUT, "weight", i), {n_embd_head_k * n_head, n_embd}, 0);

        // ... optional bias tensors and the RoPE factor tensors omitted

        layer.ffn_norm = create_tensor(tn(LLM_TENSOR_FFN_NORM, "weight", i), {n_embd}, 0);

        if (n_expert == 0) {
            layer.ffn_gate = create_tensor(tn(LLM_TENSOR_FFN_GATE, "weight", i), {n_embd,   n_ff}, 0);
            layer.ffn_down = create_tensor(tn(LLM_TENSOR_FFN_DOWN, "weight", i), {  n_ff, n_embd}, 0);
            layer.ffn_up   = create_tensor(tn(LLM_TENSOR_FFN_UP,   "weight", i), {n_embd,   n_ff}, 0);
            // ... optional MLP bias tensors omitted
        }
        // ... the mixture-of-experts branch omitted
    }
}

What happens here and what for:

• every weight of the model is named through tn(...), which builds the GGUF tensor name — token_embd.weight, blk.0.attn_norm.weight, blk.0.ffn_down.weight and so on.

  What for: the name is the key by which the loader later finds this tensor's offset in the file.

• create_tensor checks the declared shape against the metadata, picks the buffer type from the layer split computed earlier, and creates the tensor in the ggml context of that buffer type.

  What for: this is where it is decided that a layer lives on the CPU or on a particular card — before any data is touched.

• tensors flagged TENSOR_NOT_REQUIRED may legitimately be absent, and TENSOR_DUPLICATED reuses an already created tensor — that is how a model without a separate output matrix ties its output back to the token embeddings.

  What for: one loading routine has to cover many variants of the same architecture.

The data pass itself is a loop over the tensors of one buffer context inside ml.load_all_data (src/llama-model-loader.cpp). For every tensor its record is looked up in weights_map by the GGUF name — blk.0.attn_q.weight, blk.0.attn_k.weight and so on — which yields the file index and the byte offset. Then it goes one of two ways. With a mapping the tensor is pointed straight at the mapped pages when a buffer can be built over them — the copy-free case, which is what host memory gets; otherwise the bytes are copied out of the mapping into the tensor's buffer. Without a mapping they are read from the file into that buffer, and for a buffer on a graphics card uploaded to the device. After this pass the model weights are fully in memory (and on the GPU if needed).

// src/llama-model-loader.cpp — the per-tensor loop of llama_model_loader::load_all_data()
for (struct ggml_tensor * cur = ggml_get_first_tensor(ctx); cur != NULL; cur = ggml_get_next_tensor(ctx, cur)) {
    const auto * weight = get_weight(ggml_get_name(cur));
    if (weight == nullptr) {
        // this can happen with split experts models
        continue;
    }
    // ... progress_callback is called here; returning false aborts the load
    size_t n_size = ggml_nbytes(cur);
    const bool from_mapping = use_mmap || lazy.has(cur);

    if (from_mapping) {
        const auto & mapping = mappings.at(weight->idx);
        ggml_backend_buffer_t buf_mmap = nullptr;
        if (bufs.count(weight->idx)) {
            buf_mmap = bufs.at(weight->idx);
        }
        uint8_t * data = (uint8_t *) mapping->addr() + weight->offs;
        // ... the optional check_tensors validation is omitted
        GGML_ASSERT(buf_mmap || cur->data); // either we have a buffer to allocate the tensor in, or it is already allocated
        if (buf_mmap && cur->data == nullptr) {
            // no copy at all: the tensor is pointed straight at the mapped pages
            ggml_backend_tensor_alloc(buf_mmap, cur, data);
            // ... mlock growth and the used-range bookkeeping are omitted
        } else {
            ggml_backend_tensor_set(cur, data, 0, n_size);
        }
    } else {
        const auto & file = files.at(weight->idx);
        if (ggml_backend_buffer_is_host(cur->buffer)) {
            file->seek(weight->offs, SEEK_SET);
            file->read_raw(cur->data, n_size);
        } else {
            // ... into a GPU buffer: chunked async uploads through pinned host memory when the
            // ... device supports them, otherwise a read into read_buf + ggml_backend_tensor_set()
        }
    }
    size_done += n_size;
}

So the two passes together cover everything: create_tensor decides where each weight will live — the embeddings, the Q/K/V/O matrices, the normalisations and the feed-forward of every layer — and load_all_data puts the bytes there, or maps them in.

When mmap is used, the tensor data is not copied into RAM — a region of the file is mapped and the tensor points straight into it — but only where the device can build a buffer over host memory. The processor always can; on unified memory such as Apple Silicon the graphics device can too. A discrete card cannot: its bytes are copied out of the mapping into video memory, so there mmap saves RAM only for the layers that stayed on the processor. This reduces RAM consumption and speeds up the start of loading, but the file has to stay in place on disk, unchanged, for as long as the model is loaded: the mapping outlives the loader and is released only when the model is freed. The direct-I/O mode (--load-mode dio) opens the file unbuffered and rounds every read to the block size the file system reports; it is implemented on Linux only and falls back to buffered reads when the unbuffered open fails. The split across layers (tensor_split or by the free GPU memory) defines which layers to load onto which device — this way a large model can be distributed across several graphics cards. After load_tensors finishes, the model is fully ready for inference: all the weights are in memory (and on the GPU if needed), and the context can be created and llama_decode can be called.

Step 7: Creating the Inference Context

Step 7 is creating the inference context (KV-cache, scheduler, reserved graphs). The loaded model stores only the weights (tensors). To generate text, an inference context is needed — the llama_context object.

What it is needed for:

• the context is the "working environment" of one generation session: it defines how many tokens the model can "remember" (the context size), how large the portions of data supplied are (the batch size), the RoPE parameters and the attention type;
• without a context one cannot call llama_decode — it is exactly the context that stores the KV-cache, the scheduler and the reserved graphs.

The implementation is in the file src/llama-context.cpp.

What is created when the context is created and what for:

• The scheduler (sched) — decides on which device (CPU or graphics card) to execute each node of the computation graph.

  What for: to distribute the computations across the CPU and the GPU and to allocate buffers for the graph on the required devices.

• The KV-cache memory (memory) — the buffers for the keys and values of the attention mechanism.

  What for: they store the already computed keys and values for all previous positions; without the cache, for every new token one would have to recompute K and V for the whole history, which is very slow (more details in the "KV-cache" section).

• The batch allocator (balloc) — fills in the positions and logit flags in the batch when needed.

  What for: so that the calling code does not have to set the positions manually and decide for which positions to compute the logits.

• The reserved graphs (Prefill and Decode) — temporary graphs and the buffers for them.

  What for: on the first decode call memory for the graph is not allocated "on the fly" — the buffers have already been reserved for the worst case, and this reduces the latency of the first answer (the graph itself is still built anew on that first decode).

The beginning of the constructor llama_context::llama_context in src/llama-context.cpp (abridged — the elisions are marked):

// src/llama-context.cpp — llama_context::llama_context (abridged)
llama_context::llama_context(
        const llama_model & model,
              llama_context_params params) :
    model(model),
    cvec(std::make_unique<llama_adapter_cvec>()),
    loras(std::make_unique<llama_adapter_loras>()),
    balloc(std::make_unique<llama_batch_allocr>(model.hparams.n_pos_per_embd())) {
    LLAMA_LOG_INFO("%s: constructing llama_context\n", __func__);

    t_start_us = model.t_start_us;
    t_load_us  = model.t_load_us;

    const auto & hparams = model.hparams;

    cparams.n_seq_max = std::max(1u, params.n_seq_max);
    if (cparams.n_seq_max > LLAMA_MAX_SEQ) {
        throw std::runtime_error("n_seq_max must be <= " + std::to_string(LLAMA_MAX_SEQ));
    }

    // ... n_rs_seq and its clamp to 0 for architectures without recurrent rollback
    cparams.n_threads               = params.n_threads;
    cparams.n_threads_batch         = params.n_threads_batch;
    cparams.yarn_ext_factor         = params.yarn_ext_factor  >= 0.0f ? params.yarn_ext_factor  : hparams.yarn_ext_factor;
    // ... the remaining YaRN, pooling and Flash Attention parameters
    cparams.n_ctx            = params.n_ctx           == 0    ? hparams.n_ctx_train           : params.n_ctx;
    cparams.rope_freq_base   = params.rope_freq_base  == 0.0f ? hparams.rope_freq_base_train  : params.rope_freq_base;
    cparams.rope_freq_scale  = params.rope_freq_scale == 0.0f ? hparams.rope_freq_scale_train : params.rope_freq_scale;
    // ...
    if (params.attention_type == LLAMA_ATTENTION_TYPE_UNSPECIFIED) {
        cparams.causal_attn = hparams.causal_attn;
    } else {
        cparams.causal_attn = params.attention_type == LLAMA_ATTENTION_TYPE_CAUSAL;
    }

    // with causal attention, the batch size is limited by the context size
    cparams.n_batch = cparams.causal_attn ? std::min(cparams.n_ctx, params.n_batch) : params.n_batch;
    cparams.n_ubatch = std::min(cparams.n_batch, params.n_ubatch == 0 ? params.n_batch : params.n_ubatch);
    // ... n_ctx is padded to a multiple of 256 and, unless the KV-cache is unified,
    //     divided among the sequences into the per-sequence limit n_ctx_seq
    LLAMA_LOG_INFO("%s: n_ctx                 = %u\n",   __func__, cparams.n_ctx);
    LLAMA_LOG_INFO("%s: n_ctx_seq             = %u\n",   __func__, cparams.n_ctx_seq);
    LLAMA_LOG_INFO("%s: n_batch               = %u\n",   __func__, cparams.n_batch);
    LLAMA_LOG_INFO("%s: n_ubatch              = %u\n",   __func__, cparams.n_ubatch);
    // ... the backends (GPU, CPU) and the output buffer are created, then the memory
    //     module (KV-cache), and last sched_reserve() builds the scheduler and the graphs
}

What is set in the context constructor:

• the reference to the model and the loading time are saved;

• the maximum number of sequences (n_seq_max), the number of threads, the YaRN/RoPE parameters, the context size (n_ctx), the batch size (n_batch, n_ubatch), the attention type (causal) are set.

  What for:

• the size of the KV-cache, the maximum prompt length and the size of the portions in which the batch is processed depend on these parameters.

The requested n_ctx is rounded up to a multiple of 256, and unless the KV-cache is shared between the sequences it is divided among them: the per-sequence limit n_ctx_seq is what actually bounds a single dialogue, and it is the number the log prints next to n_ctx.

Next the backends (CPU and graphics cards) are initialized, the memory object memory (the KV-cache and the service buffers) is created, the scheduler sched (which distributes the graph nodes across the devices), the graphs for Prefill and Decode are reserved — so that by the first decode the buffers for them are already allocated. After this llama_decode can be called with batches of tokens.

The n_batch and n_ubatch parameters:

• n_batch — the maximum number of tokens in one batch (during Prefill the batch may hold up to n_batch prompt tokens);
• n_ubatch — the maximum size of the ubatch that is processed in one process_ubatch call;
• if the prompt is longer than n_ubatch, it is split into ubatches of n_ubatch tokens, each of which is run through the model in turn;
• the logits are needed only for the last position in the batch, so only they are copied;
• reducing n_ubatch lowers the peak memory consumption at the cost of a larger number of passes.

The context is created once per session (or when the parameters change); one context can be used for many requests in a row without recreating it.

The memory and the scheduler are set up from the llama_context constructor (file src/llama-context.cpp), but not inline. The constructor first assembles its own list of backends — one per model device, plus the accelerator backends and the CPU one — then creates the memory object through model.create_memory, and last calls the helper sched_reserve. That helper is where ggml_backend_sched_new builds the scheduler over that backend list and where the graphs are reserved, so that on the first real decode the graph is built anew but placed into the already reserved memory. The KV-cache itself does not live in src/llama-memory.cpp — that file only holds two small helpers for the memory status. The buffers are allocated in the llama_kv_cache constructor in src/llama-kv-cache.cpp: for every layer it creates a tensor of keys and a tensor of values sized to the whole context, and then asks the backend for one buffer per device.

The graph_reserve function (in src/llama-context.cpp) reserves one graph per call: it makes a dummy batch of the requested shape, calls model.build_graph for it, and hands the result to ggml_backend_sched_reserve — the scheduler splits the graph across the backends and reserves the buffers. sched_reserve calls it three times: first for Prefill (a whole ubatch of tokens at once), then for Decode (one token per sequence), then for Prefill again, so that the buffers stay at their worst-case size and are never reallocated during inference. Later, on a real process_ubatch call, the previous graph is reused when its parameters allow it (can_reuse), and otherwise it is rebuilt and allocated with ggml_backend_sched_alloc_graph; either way the allocation for the large graphs has already happened at context creation, which is what removes the latency spike on the first decode.

Step 8: The Prompt Text Arrives from the User

Step 8 — the prompt text from the user arrives at the engine. After the context is created, the user sends a message (the prompt). The model works only with numbers — tokens. That is why the text is first turned into tokens (step 9), packed into a batch (step 10), and run through the model by the llama_decode call (steps 11–14): on the first request the batch holds all the prompt tokens (Prefill fills the KV-cache), and during generation one new token (Decode). From the logits the sampler chooses one next token (step 15), it is converted into text and printed (step 16); the token is added to the batch again, and the loop repeats until the "end of output" token (EOS) or a limit. Below each of these steps is analyzed through the code: what is called, what happens and what may be non-obvious.

Prefill and Decode — what the difference is and what for:

• On the first llama_decode call with the full prompt (Prefill) the model processes all the prompt tokens in one or several ubatches; K and V are computed for them and written into the KV-cache; the logits are requested only for the last position — the first token of the answer is chosen from them.

  What for: to "run" the whole prompt once and fill the cache of keys and values, so that afterwards generation goes token by token without recomputing the history.

• On the following calls the batch holds one new token (Decode); all the layers of the model are still traversed, but only for this single position: Q, K and V are computed for it alone, while K and V of the whole preceding history are read from the cache; the logits are again only for the last position.

  What for: efficient generation token by token without recomputing the whole history — the old K and V are already in the cache.

The whole loop fits into a couple of dozen lines. Below is the generation loop of the minimal example examples/simple-chat/simple-chat.cpp (the generate function), abridged, with this article's step numbers added to the comments.

// The generation loop, abridged from examples/simple-chat/simple-chat.cpp (the "generate" lambda).
// The step numbers are this article's; the rest is the example's own code.
// BOS is added only for the very first prompt of the conversation
const bool is_first = llama_memory_seq_pos_max(llama_get_memory(ctx), 0) == -1;
// Step 9: the prompt text -> an array of tokens. Called with tokens == NULL,
// llama_tokenize returns minus the token count - that is how the buffer is sized.
const int n_prompt_tokens = -llama_tokenize(vocab, prompt.c_str(), prompt.size(), NULL, 0, is_first, true);
std::vector<llama_token> prompt_tokens(n_prompt_tokens);
if (llama_tokenize(vocab, prompt.c_str(), prompt.size(), prompt_tokens.data(), prompt_tokens.size(), is_first, true) < 0) {
    GGML_ABORT("failed to tokenize the prompt\n");
}
// Step 10: the tokens are packed into a batch - one sequence, positions tracked by llama_decode
llama_batch batch = llama_batch_get_one(prompt_tokens.data(), prompt_tokens.size());
llama_token new_token_id;
while (true) {
    // ... here the example also checks that the batch still fits into the context
    // Steps 11-14: decode, ubatches, process_ubatch; the logits land in the context buffer
    int ret = llama_decode(ctx, batch);
    if (ret != 0) {
        GGML_ABORT("failed to decode, ret = %d\n", ret);
    }
    // Step 15: sampling; -1 means "the logits of the last position"
    new_token_id = llama_sampler_sample(smpl, ctx, -1);
    // is it an end of generation?
    if (llama_vocab_is_eog(vocab, new_token_id)) {
        break;
    }
    // Step 16: token -> text; a negative n means the buffer was too small
    char buf[256];
    int n = llama_token_to_piece(vocab, new_token_id, buf, sizeof(buf), 0, true);
    if (n < 0) {
        GGML_ABORT("failed to convert token to piece\n");
    }
    std::string piece(buf, n);
    printf("%s", piece.c_str());
    fflush(stdout);
    // the next batch holds the sampled token alone - the next pass is a Decode
    batch = llama_batch_get_one(&new_token_id, 1);
}

Step 9: Tokenization — from Text to a Sequence of Tokens

Step 9 — the prompt text is turned into a sequence of tokens (integers). The text has to be turned into a sequence of integers — tokens.

What this is needed for:

• the model (the transformer layers) accepts as input not strings but vectors of numbers of fixed length;
• one row in the model's embedding table corresponds to every token.

Tokenization is the first step:

• by the vocabulary the text is cut into pieces (tokens), and each piece is matched with a number (ID);
• further on, the embeddings are taken by these IDs and fed into the model.

What a token is:

• a token is an ID (an integer) that corresponds to a piece of text: a whole word, a part of a word or a special character;
• the model vocabulary defines the "text ↔ tokens" correspondence: from text one can get an array of tokens (tokenization), and from a token the text (token_to_piece).

In the API (the include/llama.h header) the llama_tokenize function is declared; the implementation delegates the call to the model vocabulary.

The implementation in src/llama-vocab.cpp:

// Step 9: the prompt text is turned into an array of tokens (IDs); the model vocabulary sets the "text <-> tokens" correspondence
int32_t llama_tokenize(
    const struct llama_vocab * vocab,
                  const char * text,
                     int32_t   text_len,
                 llama_token * tokens,
                     int32_t   n_tokens_max,
                        bool   add_special,
                        bool   parse_special) {
    return vocab->tokenize(text, text_len, tokens, n_tokens_max, add_special, parse_special);
}

Parameters:

• vocab — the model vocabulary;
• text and text_len — the prompt string;
• tokens — the array for writing the tokens;
• n_tokens_max — its size;
• add_special — whether to add the special tokens BOS and EOS, and only if the model itself is configured for them;
• parse_special — whether to process special tags.

The real logic (BPE, SentencePiece and so on) is in the vocab->tokenize method in src/llama-vocab.cpp. The result is the number of tokens written; if the buffer turns out to be too small, the function returns minus the required number of tokens — which is how an application first calls it with an empty buffer to find out the size, and only then allocates the array. Tokenization is performed on every new user message; the vocabulary does not change and has already been loaded when the model was loaded.

The implementation of tokenization is the llama_vocab::impl::tokenize method in src/llama-vocab.cpp. Inside, depending on the vocabulary type (SPM, BPE, WPM and so on), a tokenizer session is created and its tokenize is called; for BPE, for example, llm_tokenizer_bpe_session is used, which splits the text by regular expressions and assembles the tokens by the merges.

Tokenizer types in the vocabulary:

• LLAMA_VOCAB_TYPE_SPM — SentencePiece-like (the LLaMA models and others), splitting by the SPM rules and vocabulary;
• LLAMA_VOCAB_TYPE_BPE — Byte Pair Encoding (GPT-2 and others), the merges of byte/substring pairs are stored in GGUF, tokenization is greedy merging;
• LLAMA_VOCAB_TYPE_WPM — WordPiece, the BERT-style tokenizer;
• LLAMA_VOCAB_TYPE_UGM — Unigram, the T5-style tokenizer;
• LLAMA_VOCAB_TYPE_RWKV — greedy tokenization over a trie;
• LLAMA_VOCAB_TYPE_PLAMO2 — Aho-Corasick with dynamic programming;
• LLAMA_VOCAB_TYPE_NONE — the model was published without a vocabulary at all; there is nothing to tokenize with, and a tokenization attempt simply aborts. The tokenizer_st_partition function splits the input text into fragments: ordinary text and special tokens — a special token becomes a single token of the output, and the rest goes into the tokenizer according to the vocabulary type. The parse_special flag governs only the special and control tokens; tokens declared in the model as user-defined are always split out. Fragment of llama_vocab::impl::tokenize — the common part and the BPE branch; the other branches are built the same way:

// Step 9: the text is split into fragments (plain text and special tokens),
// and each fragment is tokenized according to the vocabulary type
std::vector<llama_token> llama_vocab::impl::tokenize(
        const std::string & raw_text,
        bool add_special,
        bool parse_special) const {
    GGML_ASSERT(tokenizer && "Tokenizer not initialized. Call llama_vocab::init_tokenizer() first.");
    std::vector<llama_token> output;
    std::forward_list<fragment_buffer_variant> fragment_buffer;
    if (!raw_text.empty()) {
        fragment_buffer.emplace_front(raw_text, 0, raw_text.length());
        tokenizer_st_partition(fragment_buffer, parse_special);
    }
    switch (get_type()) {
        // ... the SPM, WPM, UGM, RWKV and PLAMO2 branches have the same shape
        case LLAMA_VOCAB_TYPE_BPE:
            {
                const llm_tokenizer_bpe * tok_bpe = static_cast<const llm_tokenizer_bpe *>(tokenizer.get());
                std::unique_ptr<llm_tokenizer_bpe_session> session;
                // ... the "hybriddna" and "whitespace" tokenizer models get their own session subclass
                session = std::make_unique<llm_tokenizer_bpe_session>(vocab, *tok_bpe);
                if (add_special) {
                    session->append_bos(output);
                }
                for (const auto & fragment : fragment_buffer) {
                    if (fragment.type == FRAGMENT_BUFFER_VARIANT_TYPE_RAW_TEXT) {
                        std::string text = fragment.raw_text.substr(fragment.offset, fragment.length);
                        // ... whitespace escaping when the vocabulary asks for it
                        session->tokenize(text, output);
                    } else { // FRAGMENT_BUFFER_VARIANT_TYPE_TOKEN
                        session->append(fragment.token, output);
                    }
                }
                if (add_special) {
                    session->append_eos(output);
                    session->check_double_bos_eos(output);
                }
            } break;
        case LLAMA_VOCAB_TYPE_NONE:
            GGML_ABORT("fatal error");
    }
    return output;
}

Step 10: Forming the Batch for One Call

Step 10 — the tokens are packed into a batch for one call to the model. One call to the model is passed through the llama_batch structure (in include/llama.h).

What the batch is needed for:

• the llama_decode function accepts exactly one batch — a set of tokens (and service fields: positions, sequence identifiers, logit flags);
• this way the engine knows which tokens to process in one pass, at which positions they stand and for which positions the logits must be returned (usually only for the last one — in order to choose the next token);
• on the first request the batch usually holds all the prompt tokens; during token-by-token generation it holds one new token.

// Step 10: the batch is a "package" of tokens for one llama_decode call; token[], pos[], seq_id[], logits[] are filled in by the application or by balloc->init
typedef struct llama_batch {
    int32_t n_tokens;
    llama_token  *  token;
    float        *  embd;
    llama_pos    *  pos;
    int32_t      *  n_seq_id;
    llama_seq_id ** seq_id;
    int8_t       *  logits;
} llama_batch;

The structure of the batch:

• the arrays have the size n_tokens;
• either token IDs (token) or ready embeddings (embd) are passed — vectors of numbers into which the tokens have already been turned (usually tokens are passed and the embeddings are computed inside);
• pos — the position of every token; seq_id and n_seq_id — which sequence the token belongs to;
• logits[i] != 0 means that for position i the logits must be returned (usually only for the last one — in order to choose the next token). Inside llama_decode the batch is first processed by the llama_batch_allocr class (file src/llama-batch.cpp): the init method checks the batch and, if fields are missing, fills them in automatically (positions from memory, logits only for the last token).

The seq_id and n_seq_id fields are used when batching several sequences (for example, several requests in one batch): every token can belong to one or several sequences; the positions (pos) are counted separately for each sequence by memory->seq_pos_max(seq_id). In the typical case there is one sequence — all the tokens have one and the same seq_id (for example, 0), and the positions go in order 0, 1, 2, ... . There are two ways an application fills the batch for the next step. The minimal one is llama_batch_get_one, which wraps the single freshly sampled token and leaves both pos and seq_id NULL, so that llama_decode continues the position on its own and puts the token into sequence 0 — this is what examples/simple and examples/simple-chat do. The other is to allocate a batch once with llama_batch_init, then before every call clear it with common_batch_clear and add tokens with common_batch_add, passing the position and the sequence ids explicitly — which is what any code serving several sequences at once has to do.

The init method of llama_batch_allocr (file src/llama-batch.cpp), which llama_context::decode calls on the incoming batch before anything else happens to it:

// src/llama-batch.cpp, llama_batch_allocr::init(): validate the incoming batch, then fill in
// every array the caller left NULL — the positions and the output flags above all
bool llama_batch_allocr::init(
        const llama_batch & batch_inp,
        const llama_vocab & vocab,
        const llama_memory_i * memory,
        uint32_t n_embd,
        uint32_t n_seq_max,
        bool output_all) {
    clear();
    batch = batch_inp;
    this->vocab = &vocab;
    GGML_ASSERT(batch.n_tokens > 0);
    // ... omitted: n_seq_max must fit LLAMA_MAX_SEQ, every seq_id must be below n_seq_max,
    //     and every batch.token[i] must be a valid id (< vocab.n_tokens()) — otherwise return false ...
    // ... omitted: if n_seq_id or seq_id is NULL, every token is assigned to sequence 0 ...
    if (!batch.pos) {
        pos.resize(batch.n_tokens);

        // initialize the starting position for each sequence based on the positions in the memory
        llama_pos p0[LLAMA_MAX_SEQ];
        for (uint32_t s = 0; s < n_seq_max; ++s) {
            if (!memory) {
                // if no memory -> start from 0
                p0[s] = 0;
            } else {
                p0[s] = memory->seq_pos_max(s) + 1;
            }
        }

        for (int32_t i = 0; i < batch.n_tokens; i++) {
            const llama_seq_id seq_id = batch.seq_id[i][0];
            pos[i] = p0[seq_id];
            // ... omitted: p0 is advanced past this position for every sequence the token belongs to ...
        }
        batch.pos = pos.data();
    }
    if (!batch.logits) {
        if (output_all) {
            output.resize(batch.n_tokens, true);   // an output for every token
        } else {
            output.resize(batch.n_tokens, false);  // an output only for the last token
            output[output.size() - 1] = true;
        }
        batch.logits = output.data();
    }
    // ... omitted: if logits were supplied but output_all is set, the missing outputs are forced on;
    //     then the per-sequence position stats, the coupled sequences and the consistency checks ...
    return true;
}

In the code:

• the tokens are checked for validity;
• if n_seq_id and seq_id are missing, every token is assigned to exactly one sequence, number 0;
• if pos is missing, each sequence continues from where it stopped in memory (memory->seq_pos_max(s) + 1) and the positions are then handed out in order; when there is no memory at all, the count starts from 0;
• if logits is missing, the logits are requested only for the last token (or for all, if output_all).

From Tokens to Embeddings (Happens Inside Step 13)

Inside step 13 the tokens of the batch are turned into embeddings. In the batch one can pass either token IDs (token) or precomputed embeddings (embd), and in the typical case the application passes tokens. The conversion is not a separate pass that runs before the batch is processed: it is the very first node of the compute graph that decode builds for each ubatch.

Why the conversion is needed:

• the model (the transformer layers) works not with token numbers but with vectors of numbers of fixed length — embeddings;
• the lookup takes each token ID and reads the matching row of the model's embedding table, and that row becomes the input of the first layer;
• without it the graph would have nothing to compute on.

What the embedding table is:

• it is a matrix of model weights of size "vocabulary size × embedding size"; one row is the vector of numbers for one token;
• in the C++ code it is the model field tok_embd, and in a GGUF file it is the tensor token_embd.weight; the names embed_tokens and tok_embeddings come from the original PyTorch checkpoints — the conversion script maps them onto token_embd, so there is nothing to look for under those names inside llama.cpp;
• it is created while the model is loaded, in the tensor list of the particular architecture (load_arch_tensors, called from load_tensors);
• when the graph runs, for every token of the batch the row with index token[i] is read and lands at position i of the graph's input;
• after this the graph computes on the embeddings (the transformer layers, attention, feed-forward, the logits).

Where this is in the code: the embedding table is the model field tok_embd, and the lookup over it is built by llm_graph_context::build_inp_embd (file src/llama-graph.cpp), which every architecture calls as the first line of its graph. It creates two input tensors — an I32 vector for the token IDs and an F32 matrix for precomputed embeddings — and puts the token branch through ggml_get_rows over the table; both branches end up in the graph, and ggml_build_forward_select marks for computing only the one that matches what the ubatch carries — tokens or ready vectors. That is the point of it: the topology of the graph stays the same from ubatch to ubatch, so the graph can be reused. The IDs themselves are written into the input tensor by llm_graph_input_embd::set_input, which process_ubatch (file src/llama-context.cpp) calls after the graph has been built and allocated, just before running it. So the row is not copied by decode on the CPU: it is read on the backend, during the computation of the graph, together with everything else — and because the embedding table is normally stored quantized, ggml_get_rows unpacks each row into floats on the way out. During decode of a single token, exactly one row of the table is read.

The lookup as it is built in the graph (src/llama-graph.cpp, llm_graph_context::build_inp_embd):

// src/llama-graph.cpp, llm_graph_context::build_inp_embd() — abridged: debug callbacks and
// asserts removed. The token -> vector lookup is not a memcpy inside decode; it is a node
// of the compute graph, executed on the backend together with the rest of the model.
ggml_tensor * llm_graph_context::build_inp_embd(ggml_tensor * tok_embd) const {
    const int64_t n_embd_inp = hparams.n_embd_inp();

    auto inp = std::make_unique<llm_graph_input_embd>(n_embd_inp);

    inp->tokens = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, ubatch.n_tokens);
    ggml_set_input(inp->tokens);

    inp->embd = ggml_new_tensor_2d(ctx0, GGML_TYPE_F32, n_embd_inp, ubatch.n_tokens);
    ggml_set_input(inp->embd);

    // select one of the 2 inputs, based on the batch contents
    std::array<ggml_tensor *, 2> inps;

    // token embeddings path (ubatch.token != nullptr)
    {
        auto & cur = inps[0];

        cur = ggml_get_rows(ctx0, tok_embd, inp->tokens);

        // ... abridged: the optional LoRA delta on the embedding table,
        //     and the padding used when n_embd_inp != n_embd
    }

    // vector embeddings path (ubatch.embd != nullptr)
    {
        auto & cur = inps[1];

        cur = inp->embd;
    }

    ggml_tensor * cur = ggml_build_forward_select(gf, inps.data(), inps.size(), ubatch.token ? 0 : 1);

    // ... abridged: the n_embd_inp != n_embd view, and the f_embedding_scale multiplication

    res->add_input(std::move(inp));

    return cur;
}

What happens and what for:

• for every index i in the ubatch the token ID ubatch.token[i] is written into the input tensor of the graph inp_tokens.

  What for: the graph runs on the backend and needs the IDs there as its own data, not as a pointer into the batch the application passed in.

• ggml_get_rows reads from the tok_embd table the row with that index — a vector of length n_embd, because the table itself is n_embd wide. A few architectures need a wider graph input, n_embd_inp — deepstack models, and gemma4-assistant, which sets its own value; there the row is padded up to that width and then narrowed back to n_embd before it reaches the first layer.

  What for: this row is exactly the embedding of the token — the input for the first transformer layer.

• the rows for all the tokens of the ubatch make up the matrix that the first layer receives.

  What for: the layers process the whole ubatch at once; without this node the graph would have no input at all.

Steps 11 and 12: Running the Batch Through the Model (Entering decode, Ubatches)

Steps 11 and 12 — the application calls llama_decode, passing it the batch. Inside, the batch is validated and prepared (step 11), then split into ubatches (step 12), and each ubatch is run through the model (step 13). Turning token IDs into embedding vectors is not a separate host-side step: it is the first node of the computation graph, so it happens during step 13 together with everything else. Below are the entry point, the loop over the ubatches, and the splitting of the batch.

What the function llama_decode is for:

• it runs one batch of tokens through the model and fills the internal buffer of the context with logits — the "raw" scores for every possible next token;
• from these logits the application (via the sampler) chooses one next token and, if needed, calls decode again with one new token in the batch;
• this repeats until the end of the answer (EOS) or a limit.

The public API is llama_decode (in src/llama-context.cpp):

// Step 11: the decode entry point. The public C function only forwards the batch
// to the C++ context; all the work happens in llama_context::decode.
int32_t llama_decode(
        llama_context * ctx,
          llama_batch   batch) {
    const int ret = ctx->decode(batch);
    // ret == 1 means "no free KV-cache slot for this batch" — a warning, not a failure,
    // so it is not logged as an error
    if (ret != 0 && ret != 1) {
        LLAMA_LOG_ERROR("%s: failed to decode, ret = %d\n", __func__, ret);
    }

    return ret;
}

Here is what happens inside llama_context::decode (src/llama-context.cpp). First the batch is validated. If this context has no memory module at all — an embedding-only context, for example — decode simply hands the batch to encode and returns. Otherwise it calls balloc->init(...), reserves the scheduler and applies any pending memory (KV-cache) updates. Then it calls memory->init_batch once: that single call splits the whole batch into ubatches and returns a memory context holding all of them. A loop then walks that context, taking one ubatch at a time with get_ubatch(), running it through process_ubatch, copying the resulting logits into the context's internal buffer, and stepping to the next ubatch with next().

The loop over the ubatches inside decode (abridged, from src/llama-context.cpp):

// src/llama-context.cpp, llama_context::decode — abridged
    memory_update(false);                   // apply any pending shifts/copies
    llama_memory_context_ptr mctx;
    // split the WHOLE batch into ubatches of at most n_ubatch tokens, in one call
    // ... on FAILED_PREPARE: optimize the cache once, retry, otherwise return 1
    mctx = memory->init_batch(*balloc, cparams.n_ubatch, output_all);
    if (!mctx) {
        return -2;
    }
    // ... reserve the output buffer for n_outputs_all rows of logits
    do {
        const auto & ubatch = mctx->get_ubatch();
        // ... n_outputs = how many rows of logits this ubatch produces
        ggml_status status;
        // step 13: build (or reuse) the graph and run it on the CPU/GPU
        const auto * res = process_ubatch(ubatch, ctx_type_to_graph_type(cparams.ctx_type), mctx.get(), status);
        if (!res) {
            // ... roll this ubatch's positions back out of the memory module
            switch (status) {
                case GGML_STATUS_ABORTED:      return  2;
                case GGML_STATUS_ALLOC_FAILED: return -2;
                case GGML_STATUS_FAILED:       return -3;
                case GGML_STATUS_SUCCESS:      GGML_ABORT("should not happen");
            }
        }
        auto * t_logits = res->get_logits();
        // step 14: pull this ubatch's logits off the backend into the context buffer —
        // skipped when every output sequence has a backend sampler attached
        if (logits.data && t_logits && n_outputs > 0 && needs_raw_logits(ubatch, sampling.samplers)) {
            ggml_backend_t backend_res = ggml_backend_sched_get_tensor_backend(sched.get(), t_logits);
            float * logits_out = logits.data + n_outputs_prev*n_vocab;
            // ... asserts that these rows really fit into the output buffer
            ggml_backend_tensor_get_async(backend_res, t_logits, logits_out, 0, n_outputs*n_vocab*sizeof(float));
        }
        // ... embeddings and backend-sampler outputs are extracted the same way
        n_outputs_prev += n_outputs;
    } while (mctx->next());
    // after the loop the application reads the logits via llama_get_logits_ith

First the batch is checked:

• that it carries either tokens or precomputed embeddings;
• that this context actually has a memory module — if it does not, decode hands the batch straight to encode.

Then the batch is initialized via balloc->init — this is needed in order to fill in the token positions and to decide for which positions to compute the logits (usually only for the last one). The scheduler is reserved and the memory (KV-cache) is updated.

Next, memory->init_batch splits the batch into ubatches of no more than n_ubatch tokens each and returns a memory context holding all of them — this is how a large batch is broken into parts that are run through the model in turn. The loop then takes them one at a time and calls process_ubatch(...) for each: inside, the computation graph is built (the embeddings, the transformer layers, the output into logits) — or an already-built one is reused if nothing about the shape of the pass changed — the computation is performed on the CPU/GPU, and the tensors with the logits are returned: the raw, unnormalised scores over the vocabulary.

init_batch is a method of the memory module, so there is one implementation per memory type; the ordinary KV-cache is llama_kv_cache::init_batch in src/llama-kv-cache.cpp (the interface itself is declared in src/llama-memory.h):

// src/llama-kv-cache.cpp, llama_kv_cache::init_batch — abridged
// Step 12: the whole batch is cut into ubatches here, in one call, before any of them runs
llama_memory_context_ptr llama_kv_cache::init_batch(
            llama_batch_allocr & balloc,
            uint32_t n_ubatch,
            bool embd_all) {
    do {
        balloc.split_reset();

        std::vector<llama_ubatch> ubatches;
        while (true) {
            // each call takes the next (at most) n_ubatch not-yet-used tokens out of the batch
            auto ubatch = n_stream == 1 ? balloc.split_simple(n_ubatch) : balloc.split_equal(n_ubatch, true, 0);
            if (ubatch.n_tokens == 0) {
                break;  // the batch is exhausted
            }
            ubatches.push_back(std::move(ubatch));
        }

        if (balloc.get_n_used() < balloc.get_n_tokens()) {
            break;  // some tokens could not be placed into any ubatch
        }

        // find a KV-cache slot for every ubatch before committing to any of them
        auto sinfos = prepare(ubatches);
        if (sinfos.empty()) {
            break;
        }

        return std::make_unique<llama_kv_cache_context>(
                this, std::move(sinfos), std::move(ubatches));
    } while (false);

    // no room for this batch — decode turns this into return code 1
    return std::make_unique<llama_kv_cache_context>(LLAMA_MEMORY_STATUS_FAILED_PREPARE);
}

What init_batch does and what for:

• it cuts the whole batch into portions of no more than n_ubatch tokens each, in one pass.

  What for: one process_ubatch call processes a limited number of tokens, so the memory for the graph's intermediate tensors does not blow up on a long prompt.

• for each portion it copies the tokens, positions and output flags into a ubatch.

  What for: process_ubatch gets a ready ubatch and builds the graph for exactly that.

• before returning, it reserves a KV-cache slot for every ubatch and only then commits.

  What for: if the cache cannot hold the whole batch, decode learns this before a single ubatch has run and can report it to the caller instead of leaving half a batch in the cache.

• the returned memory context holds the list of ubatches; the loop in decode walks it with get_ubatch() and next().

  What for: a long prompt is processed in several passes through the model, without one huge graph.

The loop over the ubatches in llama_context::decode works like this: the memory context returned by memory->init_batch yields one ubatch at a time via get_ubatch(), process_ubatch(ubatch, gtype, mctx, status) is called for it, and next() advances to the following one until they run out. The graph type gtype is not chosen per ubatch — it follows from the context type and is the same for every pass of a given context. What does differ between a long prompt pass and a single-token pass is more mundane: a batched pass uses the batch thread count rather than the generation one, and llama.cpp reserves worst-case compute buffers for both shapes when the context is created. After every process_ubatch the logits for the positions marked as outputs are copied out of the backend into the context's internal buffer, from where llama_get_logits_ith reads them — but only if at least one of those positions belongs to a sequence without a backend sampler of its own: when every one of them has a sampler attached, the raw logits are not copied out at all. Building the graph anew for every ubatch would be wasteful, so llama.cpp reuses the previous one whenever the parameters that fully determine its topology have not changed.

The order of calls during one decode (summary):

• llama_decode(ctx, batch) → ctx->decode(batch);
• in decode: validating the batch (and, for a context with no memory module, handing it to encode and returning), balloc->init(...), reserving the scheduler, applying pending memory (KV-cache) updates;
• memory->init_batch(...) — one call that splits the whole batch into ubatches and reserves a cache slot for each; then the output buffer is reserved;
• the loop: get_ubatch() gives the next ubatch, process_ubatch(ubatch, ...) is called — inside it model.build_graph (or the reuse of the previous graph), res->set_inputs, graph_compute — the logits are copied into the context buffer, and next() moves on to the following ubatch;
• after the loop the application reads the logits via llama_get_logits_ith, and the sampler returns the next token.

The logits are copied into the output buffer of the context. From there llama_get_logits_ith reads them — from these numbers the sampler chooses the next token (for example, the most probable one or by temperature/top_p).

The return value of llama_decode: 0 — success; 1 — there was no free slot in the KV-cache for this batch (reduce the batch or enlarge the context); 2 — the computation was aborted by the callback; -1 — the batch itself is invalid; anything below -1 — a fatal error, most often a failed buffer allocation. A positive code is a warning rather than a failure, but note that 2 is positive as well, so checking only for a negative value would let an abort pass unnoticed — check for anything other than 0. After an abort or a fatal error the ubatches that already ran stay in the context's memory; how far the batch actually got can be found out with llama_memory_seq_pos_min and llama_memory_seq_pos_max. The minimal examples that ship with llama.cpp simply treat any non-zero code as fatal and stop. The application must check the return value and, on an error, stop the generation or print an error message.

Step 13a: Preparing to Run the Ubatch (Memory, Graph Parameters)

Step 13 is split in the article into three parts: preparing to run the ubatch, building the graph and allocating the buffers, writing the inputs and executing the graph. All the work on one ubatch is performed in the llama_context::process_ubatch method (file src/llama-context.cpp). It is called from llama_context::decode in a loop. The split into ubatches happens earlier and only once: memory->init_batch is called with the whole batch and returns a memory context object that already holds all the ubatches together with the cache state they need. The loop then walks them — mctx->get_ubatch() gives the current one, mctx->next() advances to the following one and returns false when they run out — and calls process_ubatch once per ubatch. Inside, the following happens step by step: preparing the memory, deciding whether to reuse an already built graph or to build a new one, building the graph and allocating buffers if needed, writing the input data into the graph, executing the graph on the CPU/GPU. Below is the first part of step 13: the entry point and the preparation.

What the computation graph is needed for:

• the model (a transformer) is a chain of operations: the embeddings, the attention layers (Q, K, V, softmax, weighted sum), the normalizations, the feed-forward and so on;
• instead of calling every operation manually, the engine builds a graph — a list of nodes (operations) and the links between them;
• the scheduler then traverses the graph in topological order and executes the operations on the CPU or the GPU;
• this way the nodes can be distributed across the devices automatically and the graph can be reused when the batch sizes are the same.

The signature and the input data of process_ubatch (a quote from src/llama-context.cpp):

// src/llama-context.cpp, llama_context::process_ubatch — one ubatch is run through the model;
// called from llama_context::decode in the loop over the ubatches
llm_graph_result * llama_context::process_ubatch(const llama_ubatch & ubatch, llm_graph_type gtype, llama_memory_context_i * mctx, ggml_status & ret) {
    // 1) commit the memory state prepared for this ubatch (the KV-cache cells) before the graph is built
    if (mctx && !mctx->apply()) {
        LLAMA_LOG_ERROR("%s: failed to apply memory context\n", __func__);
        ret = GGML_STATUS_FAILED;
        return nullptr;
    }

    // 2) the result of the PREVIOUS build: it owns the graph and its input tensors
    auto * res = gf_res_prev.get();
    auto * gf  = res->get_gf();

    // 3) everything the topology of the graph depends on; the graph may be reused only
    //    when these parameters match the ones it was built with
    const auto gparams = graph_params(res, ubatch, mctx, gtype);

    // ... omitted: the reuse decision, build_graph, set_inputs and graph_compute — see below
}

What happens at the beginning of process_ubatch and what for:

• mctx->apply() is called if a memory context (mctx) was passed.

  What for:

• the memory context updates the state of the KV-cache and the service buffers (for example, it shifts the pointers to the next free positions); before the graph is built, the graph will access these buffers — they must be in an up-to-date state.

• gf_res_prev is the result of the previous graph build: the graph itself, its input tensors and the parameters it was built with. On the very first call it is empty, so the graph is built from scratch; afterwards it is the candidate for reuse.

  What for:

• keeping the last graph is what makes reuse possible at all — if the new parameters match the ones stored in it, the same graph can be run again with new input values.

• gparams gathers everything the shape of the graph depends on: the model architecture and its hyper-parameters, the context parameters, a copy of the current ubatch (how many tokens it holds, how they are grouped by sequence, whether it carries token ids or precomputed embeddings), the graph type, the scheduler, the loaded LoRA adapters and control vector, the memory context, the per-sequence samplers and the number of positions for which outputs are wanted.

  What for:

• the rule is that two graphs with equal gparams have the same topology; that is exactly what makes the reuse check below possible, and the same values then set the sizes of the nodes and the input tensors while the graph is being built.

If mctx->apply() returns false, the function immediately returns nullptr and GGML_STATUS_FAILED is written into ret — the calling code (decode) will handle the error and stop the loop over the ubatches.

A quote in terms of meaning: what mctx->apply() does and what goes into gparams (according to the code in src/llama-context.cpp and src/llama-kv-cache.cpp):

// mctx->apply() — commits the memory state prepared for this ubatch. For the KV cache this means
// writing the position and the sequence id of every token into the cells reserved for it and
// recomputing how many cells the graph will have to read (src/llama-memory.h, src/llama-kv-cache.cpp)
if (mctx && !mctx->apply()) { ret = GGML_STATUS_FAILED; return nullptr; }

// graph_params() collects everything the topology of the graph depends on: the architecture, the model
// and context parameters, a copy of the ubatch, the graph type, the scheduler, the adapters, the memory
// context, the samplers and n_outputs — the reuse check compares exactly this set
const auto gparams = graph_params(res, ubatch, mctx, gtype);

Step 13b: Building the Computation Graph and Allocating the Buffers

Step 13 (continued) — building the graph and allocating buffers for it. After the memory has been prepared and gparams formed, the engine decides: is it possible to reuse the already built graph (the same ubatch shape and the same build parameters), or does the graph have to be built anew and buffers allocated for it. If the graph is reused, it immediately proceeds to writing the inputs and executing. If not, res->reset(), ggml_backend_sched_reset(sched.get()), model.build_graph(gparams) and ggml_backend_sched_alloc_graph(sched.get(), gf) are called. Below is a step-by-step account of what happens after what and what for; with several quotes from the code.

A quote: the decision about reuse and the building of the graph (a fragment of process_ubatch, src/llama-context.cpp):

    // the graph is rebuilt only when the parameters changed; repeated single-token decodes reuse it
    if (!graph_reuse_disable && res->can_reuse(gparams)) {
        // ... omitted: with pipeline parallelism the previous, still running compute is synchronized here,
        //     otherwise set_inputs would overwrite tensors that are still being read
        n_reused++;  // the reuse counter (shown by the profiler as "graphs reused")
    } else {
        res->reset();   // drop the old graph together with its input tensors

        ggml_backend_sched_reset(sched.get());  // drop the scheduler's split and its allocations
        ggml_backend_sched_set_eval_callback(sched.get(), cparams.cb_eval, cparams.cb_eval_user_data);

        gf = model.build_graph(gparams);  // embeddings → the transformer layers → the logits

        if (!gf) {
            LLAMA_LOG_ERROR("%s: failed to initialize graph\n", __func__);
            ret = GGML_STATUS_FAILED;
            return nullptr;
        }

        // the scheduler splits the graph across the devices and allocates every tensor
        if (!ggml_backend_sched_alloc_graph(sched.get(), gf)) {
            LLAMA_LOG_ERROR("%s: failed to allocate graph\n", __func__);
            ret = GGML_STATUS_ALLOC_FAILED;
            return nullptr;
        }
    }