LLama.cpp AI LLM Engine — How It Works
Foreword
Llama.cpp is a program (engine) for running large language models (LLM) on a computer or on mobile devices — on the processor (CPU) or the graphics card (GPU); I wrote the Offline AI Launcher application, which lets you use this same engine on Android.
Why this is needed:
- to get answers from neural network models (chat, text completion, code) without sending data to the cloud;
- the choice of C++ is needed for fast work with memory and hardware; the same code builds for Windows, Linux, macOS and Android.
Model "weights" are a huge set of numbers (millions or billions) on which all computations of the neural network are based: multiplication matrices, biases and so on. They are obtained during model training and saved to a file. During inference the engine only reads these numbers and applies them to the input data, without changing the weights themselves. Inference is the process of "querying" an already trained model: you provide text, and the model produces an answer step by step.
The engine works with the GGUF file format. GGUF is the format in which the saved model is stored:
- the "weights" (the numbers of the neural network);
- metadata (sizes, architecture type);
- the vocabulary (the "text ↔ numbers" correspondence for tokens).
In essence this is a single container file from which the engine reads everything it needs into memory. Quantization is supported (reducing the model size by storing numbers more coarsely) as well as hybrid computation (part on the CPU, part on the GPU). Models in GGUF format can be taken from Hugging Face.
The goal of this article is to walk step by step through how inference is arranged in llama.cpp: from loading the model to the appearance of the next token in the answer.
A token is a number corresponding to a piece of text (a word or part of a word); the model works only with numbers, and the vocabulary translates "text → tokens" and back.
The article is built as a scenario:
- first the preparation (initialization, model loading, context creation);
- then generation on request (tokenization, batch, decode, sampling).
Which models the engine works with:
- LLaMA (Meta);
- Qwen (Alibaba);
- Gemma (Google);
- Mistral;
- Phi and others.
The differences are in size, context length and details; in the code these are different hyperparameters and weight tensors. The overall scenario is one and the same.
Which projects use it:
- LM Studio;
- Ollama;
- GPT4All;
- Offline AI Launcher (running models on a smartphone);
- KoboldCpp;
- Text Generation WebUI.
Repository: github.com/ggml-org/llama.cpp. The code examples in the article may differ from your version.
How to read the article:
- the sections go in the order the scenario is executed;
- in each section, for every action there is a quote from the code and an explanation of what happens and why;
- the source files are indicated in the text;
- the article is aimed at an unprepared reader: all terms are explained in the glossary at the beginning of the article, all steps are supplied with code quotes and explanations.
Glossary of Terms
Batch — a set of tokens (or embeddings), positions and logit flags processed in one decode call. During Prefill the batch contains many prompt tokens; during Decode it contains one new token.
Backend — the computation "engine": CPU or graphics card (GPU). The scheduler distributes graph nodes across backends.
Decode — the generation stage for a single token: the batch holds one new token, the graph computes the logits for this position, and K and V are appended to the KV-cache.
Embedding — a vector of numbers into which a token is turned before being fed into the model layers; one row of the model's embedding matrix.
EOS (end of sequence) — a special "end of output" token; the application stops generation upon it.
GGUF — the model file format: header, metadata (key–value), tensor data. Quantization and mmap are supported.
Hyperparameters (hparams) — the numbers that define the model sizes: context length, number of layers, embedding size, number of attention heads and so on.
Inference — the process of getting an answer from the model: a prompt is supplied, and the model produces the next token step by step.
KV-cache — the cache of keys and values of the attention mechanism; it stores the already computed K and V for all previous positions so that they do not have to be recomputed at every step.
Logits — the model's "raw" scores for every token of the vocabulary before softmax; the sampler picks the next token from them.
Prefill — the prompt processing stage: the batch contains all (or many) prompt tokens, K and V are computed for them and written into the KV-cache.
Sampling — the choice of a single token from the logits (greedy, random with temperature/top_p and so on).
Tensor — a multidimensional array of numbers (model weights, embeddings, keys, values, logits and so on).
Token — an integer (ID) corresponding to a piece of text (a word or part of a word); the model works only with tokens.
Tokenizer — the vocabulary component that turns text into tokens (SPM, BPE and so on) and back.
Computation graph — a list of operations (matrices, additions, activations) and the links between them; the engine performs all the model computations according to it.
Ubatch — a part of the batch that is processed in one process_ubatch call. If the prompt is longer than n_ubatch, the batch is split into ubatches of n_ubatch tokens each; every ubatch is run through the model in turn.
What for:
- to limit peak memory consumption for a long prompt.
Scheduler (sched) — the GGML component that distributes the nodes of the computation graph across backends (CPU, GPU) and allocates buffers for the graph on those devices. When the graph is executed, the scheduler traverses the nodes in topological order and launches the operations on the selected devices.
Process Diagram: All Steps in Order
Below is the whole chain from starting the engine to the appearance of the next token in the answer. Each item is analyzed in detail further in the article, with code quotes and explanations.
Preparation (once at startup or when the model is changed):
step 1: backend initialization (llama_backend_init) →
step 2: loading the model from a file (llama_model_load_from_file → the GGUF loader) →
step 3: determining the model type (load_arch) →
step 4: loading the hyperparameters — sizes, context (load_hparams) →
step 5: loading the vocabulary and the tokenizer (load_vocab) →
step 6: loading the weights into memory or onto the GPU (load_tensors) →
step 7: creating the inference context — KV-cache, scheduler (llama_new_context).
Generation (for each message and each new token in the answer):
step 8: the prompt text arrives from the user →
step 9: tokenization — the text is turned into a sequence of tokens (llama_tokenize) →
step 10: forming the batch — the tokens are packed for one call (llama_batch_add, balloc->init) →
step 11: llama_decode — if needed, the tokens are turned into embeddings (encode) →
step 12: for a long prompt the batch is split into ubatches (memory->init_batch) →
step 13: each ubatch is run through the model: building the graph → execution on CPU/GPU (process_ubatch → build_graph → graph_compute) →
step 14: the logits for the last position are copied into the context buffer →
step 15: sampling — one next token is chosen from the logits →
step 16: the token is converted to text and printed (llama_token_to_piece); if it is not EOS, the new token is added to the batch and control goes to step 11; otherwise the loop finishes.
The General LLM Inference Process
The diagram above shows what happens after what. In terms of meaning, the process splits into two stages. The first is preparation: backend initialization, loading the model from GGUF (architecture, hyperparameters, vocabulary, weights) and creating the context. It is performed once at startup or when the model is changed. The second stage is generation: with every user message the text is turned into tokens, packed into a batch, run through the model (llama_decode), the next token is chosen from the logits, it is converted into text and printed; the loop repeats until the "end of output" token (EOS) or a limit. This loop repeats for every new message and for every new token in the answer. Below, each step of the diagram is analyzed separately: what exactly is called in the code, what happens and what may be non-obvious to an unprepared reader.
Repository Structure and Main Files
In the llama.cpp repository the main parts of the engine are spread across folders and files.
Why it is done this way:
- to separate responsibilities: file loading, the model, the vocabulary, the context and the batch are in different files;
- this makes it easier to search the code and debug it.
Below are listed the files that directly relate to model loading and inference; for each one it is stated what it contains and why it is needed.
Entry point and model loading:
- src/llama.cpp — this is where the functions llama_backend_init, llama_model_load_from_file, llama_model_load (static) live.
What for:
this is the "front door" into the engine: the application calls these functions to initialize the library and load the model; here too the loader is created and load_arch, load_hparams, load_vocab, load_tensors are called in turn.
- src/llama-model-loader.cpp — the llama_model_loader class: opening the GGUF file, building the tensor index (weights_map), the get_tensor and load_tensor_data methods.
What for:
the loader is needed so that, given a tensor name, one knows where its data lies in the file and how to read it or map it into memory (mmap); without it one cannot load the architecture, hyperparameters, vocabulary and weights step by step.
Model and vocabulary:
- src/llama-model.cpp — the llama_model class, the load_arch, load_hparams, load_vocab, load_tensors methods, creating tensors and assigning buffers.
What for:
the model object stores everything that was read from the file: the architecture type, the hyperparameters, the vocabulary and the weights themselves (tensors); the load_* methods fill in this data from the loader one after another.
- src/llama-vocab.cpp — the llama_vocab class, the implementation of llama_vocab::impl::load (loading the vocabulary from GGUF), tokenization and the reverse translation of tokens into text (tokenize, token_to_piece, detokenize).
What for:
the vocabulary is needed to turn text into numbers (tokens) on input and numbers back into text on output; without it the model can neither accept a prompt nor produce a readable answer.
Context and decode:
- src/llama-context.cpp — the llama_context class: creating the context (memory, scheduler, graph reservation), the decode method, process_ubatch, llama_get_logits_ith.
What for:
the context is the "working environment" of one generation session: it defines the context size, the KV-cache, the computation scheduler; the decode method runs the batch through the model and returns the logits.
- src/llama-batch.cpp — the batch structure, the llama_batch_allocr class, the init method (filling in positions and logit flags).
What for:
the batch is a "package" of tokens for one decode call; the allocator checks the batch and, if fields are missing, fills them in (positions from memory, logits only for the last token) so that the calling code does not need to set everything manually.
- src/llama-memory.cpp — allocation and update of the KV-cache, init_batch (splitting into ubatches).
What for:
the KV-cache stores the already computed keys and values for all previous positions so that they do not have to be recomputed at every step; init_batch splits a large batch into ubatches of limited size so as not to overflow memory.
Libraries and backends:
- ggml (a submodule or a separate folder) — the computation graph (GGML), tensor types, the scheduler (ggml_backend_sched), the CPU/GPU backends.
What for:
the graph describes which operations (matrix multiplications, activations and so on) to perform and in what order; the scheduler decides on which device (CPU or GPU) to compute each node of the graph.
- gguf — reading and writing the GGUF format (header, metadata, tensors).
What for:
the GGUF format defines how the header, the metadata and the tensor data lie in the file; the gguf library provides functions to read them without manual byte parsing.
- include/llama.h — the API header for applications: declarations of llama_model_load_from_file, llama_context, llama_decode, llama_tokenize, llama_get_logits_ith, the samplers and so on.
What for:
the application includes this header and calls the declared functions without going into the internal files of the engine.
Main Types and Structures (For Reference)
To find your way around the code it is useful to know the main types. Below is what each type stores and what it is needed for.
Main structures:
- llama_model — the object of the loaded model. It stores: the hyperparameters (hparams), the vocabulary (vocab), the weight tensors, the list of devices (devices), the distribution of layers across CPU/GPU.
What for:
the model is everything that was read from the file and is needed for computations; one model object can be used for several contexts (several generation sessions).
- llama_context — the inference context. It contains: a reference to the model, the context parameters (cparams: n_ctx, n_batch, n_ubatch, n_threads and so on), the batch allocator (balloc), the KV-cache memory (memory), the scheduler (sched), the reserved graphs (gf_res_prev and the like), the logits buffer.
What for:
the context is the "working environment" of one session: the context size, the cache of keys and values, the scheduler and the graphs for decode; on every request decode is called for exactly this context.
- llama_batch — an array of tokens (or embeddings), positions, sequence identifiers and logit flags.
What for:
one llama_decode call accepts one batch; it passes which tokens to process, at which positions and for which positions to return the logits (usually only for the last one).
- llama_model_loader — the GGUF loader. It contains: the metadata (meta), the tensor map (weights_map), the open files.
What for:
the loader lives only during model loading; through it the architecture, hyperparameters, vocabulary and tensors are read one after another; after loading it is not needed.
- ggml_context — the GGML graph context: the nodes and tensors of the graph are created in it.
What for:
the graph describes a sequence of operations (multiplications, activations and so on); all the graph nodes are created in one such context.
- ggml_backend_sched — the scheduler: the list of backends and the logic of distributing the graph nodes across devices.
What for:
the scheduler decides on which device (CPU or GPU) to execute each node of the graph, and allocates buffers for the graph on those devices.
What the main functions return:
- llama_model_load_from_file returns llama_model* or nullptr on error.
What for:
the application checks the pointer: if it is not nullptr, the model is loaded and the context can be created.
- llama_model_load returns 0 on success, -1 on error, -2 on cancellation (for example, via the progress callback).
What for:
based on the return value the calling code decides whether to delete the model and exit.
- llama_decode returns 0 on success, 1 if more input data is needed, and a negative value on error.
What for:
from the return value the application understands whether decode succeeded and whether the logits can be read.
- llama_tokenize returns the number of tokens written.
What for:
to know how many elements of the token array are filled.
- llama_get_logits_ith returns a pointer to a float array of size n_vocab.
What for:
from this array the sampler chooses the next token (one number per each token of the vocabulary).
Step 1: Backend Initialization
Step 1 in the process diagram is backend initialization. Before loading the model and running inference, the application calls llama_backend_init() once. The engine relies on internal structures: a timer for speed measurements and tables for working with numbers in the f16 format (half precision). They are created on the first call; without initialization, model loading or decode may behave incorrectly. Below is the quote in full and in fragments (file src/llama.cpp).
// Step 1: the single entry point of engine initialization; called once at application startupvoid llama_backend_init(void) { // Turn on the precise timer — later it is used to measure the loading and decode time ggml_time_init(); // Initialize the tables of the f16 format (half precision) once: // the weights and part of the computations are stored in f16 — memory saving and speedup on the GPU { struct ggml_init_params params = { 0, NULL, false }; // zero buffer — tables only struct ggml_context * ctx = ggml_init(params); ggml_free(ctx); // the context is freed, the tables stay initialized }}What happens in the code step by step:
// Fragment 1: the timer is needed to measure the model loading time and the decode time laterggml_time_init();
// Fragment 2: zero buffer — only the f16 tables are initialized, tensor data is not allocatedstruct ggml_init_params params = { 0, NULL, false };struct ggml_context * ctx = ggml_init(params);ggml_free(ctx); // the f16 tables stay initialized for the whole lifetime of the processFirst the timer for subsequent speed measurements is switched on. Then a temporary GGML context with a zero buffer is created — this is enough to initialize the internal f16 format tables once (conversion between float32 and float16). The context is freed right away, but the tables remain. After this the engine is ready to load the model and build the graph; the f16 format is used for the weights and part of the computations — it saves memory and speeds up work on the GPU.
Step 2: Loading the Model from a File (Entry Point)
Step 2 is loading the model from a file. It begins with the call to llama_model_load_from_file: the application passes the path to the .gguf and the parameters, and the engine returns a ready model object or nullptr on error.
What this function is for:
- it is the single entry point for loading a model from a file;
- the application passes the file path and the parameters, and the engine returns a ready model object or nullptr on error.
A quote from the code (file src/llama.cpp):
// Model loading entry point (step 2): the path to the .gguf and the parameters (mmap, n_gpu_layers etc.)struct llama_model * llama_model_load_from_file( const char * path_model, struct llama_model_params params) { // splits — paths to the model parts if it is split into several files; empty for a single file std::vector<std::string> splits = {}; return llama_model_load_from_file_impl(path_model, splits, params);}The function accepts the path to the model file (usually .gguf) and the loading parameters — the application specifies where to read the model from and how to load it (mmap, the number of layers on the GPU and so on). With a single file splits is empty; with a split model the paths to the parts are passed in splits. All further logic (backend check, progress callback, model creation, the call to llama_model_load) is concentrated in llama_model_load_from_file_impl.
The loading parameters (llama_model_params) that matter for understanding:
- use_mmap — use file mapping into memory instead of reading into a buffer.
What for:
it saves RAM and speeds up the start of loading; the file stays open.
- use_direct_io — direct input-output.
What for:
for some disks and OSes it gives a more predictable read speed.
- n_gpu_layers — how many layers to load onto the GPU (the rest onto the CPU).
What for:
so that part of the computations goes on the graphics card and part on the processor.
- progress_callback — the loading progress callback.
What for:
the application can show a progress bar or cancel the loading (return false).
- vocab_only — load only the vocabulary (without the weights).
What for:
when only the tokenizer is needed, without the heavy weights. The full list of parameters is in include/llama.h in the llama_model_params structure.
Step 2: Preparing to Read the File
Step 2 (continued) — inside the entry point the preparation for reading the file is performed. In llama_model_load_from_file_impl the main preparation before reading the file is carried out.
Why it is done this way:
- before loading one needs to make sure that there is a backend for computations, to set up progress display and to create the model object;
- only after that the internal function llama_model_load is called, which reads the file step by step.
A quote of the beginning of the function (file src/llama.cpp):
static struct llama_model * llama_model_load_from_file_impl( const std::string & path_model, std::vector<std::string> & splits, struct llama_model_params params) { ggml_time_init(); // timer for measuring the loading time // If we are loading not only the vocabulary — check that there is a backend (CPU or GPU) if (!params.vocab_only && ggml_backend_reg_count() == 0) { LLAMA_LOG_ERROR("%s: no backends are loaded...\n", __func__); return nullptr; } unsigned cur_percentage = 0; // If the progress callback was not passed — substitute our own: print dots up to 100% if (params.progress_callback == NULL) { params.progress_callback_user_data = &cur_percentage; params.progress_callback = [](float progress, void * ctx) { unsigned * cur_percentage_p = (unsigned *) ctx; unsigned percentage = (unsigned) (100 * progress); while (percentage > *cur_percentage_p) { *cur_percentage_p = percentage; LLAMA_LOG_CONT("."); if (percentage >= 100) { LLAMA_LOG_CONT("\n"); } } return true; // do not cancel the loading }; } llama_model * model = new llama_model(params); // create the model objectWhat happens in this fragment and what for:
- ggml_time_init() — the timer is switched on.
What for:
so that later it is possible to measure how long the loading took.
- The check ggml_backend_reg_count() == 0 — whether there is at least one backend (CPU or GPU).
What for:
without a backend it will be impossible to perform the model computations; when loading only the vocabulary (vocab_only) a backend is not required.
- Setting up the progress callback — if the application did not pass its own, a default callback that prints dots is substituted.
What for:
the user sees that the loading is in progress; if desired one can pass one's own callback and show a progress bar or cancel the loading (return false).
- new llama_model(params) — the model object is created with the passed parameters.
What for:
the architecture, hyperparameters, vocabulary and tensors from the file will later be written into this object.
Further in the same function the list of devices is formed and llama_model_load is called. Quote:
// List of devices: if not set in params — take all the available ones (CPU + GPU)std::vector<ggml_backend_dev_t> devs = params.devices.empty() ? ggml_backend_dev_get_all() : params.devices;model->devices = devs;// Internal loading: reads the file, fills in the model (arch, hparams, vocab, tensors)int ret = llama_model_load(path_model, splits, *model, params);if (ret != 0) { delete model; return nullptr;}return model;- devs — the list of devices (CPU and graphics cards).
What for:
it determines onto which devices the model layers will be loaded (see load_tensors).
- llama_model_load(path_model, splits, *model, params) — reads the file and fills in the model step by step (loader, load_arch, load_hparams, load_vocab, load_tensors).
What for:
all the logic of reading GGUF and filling in the model is concentrated in one function.
- On a non-zero return the model is deleted and nullptr is returned.
What for:
from the nullptr the application understands that the loading failed and does not use an incomplete model.
The progress callback is invoked inside load_tensors while reading every tensor: it is passed a number from 0.0 to 1.0 (the share of loaded data). If the callback returns false, the loading is interrupted and llama_model_load returns -2 (cancellation).
What for:
- the application can cancel a long loading or show a progress bar.
Steps 2–6: Step-by-Step Loading of the Model from a File
Steps 2–6 are performed inside a single function llama_model_load: the loader is created (step 2), then load_arch (step 3), load_hparams (step 4), load_vocab (step 5), load_tensors (step 6) are called in turn.
What the function llama_model_load is for: it performs the step-by-step loading of the model from the file: it creates the GGUF loader (opens the file and builds the tensor map), then loads the architecture, hyperparameters, vocabulary and tensors one after another. The order matters: the architecture defines the set of keys in GGUF; the hyperparameters are read by these keys; the vocabulary is loaded taking the architecture into account; the tensors are created according to the known sizes and filled in from the file.
A quote of the whole function (file src/llama.cpp):
// Returns 0 on success, -1 on error, -2 on cancellation (the progress callback returned false)static int llama_model_load(const std::string & fname, std::vector<std::string> & splits, llama_model & model, llama_model_params & params) { model.t_load_us = 0; time_meas tm(model.t_load_us); // measuring the loading time model.t_start_us = tm.t_start_us; try { // Step 2 (continued): the loader opens GGUF, reads the header and the metadata, builds the tensor map (weights_map) llama_model_loader ml(fname, splits, params.use_mmap, params.use_direct_io, params.check_tensors, params.no_alloc, params.kv_overrides, params.tensor_buft_overrides); ml.print_info(); model.hparams.vocab_only = params.vocab_only; model.hparams.no_alloc = params.no_alloc; // Step 3: the model type (LLaMA, Gemma etc.) — the field names in GGUF depend on it try { model.load_arch(ml); } catch(const std::exception & e) { throw std::runtime_error("error loading model architecture: " + std::string(e.what())); } // Step 4: model sizes, context length, number of layers — needed to allocate memory and build the graph try { model.load_hparams(ml); } catch(const std::exception & e) { throw std::runtime_error("error loading model hyperparameters: " + std::string(e.what())); } if (model.arch == LLM_ARCH_CLIP) { throw std::runtime_error("CLIP cannot be used as main model, use it with --mmproj instead"); } // Step 5: the vocabulary and the tokenizer — text ↔ tokens on input and output try { model.load_vocab(ml); } catch(const std::exception & e) { throw std::runtime_error("error loading model vocabulary: " + std::string(e.what())); } model.load_stats(ml); model.print_info(); if (params.vocab_only) { LLAMA_LOG_INFO("%s: vocab only - skipping tensors\n", __func__); return 0; } // Step 6: the model weights from the file into memory (or mmap) on CPU/GPU if (!model.load_tensors(ml)) { return -2; } } catch (const std::exception & err) { LLAMA_LOG_ERROR("%s: error loading model: %s\n", __func__, err.what()); return -1; } return 0;}Errors and cancellation of loading:
- on an error in any of the steps (load_arch, load_hparams, load_vocab, load_tensors) an exception is thrown; in the catch block the message is logged and -1 is returned;
- on cancellation via the progress callback (the callback returns false inside load_tensors) load_tensors returns false, no exception is thrown, but llama_model_load returns -2 (cancellation);
- the calling code (llama_model_load_from_file_impl) on a non-zero return deletes the model and returns nullptr;
- this way the application can cancel a long loading via the callback and release resources correctly.
Step by step (what happens and what for):
- the loading time is reset and the timer is started.
What for:
- so that later it is possible to measure how long the loading took.
- the loader llama_model_loader ml(...) is created — it opens the GGUF file and reads the metadata, builds weights_map.
What for:
- without the loader one cannot read the architecture, hyperparameters, vocabulary and tensors from the file.
- ml.print_info() is called — it prints to the log the architecture, the number of tensors, the file size.
What for:
- so that the user sees the progress and the information about the file.
- model.load_arch(ml) is called — we determine the model type (LLaMA, Gemma, Qwen and so on).
What for:
- the field names in GGUF and which tensors to create depend on the architecture.
- model.load_hparams(ml) is called — we read the dimensions and parameters (context length, number of layers and the like).
What for:
- to know the "shape" of the model and allocate memory for it.
- for CLIP an error will be thrown — it is used separately as a projector, not as the main model.
- model.load_vocab(ml) is called — we load the token vocabulary.
What for:
- so that later text can be turned into numbers (tokens) and back during generation.
- load_stats(ml) and print_info are called — they print the statistics. If we are loading only the vocabulary (vocab_only == true), this is the exit point. Otherwise model.load_tensors(ml) is called — the model weights are read and laid out in memory. On success 0 is returned, on cancellation (the progress callback returned false) -2, on error -1.
The order of calls when loading the model (summary):
- llama_model_load_from_file → llama_model_load_from_file_impl;
- in impl: backend check, progress callback, new llama_model(params), forming the list of devices, llama_model_load(path_model, splits, *model, params);
- inside llama_model_load: creating llama_model_loader, load_arch, load_hparams, load_vocab, load_stats, print_info, and if needed load_tensors. All these steps are executed sequentially; on an error in any of them the loading is interrupted.
The order of the calls load_arch → load_hparams → load_vocab matters: the architecture defines the set of GGUF keys; the hyperparameters are read by these keys; the vocabulary is loaded taking the architecture into account (for example, the key names for the tokenizer). That is why vocab.load(ml, kv) is called exactly after load_hparams(ml) — by this moment both the architecture and the hyperparameters are already known, and the loader can correctly read the tokenizer type, the token lists and the merges.
Step 2: The GGUF Loader — Opening the File and the Tensor Map
Step 2 (inside llama_model_load) — the GGUF loader is created: the file is opened, the header and the metadata are read, the tensor map is built.
What the GGUF loader is for: to open the model file, read the header and the metadata (without the weights themselves) and build a "map" — so that, given a tensor name, one knows where its data lies in the file and what size it is. Without this map one cannot later load the weights tensor by tensor. Tensors here are multidimensional arrays of numbers (matrices and vectors) that store the weights of the neural network; every model layer is several tensors, and the loader must know the name, size and offset in the file for each one.
Creating the loader is a call to the llama_model_loader constructor. In the llama_model_load code it looks like this:
// The loader constructor (step 2): opens GGUF, reads the header and the metadata, builds the tensor mapllama_model_loader::llama_model_loader( const std::string & fname, std::vector<std::string> & splits, bool use_mmap, bool use_direct_io, bool check_tensors, bool no_alloc, const llama_model_kv_override * param_overrides_p, const llama_model_tensor_buft_override * param_tensor_buft_overrides_p) { // no_alloc = true: we read only the header and the metadata, the tensor data is not loaded yet struct ggml_context * ctx = NULL; struct gguf_init_params params = { /*.no_alloc = */ true, /*.ctx = */ &ctx, }; // Header + metadata (key–value) into meta; in ctx — the list of tensors with names, types, offsets in the file meta.reset(gguf_init_from_file(fname.c_str(), params)); if (!meta) { throw std::runtime_error(format("%s: failed to load model from %s", __func__, fname.c_str())); } // The architecture name (llama, qwen2 etc.) — the field names in load_hparams and load_vocab depend on it get_key(llm_kv(LLM_KV_GENERAL_ARCHITECTURE), arch_name, false); llm_kv = LLM_KV(llm_arch_from_string(arch_name)); // The file is open for reading — later we will read or map the tensor data by the offsets from weights_map files.emplace_back(new llama_file(fname.c_str(), "rb", use_direct_io)); contexts.emplace_back(ctx); // Map: tensor name → (file, offset, metadata, tensor); in load_tensor_data we find by name where to read from for (ggml_tensor * cur = ggml_get_first_tensor(ctx); cur; cur = ggml_get_next_tensor(ctx, cur)) { std::string tensor_name = std::string(cur->name); if (weights_map.find(tensor_name) != weights_map.end()) { throw std::runtime_error(format("invalid model: tensor '%s' is duplicated", ggml_get_name(cur))); } n_elements += ggml_nelements(cur); n_bytes += ggml_nbytes(cur); weights_map.emplace(tensor_name, llama_tensor_weight(files.back().get(), 0, meta.get(), cur)); }}In the constructor gguf_init_from_file reads the GGUF header and the metadata (without the weights themselves) — this is how the list of tensors and the key–value pairs (architecture, hyperparameters) are obtained without reading the heavy data into memory. The architecture name (general.architecture) is taken from the metadata — the names of the remaining fields in GGUF depend on it (different models have different keys). The file is opened for reading; later the tensor data will be read or mapped by the offsets from weights_map. From the list of tensors in GGUF, for each one the name, size and offset in the file are written into weights_map — when load_tensor_data is called, the loader will find the tensor in the map by name and read the data at the offset.
The llama_tensor_weight structure (an element of weights_map) stores a reference to the file, the offset in the file, a reference to the GGUF metadata and a pointer to the tensor in the GGUF context — by these the loader later reads the bytes into the tensor buffer. The ml.get_key(...) method reads from the GGUF metadata the value for a key (the key name depends on the architecture — it is returned by kv(...)). This is how the loader obtains, for example, the architecture name, the tokenizer type, the hyperparameters.
After the loader is created, ml.print_info() is called in llama_model_load. A quote from the code:
// In llama_model_load after creating llama_model_loader:ml.print_info(); // prints to the log the architecture, the number of tensors, the file size — for debugging and for the userThe gguf_init_from_file function (the GGUF library) opens the file and reads the header:
- the format version;
- the number of metadata keys;
- the number of tensors.
The metadata is read into the gguf_context (key–value pairs); the tensor data itself is not loaded at this step — only the names, types and offsets in the file. The print_info method (src/llama-model-loader.cpp) prints information about the file to the log:
- the architecture;
- the number of tensors;
- the size in bytes.
The GGUF Format (For Reference)
GGUF (GPT-Generated Unified Format) is a binary format for storing machine learning models.
What it is needed for:
- to store in one file both the model weights and the metadata (sizes, architecture type) and the vocabulary;
- from the header and the metadata the loader builds a "map" and then, on demand, reads the needed pieces of the file or maps them into memory (mmap).
Structure of the GGUF header:
- the magic number (format identification — by it one understands that this is GGUF);
- the format version (for compatibility when the format changes);
- the number of metadata keys (n_kv);
- the number of tensors (n_tensors).
The metadata is stored as an array of "key — value" pairs. For each pair the following are recorded:
- the key type (string, number, array and so on);
- the key name;
- the value (architecture name, dimensions, RoPE parameters, tokenizer type, token lists and so on).
The tensors in the file come after the metadata. For each tensor the following are recorded:
- the name;
- the element type (F32, F16, Q8_0, Q4_K_M and so on — from full precision to quantized formats);
- the dimensions (for example, [n_layer, n_embd, n_embd]);
- the offset in the file at which the data begins.
Why the loader needs this: from this information it builds weights_map (the map "tensor name → where the data lies in the file") and, when load_tensor_data is called, reads or maps the corresponding region of the file.
The format version and the element types:
- the format version is set in the GGUF header.
What for:
- when the format changes, the version lets the loader understand which version it is working with; on incompatibility the loader can raise an error or ignore unknown fields.
- from the tensor element type (F32, F16, Q8_0, Q4_K_M and so on) the loader knows how many bytes one element takes and how to interpret the data when copying into a buffer or when using mmap.
What for:
- without this it is impossible to read or map the tensor data into memory correctly; different types have different sizes and different byte interpretation.
The tensor data types in GGUF define how to interpret the bytes: F32, F16, Q8_0, Q4_K_M and so on — from full precision to quantized formats. Quantization reduces the model size and speeds up computations at the cost of an approximate representation of the weights. When loading, the engine creates the tensors in the required format and copies or maps the data from the file into buffers on the CPU or the GPU.
Reading the GGUF metadata is done through the functions of the gguf library:
- gguf_get_n_kv — the number of keys;
- gguf_get_key — the key name by index;
- gguf_get_kv_type — the value type (string, number, array and so on);
- gguf_get_val_* — the value by index or by name.
The model loader wraps this in the get_key(key, value) method taking the architecture into account: the key is converted into a field name in GGUF (for example, llama.embedding_length for LLaMA).
Step 3: Determining the Model Type (Architecture)
Step 3 is determining the model type (LLaMA, Gemma, Qwen and so on) by the architecture name from GGUF. The llama_model::load_arch method determines the model type by the name from GGUF. What this is needed for: the field names in the GGUF metadata and which tensors to create when loading the weights depend on the architecture type (LLaMA, Gemma, Qwen and so on); without this one cannot correctly read the hyperparameters and the vocabulary. Quote (file src/llama-model.cpp):
// Step 3: by the architecture name from GGUF we choose the model type — the keys in load_hparams and load_vocab depend on itvoid llama_model::load_arch(llama_model_loader & ml) { arch = ml.get_arch(); // reads general.architecture (a string) and turns it into an enum: LLM_ARCH_LLAMA, LLM_ARCH_GEMMA etc. if (arch == LLM_ARCH_UNKNOWN) { throw std::runtime_error("unknown model architecture: '" + ml.get_arch_name() + "'"); }}What happens and what for:
- ml.get_arch() reads the general.architecture key (a string) from the GGUF metadata and converts it into the llm_arch enum.
What for:
further on, this enum is used to choose the field names for the hyperparameters and the vocabulary (different architectures have different keys in GGUF).
- If the type is unknown (LLM_ARCH_UNKNOWN), an error is thrown.
What for:
the engine cannot work with an unknown architecture; the application will get an error message and will be able to report it to the user.
The implementation of get_arch() in the loader (src/llama-model-loader.cpp):
// Helper method of the loader: reads general.architecture from GGUF and turns the string into an enumllm_arch llama_model_loader::get_arch() const { std::string arch_name; get_key(llm_kv(LLM_KV_GENERAL_ARCHITECTURE), arch_name, false); // the key from meta (the GGUF metadata) return llm_arch_from_string(arch_name); // "llama" → LLM_ARCH_LLAMA, "qwen2" → LLM_ARCH_QWEN etc.}What these calls do:
- get_key(llm_kv(LLM_KV_GENERAL_ARCHITECTURE), arch_name, false) — the value for the "general architecture" key (for example, llama, qwen2) is read from the GGUF metadata and written into arch_name.
What for:
without the architecture name one cannot choose the set of keys for the hyperparameters and the vocabulary.
- llm_arch_from_string(arch_name) — the string is turned into the llm_arch enum (LLM_ARCH_LLAMA, LLM_ARCH_QWEN and so on).
What for:
by this enum the field names in GGUF for load_hparams and load_vocab are chosen further on.
- Examples of enum values: LLM_ARCH_LLAMA, LLM_ARCH_GEMMA, LLM_ARCH_QWEN, LLM_ARCH_PHI, LLM_ARCH_MISTRAL, LLM_ARCH_CLIP and so on. For every architecture its own set of GGUF keys (LLM_KV) is defined, by which the loader reads the field names (hyperparameters, vocabulary, tensor names).
What for:
without the correct set of keys one cannot read the hyperparameters and the vocabulary from GGUF.
- The tensor names in GGUF depend on the architecture: for LLaMA they are blk.N.attn_q.weight, blk.N.attn_k.weight and so on; other models have their own prefixes and suffixes. When support for a new architecture is added, a new enum and a set of LLM_KV keys are added to the code. The hyperparameters are used when creating the tensors in load_tensors (matrix sizes, the number of layers) and when creating the context (n_ctx, n_batch, the RoPE parameters and so on).
Step 4: Loading the Hyperparameters — Sizes and Context
Step 4 is loading the hyperparameters (model sizes, context length, number of layers and so on) from the GGUF metadata. The llama_model::load_hparams method fills in the hparams structure (the hyperparameters) from the GGUF metadata. What this is needed for: the hyperparameters define the "shape" of the model — the context length, the number of layers, the embedding size, the number of attention heads and the like; without them one cannot allocate memory for the tensors and build the computation graph. A quote with the main keys (file src/llama-model.cpp):
// Step 4: read the hyperparameters from GGUF — model sizes, context length, number of layers; memory allocation and the graph depend on themvoid llama_model::load_hparams(llama_model_loader & ml) { const gguf_context * ctx = ml.meta.get(); // Save all the key–value pairs from GGUF (except arrays) for reference for (int i = 0; i < gguf_get_n_kv(ctx); i++) { gguf_type type = gguf_get_kv_type(ctx, i); if (type == GGUF_TYPE_ARRAY) { continue; } const char * name = gguf_get_key(ctx, i); const std::string value = gguf_kv_to_str(ctx, i); gguf_kv.emplace(name, value); } ml.get_key(LLM_KV_GENERAL_NAME, name, false); if (hparams.vocab_only || ml.get_arch() == LLM_ARCH_CLIP) { return; } // Main sizes (the key names depend on the architecture — load_arch has already been called) ml.get_key(LLM_KV_CONTEXT_LENGTH, hparams.n_ctx_train); // the context length during training ml.get_key(LLM_KV_EMBEDDING_LENGTH, hparams.n_embd); // the embedding (hidden layer) size ml.get_key(LLM_KV_EMBEDDING_LENGTH_OUT, hparams.n_embd_out_impl, false); ml.get_key(LLM_KV_BLOCK_COUNT, hparams.n_layer); // the number of transformer layers ml.get_key(LLM_KV_EXPERT_COUNT, hparams.n_expert, false); // for MoE models ml.get_key_or_arr(LLM_KV_FEED_FORWARD_LENGTH, hparams.n_ff_arr, hparams.n_layer, false); // the FF size per layer ml.get_key_or_arr(LLM_KV_ATTENTION_HEAD_COUNT, hparams.n_head_arr, hparams.n_layer, false); // the number of attention heads hparams.n_head_kv_arr = hparams.n_head_arr; ml.get_key_or_arr(LLM_KV_ATTENTION_HEAD_COUNT_KV, hparams.n_head_kv_arr, hparams.n_layer, false); // the number of KV heads (GQA) ml.get_key(LLM_KV_ROPE_FREQ_BASE, hparams.rope_freq_base_train, false); // the base RoPE frequency // rope_scaling_type, rope_freq_scale etc.}What happens and what for:
- In the loop all key–value pairs from GGUF (except arrays) are read and saved into gguf_kv.
What for:
- so that later, if needed, any field can be accessed by name.
- Then, using the keys that depend on the architecture, the hparams fields are filled in:
- n_ctx_train — the context length during training.
What for:
- the size of the KV-cache and the maximum input length depend on it.
- n_embd — the embedding (hidden layer) size.
What for:
- the sizes of the weight matrices depend on it.
- n_layer — the number of transformer layers.
What for:
- how many tensors to create in load_tensors depends on it.
- n_ff_arr, n_head_arr, n_head_kv_arr — the feed-forward sizes and the number of attention heads per layer.
What for:
- the sizes of the Q, K, V and feed-forward matrices depend on them.
- rope_freq_base_train and others — the RoPE (positional encoding) parameters.
What for:
- they define how the positional encodings are applied in the graph. As a result hparams fully describes the sizes of the model — this is enough to allocate memory for the tensors and build the computation graph.
Some of the GGUF keys for the hyperparameters (depending on the architecture):
- llama.context_length — the context length during training;
- llama.embedding_length — the size of the hidden layer (the embedding);
- llama.block_count — the number of transformer layers;
- llama.attention.head_count — the number of attention heads;
- llama.attention.head_count_kv — the number of KV heads (for GQA);
- llama.feed_forward_length — the size of the intermediate feed-forward layer;
- llama.rope.freq_base — the base RoPE frequency.
For MoE models the keys for the number of experts and so on are added. The get_key_or_arr method reads either a single value or an array per layer (when the sizes differ across layers). After the hyperparameters are loaded, the model knows the "shape" of all the tensors — this is enough to allocate buffers and build the graph in load_tensors and when creating the context.
Step 5: Loading the Vocabulary and the Tokenizer
Step 5 is loading the vocabulary and the tokenizer: by the vocabulary, text is turned into tokens on input and back into text on output. The token vocabulary is a correspondence table between pieces of text and integers (tokens); by it text is split into tokens on input and assembled back into text on output. Loading the vocabulary is performed after loading the architecture and the hyperparameters: in llama_model_load first load_arch(ml) and load_hparams(ml) are called, and then load_vocab(ml). Inside load_vocab, vocab.load(ml, kv) is called. It is loaded in llama_model::load_vocab (src/llama-model.cpp):
void llama_model::load_vocab(llama_model_loader & ml) { const auto kv = LLM_KV(arch);
vocab.load(ml, kv);}For the current architecture the set of vocabulary keys is taken, after which vocab.load(ml, kv) is called. The implementation is the llama_vocab::impl::load method in src/llama-vocab.cpp. The following are read from GGUF:
- the tokenizer type (SPM, BPE, WPM and so on);
- the token lists;
- the token types;
- the BPE merges (if present);
- the special tokens — everything needed to turn text into a sequence of numbers (tokens) and back.
After this the model is able to tokenize the prompt and translate the generated tokens back into text.
The vocab.load(ml, kv) call is performed inside llama_model::load_vocab after the set of keys kv has been obtained for the current architecture (via LLM_KV(arch)). The loader ml has already opened GGUF and read the metadata; the architecture and the hyperparameters have been loaded by this moment in load_arch and load_hparams. The beginning of the llama_vocab::impl::load implementation (file src/llama-vocab.cpp):
void llama_vocab::impl::load(llama_model_loader & ml, const LLM_KV & kv) { struct gguf_context * ctx = ml.meta.get(); // Tokenizer type: llama → SPM, gpt2 → BPE etc. ml.get_key(LLM_KV_TOKENIZER_MODEL, tokenizer_model); ml.get_key(LLM_KV_TOKENIZER_PRE, tokenizer_pre, false); ml.get_key(LLM_KV_TOKENIZER_TOKEN_TYPE_COUNT, n_token_types, false); if (tokenizer_model == "no_vocab" || tokenizer_model == "none") { type = LLAMA_VOCAB_TYPE_NONE; special_bos_id = LLAMA_TOKEN_NULL; special_eos_id = LLAMA_TOKEN_NULL; special_unk_id = LLAMA_TOKEN_NULL; return; } if (tokenizer_model == "llama") { type = LLAMA_VOCAB_TYPE_SPM; // SentencePiece-like special_bos_id = 1; special_eos_id = 2; special_unk_id = 0; } else if (tokenizer_model == "gpt2") { type = LLAMA_VOCAB_TYPE_BPE; // Read the BPE merges (pairs of substrings for greedy merging) const int merges_keyidx = gguf_find_key(ctx, kv(LLM_KV_TOKENIZER_MERGES).c_str()); const int n_merges = gguf_get_arr_n(ctx, merges_keyidx); for (int i = 0; i < n_merges; i++) { const std::string word = gguf_get_arr_str(ctx, merges_keyidx, i); std::string first, second; const size_t pos = word.find(' ', 1); if (pos != std::string::npos) { first = word.substr(0, pos); second = word.substr(pos + 1); } bpe_ranks.emplace(std::make_pair(first, second), i); } } // The token array: for every ID — a string (a piece of text) const int token_idx = gguf_find_key(ctx, kv(LLM_KV_TOKENIZER_LIST).c_str()); if (token_idx == -1) { throw std::runtime_error("cannot find tokenizer vocab in model file\n"); } uint32_t n_tokens = gguf_get_arr_n(ctx, token_idx); id_to_token.resize(n_tokens); for (uint32_t i = 0; i < n_tokens; i++) { std::string word = gguf_get_arr_str(ctx, token_idx, i); token_to_id[word] = i; // text → ID max_token_len = std::max(max_token_len, (int) word.size()); auto & token_data = id_to_token[i]; token_data.text = std::move(word); // ID → text token_data.score = scores ? scores[i] : 0.0f; token_data.attr = LLAMA_TOKEN_ATTR_NORMAL; } init_tokenizer(type); // initialization of the tokenizer by type (SPM, BPE etc.)}In the code one can see: determining the vocabulary type by tokenizer_model (llama → SPM, gpt2 → BPE with reading of the merges), searching for the token array in GGUF, a loop over all tokens — filling in id_to_token and token_to_id, the call to init_tokenizer(type) to initialize the tokenizer (SPM, BPE and so on). Further in the same function the special tokens (BOS, EOS, UNK and the like) are read and the add_bos, add_eos flags are set.
Special tokens:
- BOS (begin of sequence) — the beginning-of-sequence token, added before the prompt if needed.
- EOS (end of sequence) — the end-of-output token, upon which the application stops generation.
- UNK (unknown) — the token for unknown characters or those not present in the vocabulary.
Their IDs are stored in special_bos_id, special_eos_id, special_unk_id. In GGUF the vocabulary may have keys such as tokenizer.ggml.bos_token_id, tokenizer.ggml.eos_token_id and so on — they are read at the end of llama_vocab::impl::load and written into the corresponding fields. The add_bos and add_eos flags define whether these tokens should be added automatically during tokenization (it depends on the model and the chat format). After init_tokenizer(type) the vocabulary is ready for tokenization and for the reverse translation of tokens into text; these operations are used on every user request.
Step 6: Loading the Weights into Memory or onto the GPU
Step 6 — the model weights (tensors) are loaded from the file into memory or onto the GPU. The llama_model::load_tensors method (src/llama-model.cpp) creates the model tensors in memory (those very weight arrays), assigns buffers to them — regions of memory on the CPU or the graphics card — and fills them with data from the GGUF file. Below are quotes of the beginning of the method and of the key fragments.
bool llama_model::load_tensors(llama_model_loader & ml) { const auto & split_mode = params.split_mode; const auto & use_mlock = params.use_mlock; const auto & tensor_split = params.tensor_split; const int n_layer = hparams.n_layer; const int n_gpu_layers = this->n_gpu_layers(); // how many layers to load onto the GPU const bool use_mmap_buffer = true; LLAMA_LOG_INFO("%s: loading model tensors, this can take a while... (mmap = %s, direct_io = %s)\n", __func__, ml.use_mmap ? "true" : "false", ml.use_direct_io ? "true" : "false"); // Lists of buffer types for the CPU and for each graphics card — where the tensors end up depends on them pimpl->cpu_buft_list = make_cpu_buft_list(devices, params.use_extra_bufts, params.no_host); for (auto * dev : devices) { buft_list_t buft_list = make_gpu_buft_list(dev, split_mode, tensor_split); buft_list.insert(buft_list.end(), pimpl->cpu_buft_list.begin(), pimpl->cpu_buft_list.end()); pimpl->gpu_buft_list.emplace(dev, std::move(buft_list)); } ggml_backend_dev_t cpu_dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU); // The split points of the layers across devices (by free memory or tensor_split) const int i_gpu_start = std::max(int(hparams.n_layer) + 1 - n_gpu_layers, 0); const int act_gpu_layers = devices.empty() ? 0 : std::min(n_gpu_layers, int(n_layer) + 1); // For the layer number il we return the device and the list of buffers: the input is always CPU, the layers — by the split auto get_layer_buft_list = [&](int il) -> llama_model::impl::layer_dev { const bool is_swa = il < int(hparams.n_layer) && hparams.is_swa(il); if (il < i_gpu_start || (il - i_gpu_start) >= act_gpu_layers) { return {cpu_dev, &pimpl->cpu_buft_list}; } const int layer_gpu = std::upper_bound(splits.begin(), splits.begin() + n_devices(), float(il - i_gpu_start)/act_gpu_layers) - splits.begin(); auto * dev = devices.at(layer_gpu); return {dev, &pimpl->gpu_buft_list.at(dev)}; }; pimpl->dev_input = { cpu_dev, &pimpl->cpu_buft_list }; // the input tensors are always on the CPU pimpl->dev_layer.resize(n_layer); for (int il = 0; il < n_layer; ++il) { pimpl->dev_layer[il] = get_layer_buft_list(il); // each layer — on the CPU or on one of the GPUs } pimpl->dev_output = get_layer_buft_list(n_layer); // the output by the same splitWhat happens at the beginning of load_tensors and what for:
- lists of buffer types are formed for the CPU and for each graphics card
What for:
they determine onto which devices the model tensors will be placed — the input ones usually on the CPU, the layers according to the CPU/GPU split;
- from the settings (tensor_split) or from the free memory it is decided which layers to compute on the CPU and which on the GPU
What for:
to load the devices evenly and not to overflow the memory of one graphics card.
The get_layer_buft_list(il) function returns, for a layer number, the device and the list of buffers: the input is always on the CPU, the repeating layers may be on the CPU or the GPU, the output according to the split. Further on, the tensors are created in a loop: the embeddings, the output, and for each layer the attention matrices (Q/K/V/O), the normalizations, the feed-forward. Every tensor is created in the required buffer, the data is read from GGUF (ml.load_all_data(...) or mmap). As a result the model weights end up in memory (and on the graphics card if needed), and the model is ready to generate text.
The implementation of reading the weights from the file consists of the ml.get_tensor(name) and ml.load_tensor_data(tensor) calls (or mmap) in src/llama-model-loader.cpp: by the tensor name the offset and the size are taken from weights_map, and the data is copied into the tensor buffer on the CPU or the GPU. For large models mmap is used so as not to duplicate the data in RAM.
A quote of the load_tensor_data method (the loader, src/llama-model-loader.cpp**):**
// Step 6 (detail): loads the data of one tensor — from weights_map we know the offset in the filevoid llama_model_loader::load_tensor_data(ggml_tensor * tensor) { auto it = weights_map.find(ggml_get_name(tensor)); if (it == weights_map.end()) return; llama_tensor_weight & w = it->second; // mmap: a region of the file is mapped into memory (the data is not copied — it is read on access). Otherwise — we read the bytes into a buffer if (use_mmap) { ggml_backend_tensor_set_from_file(tensor, w.file, w.offset); } else { w.file->seek(w.offset, SEEK_SET); w.file->read_raw(tensor->data, ggml_nbytes(tensor)); }}What happens in load_tensor_data and what for:
- by the tensor name a record is looked up in weights_map (the map is built in the loader constructor).
What for:
without the map one cannot find out the offset and the size of the tensor data in the file.
- when use_mmap == true the file region is mapped into memory via ggml_backend_tensor_set_from_file — the data is not copied, it is read as it is accessed.
What for:
saving RAM and speeding up the start of loading.
- when use_mmap == false the file is positioned at w.offset and the bytes are read into tensor->data.
What for:
a full copy of the data into the buffer (needed if the file will be closed or the data will be changed).
A typical fragment of loading tensors in a loop over the layers (the logic from llama_model::load_tensors in src/llama-model.cpp): for every model tensor (the embeddings, the weights of the attention and feed-forward layers) ml.get_tensor(name) is called — the loader returns a pointer to a GGML tensor with an attached buffer; then ml.load_tensor_data(tensor) reads the data from the file into this buffer (or sets up mmap). The tensor names depend on the architecture (for example, blk.0.attn_q.weight, blk.0.attn_k.weight and so on). After the pass over all the tensors, the model weights are fully in memory (and on the GPU if needed).
// Simplified scheme of loading the tensors in load_tensors (the logic)for (const auto & it : ml.weights_map) { const std::string & name = it.first; ggml_tensor * cur = ggml_get_tensor(ml.ctx_gguf, name.c_str()); if (cur == nullptr) continue; ggml_tensor * tensor = ml.get_tensor(name); if (tensor == nullptr) continue; ml.load_tensor_data(tensor); // the tensor is already bound to a buffer on the CPU or GPU, the data is read or mapped}The get_tensor function in the loader finds the record in weights_map by name, creates a GGML tensor in the required backend (CPU or GPU according to the layer split) and returns the pointer. load_tensor_data reads the bytes at the offset in the file into the tensor buffer or sets up the mapping of the file into memory (mmap). This is how all the weights are loaded: the embeddings, the Q/K/V/O matrices, the normalizations, the feed-forward for every layer.
When mmap is used, the tensor data is not copied into RAM — instead a region of the file is mapped. This reduces RAM consumption and speeds up the start of loading, but it requires the file to stay open while the model is in use. With use_direct_io the reading is done with the alignment and parameters suitable for direct disk access. The split across layers (tensor_split or by the free GPU memory) defines which layers to load onto which device — this way a large model can be distributed across several graphics cards. After load_tensors finishes, the model is fully ready for inference: all the weights are in memory (and on the GPU if needed), and the context can be created and llama_decode can be called.
Step 7: Creating the Inference Context
Step 7 is creating the inference context (KV-cache, scheduler, reserved graphs). The loaded model stores only the weights (tensors). To generate text, an inference context is needed — the llama_context object.
What it is needed for:
- the context is the "working environment" of one generation session: it defines how many tokens the model can "remember" (the context size), how large the portions of data supplied are (the batch size), the RoPE parameters and the attention type;
- without a context one cannot call llama_decode — it is exactly the context that stores the KV-cache, the scheduler and the reserved graphs.
The implementation is in the file src/llama-context.cpp.
What is created when the context is created and what for:
- The scheduler (sched) — decides on which device (CPU or graphics card) to execute each node of the computation graph.
What for:
to distribute the computations across the CPU and the GPU and to allocate buffers for the graph on the required devices.
- The KV-cache memory (memory) — the buffers for the keys and values of the attention mechanism.
What for:
they store the already computed keys and values for all previous positions; without the cache, for every new token one would have to recompute K and V for the whole history, which is very slow (more details in the "KV-cache" section).
- The batch allocator (balloc) — fills in the positions and logit flags in the batch when needed.
What for:
so that the calling code does not have to set the positions manually and decide for which positions to compute the logits.
- The reserved graphs (Prefill and Decode) — temporary graphs and the buffers for them.
What for:
on the first decode call the graph is not built from scratch and memory is not allocated "on the fly" — this reduces the latency of the first answer.
The beginning of the constructor:
llama_context::llama_context( const llama_model & model, llama_context_params params) : model(model), balloc(std::make_unique<llama_batch_allocr>(model.hparams.n_pos_per_embd())) { LLAMA_LOG_INFO("%s: constructing llama_context\n", __func__); t_start_us = model.t_start_us; t_load_us = model.t_load_us; const auto & hparams = model.hparams; cparams.n_seq_max = std::max(1u, params.n_seq_max); if (cparams.n_seq_max > LLAMA_MAX_SEQ) { throw std::runtime_error("n_seq_max must be <= " + std::to_string(LLAMA_MAX_SEQ)); } cparams.n_threads = params.n_threads; cparams.n_threads_batch = params.n_threads_batch; cparams.yarn_ext_factor = params.yarn_ext_factor >= 0.0f ? params.yarn_ext_factor : hparams.yarn_ext_factor; cparams.yarn_attn_factor = params.yarn_attn_factor >= 0.0f ? params.yarn_attn_factor : hparams.yarn_attn_factor; // ... cparams.n_ctx = params.n_ctx == 0 ? hparams.n_ctx_train : params.n_ctx; cparams.rope_freq_base = params.rope_freq_base == 0.0f ? hparams.rope_freq_base_train : params.rope_freq_base; cparams.rope_freq_scale = params.rope_freq_scale == 0.0f ? hparams.rope_freq_scale_train : params.rope_freq_scale; // ... cparams.causal_attn = ...; cparams.n_batch = cparams.causal_attn ? std::min(cparams.n_ctx, params.n_batch) : params.n_batch; cparams.n_ubatch = std::min(cparams.n_batch, params.n_ubatch == 0 ? params.n_batch : params.n_ubatch); // ... LLAMA_LOG_INFO("%s: n_ctx = %u\n", __func__, cparams.n_ctx); LLAMA_LOG_INFO("%s: n_batch = %u\n", __func__, cparams.n_batch); LLAMA_LOG_INFO("%s: n_ubatch = %u\n", __func__, cparams.n_ubatch); // Initialization of the backends (GPU, CPU), memory (KV-cache), sched (the scheduler)}What is set in the context constructor:
- the reference to the model and the loading time are saved;
- the maximum number of sequences (n_seq_max), the number of threads, the YaRN/RoPE parameters, the context size (n_ctx), the batch size (n_batch, n_ubatch), the attention type (causal) are set.
What for:
- the size of the KV-cache, the maximum prompt length and how large the portions in which the batch is processed are depend on these parameters.
Next the backends (CPU and graphics cards) are initialized, the memory object memory (the KV-cache and the service buffers) is created, the scheduler sched (which distributes the graph nodes across the devices), the graphs for Prefill and Decode are reserved — so that on the first decode the graph is not built from scratch. After this llama_decode can be called with batches of tokens.
The n_batch and n_ubatch parameters:
- n_batch — the maximum number of tokens in one batch (during Prefill the batch may hold up to n_batch prompt tokens);
- n_ubatch — the maximum size of the ubatch that is processed in one process_ubatch call;
- if the prompt is longer than n_ubatch, it is split into ubatches of n_ubatch tokens, each of which is run through the model in turn;
- the logits are needed only for the last position in the batch, so only they are copied;
- reducing n_ubatch lowers the peak memory consumption at the cost of a larger number of passes.
The context is created once per session (or when the parameters change); one context can be used for many requests in a row without recreating it.
The implementation of the memory and scheduler initialization is in the llama_context constructor (file src/llama-context.cpp): ggml_backend_sched is created taking the list of the model's backends into account, then graph_reserve is called — temporary graphs are built for Prefill and Decode, the scheduler allocates buffers for them, so that on the first real decode the graph is merely placed into the already reserved memory. The memory object (implementation in src/llama-memory.cpp) allocates the buffers for the KV-cache for every layer and every position in the context.
The graph_reserve function (in src/llama-context.cpp) creates two reserved graphs: one for Prefill (many tokens at once), the other for Decode (one token). For each of them model.build_graph is called with the corresponding parameters (graph type, batch size), and then ggml_backend_sched_alloc_graph — the scheduler splits the graph across the backends and reserves the buffers. On the first process_ubatch call the graph is either reused (if the sizes match) or built anew; in any case the allocation for the large graphs has already been done when the context was created, which reduces the latency on the first decode.
Step 8: The Prompt Text Arrives from the User
Step 8 — the prompt text from the user arrives at the engine. After the context is created, the user sends a message (the prompt). The model works only with numbers — tokens. That is why the text is first turned into tokens (step 9), packed into a batch (step 10), and run through the model by the llama_decode call (steps 11–14): on the first request the batch holds all the prompt tokens (Prefill fills the KV-cache), and during generation one new token (Decode). From the logits the sampler chooses one next token (step 15), it is converted into text and printed (step 16); the token is added to the batch again, and the loop repeats until the "end of output" token (EOS) or a limit. Below each of these steps is analyzed through the code: what is called, what happens and what may be non-obvious.
Prefill and Decode — what the difference is and what for:
- On the first llama_decode call with the full prompt (Prefill) the model processes all the prompt tokens in one or several ubatches; K and V are computed for them and written into the KV-cache; the logits are requested only for the last position — the first token of the answer is chosen from them.
What for:
to "run" the whole prompt once and fill the cache of keys and values, so that afterwards generation goes token by token without recomputing the history.
- On the following calls the batch holds one new token (Decode); every time only the embedding of this token, one attention layer (Q, K, V for one position, reading K and V from the cache) and the remaining layers are computed; the logits are again only for the last position.
What for:
efficient generation token by token without recomputing the whole history — the old K and V are already in the cache.
// Simplified generation loop (steps 8–16): how the application calls the API (pseudocode following the logic of examples/)// Step 9: the prompt text → an array of tokensn_tokens = llama_tokenize(model, prompt, tokens, n_max, add_bos, true);llama_batch_clear(batch);for (int i = 0; i < n_tokens; i++) llama_batch_add(batch, tokens[i], i, {0}, false);llama_batch_set_logits(batch, n_tokens - 1, true); // we request the logits only for the last position (step 14)while (true) { if (llama_decode(ctx, batch) != 0) break; // steps 11–14: encode, ubatches, process_ubatch, logits in the buffer float * logits = llama_get_logits_ith(ctx, batch.n_tokens - 1); // step 14: read the logits llama_token next = llama_sampler_sample(sampler, ctx, batch.n_tokens - 1); // step 15: sampling if (next == llama_token_eos(model)) break; // EOS — the end of the output char buf[256]; int n = llama_token_to_piece(model, next, buf, sizeof(buf)); // step 16: token → text printf("%.*s", n, buf); llama_batch_clear(batch); llama_batch_add(batch, next, n_cur, {0}, true); // one new token — the next iteration will be Decode n_cur++;}Step 9: Tokenization — from Text to a Sequence of Tokens
Step 9 — the prompt text is turned into a sequence of tokens (integers). The text has to be turned into a sequence of integers — tokens.
What this is needed for:
- the model (the transformer layers) accepts as input not strings but vectors of numbers of fixed length;
- one row in the model's embedding table corresponds to every token.
Tokenization is the first step:
- by the vocabulary the text is cut into pieces (tokens), and each piece is matched with a number (ID);
- further on, the embeddings are taken by these IDs and fed into the model.
What a token is:
- a token is an ID (an integer) that corresponds to a piece of text: a whole word, a part of a word or a service character;
- the model vocabulary defines the "text ↔ tokens" correspondence: from text one can get an array of tokens (tokenization), and from a token the text (token_to_piece).
In the API (the include/llama.h header) the llama_tokenize function is declared; the implementation delegates the call to the model vocabulary.
The implementation in src/llama-vocab.cpp:
// Step 9: the prompt text is turned into an array of tokens (IDs); the model vocabulary sets the "text <-> tokens" correspondenceint32_t llama_tokenize( const struct llama_vocab * vocab, const char * text, int32_t text_len, llama_token * tokens, int32_t n_tokens_max, bool add_special, bool parse_special) { return vocab->tokenize(text, text_len, tokens, n_tokens_max, add_special, parse_special);}Parameters:
- vocab — the model vocabulary;
- text and text_len — the prompt string;
- tokens — the array for writing the tokens;
- n_tokens_max — its size;
- add_special — whether to add the service beginning token (BOS);
- parse_special — whether to process special tags.
The real logic (BPE, SentencePiece and so on) is in the vocab->tokenize method in src/llama-vocab.cpp. The result is the number of tokens written. Tokenization is performed on every new user message; the vocabulary does not change and has already been loaded when the model was loaded.
The implementation of tokenization is the llama_vocab::impl::tokenize method in src/llama-vocab.cpp. Inside, depending on the vocabulary type (SPM, BPE, WPM and so on), a tokenizer session is created and its tokenize is called; for BPE, for example, llm_tokenizer_bpe_session is used, which splits the text by regular expressions and assembles the tokens by the merges.
Tokenizer types in the vocabulary:
- LLAMA_VOCAB_TYPE_SPM — SentencePiece-like (the LLaMA models and others), splitting by the SPM rules and vocabulary;
- LLAMA_VOCAB_TYPE_BPE — Byte Pair Encoding (GPT-2 and others), the merges of byte/substring pairs are stored in GGUF, tokenization is greedy merging;
- LLAMA_VOCAB_TYPE_WPM — WordPiece and similar;
- LLAMA_VOCAB_TYPE_NONE — the vocabulary is not used (for example, for CLIP). The tokenizer_st_partition function splits the input text into fragments: ordinary text and special tags (if parse_special is enabled) — the tags are turned into a single token, and the rest goes into the tokenizer according to the vocabulary type. Fragment:
std::vector<llama_token> llama_vocab::impl::tokenize( const std::string & raw_text, bool add_special, bool parse_special) const { GGML_ASSERT(tokenizer && "Tokenizer not initialized."); std::vector<llama_token> output; std::forward_list<fragment_buffer_variant> fragment_buffer; if (!raw_text.empty()) { fragment_buffer.emplace_front(raw_text, 0, raw_text.length()); tokenizer_st_partition(fragment_buffer, parse_special); } switch (get_type()) { case LLAMA_VOCAB_TYPE_SPM: { llm_tokenizer_spm_session session(vocab); if (add_special && add_bos) output.push_back(special_bos_id); for (const auto & fragment : fragment_buffer) { if (fragment.type == FRAGMENT_BUFFER_VARIANT_TYPE_RAW_TEXT) { std::string text = fragment.raw_text.substr(fragment.offset, fragment.length); session.tokenize(text, output); } else { output.push_back(fragment.token); } } if (add_special && add_eos) output.push_back(special_eos_id); } break; case LLAMA_VOCAB_TYPE_BPE: { llm_tokenizer_bpe_session session(vocab, *static_cast<llm_tokenizer_bpe*>(tokenizer.get())); if (add_special) session.append_bos(output); for (const auto & fragment : fragment_buffer) { if (fragment.type == FRAGMENT_BUFFER_VARIANT_TYPE_RAW_TEXT) session.tokenize(fragment.raw_text.substr(fragment.offset, fragment.length), output); else session.append(fragment.token, output); } if (add_special) session.append_eos(output); } break; default: break; } return output;}Step 10: Forming the Batch for One Call
Step 10 — the tokens are packed into a batch for one call to the model. One call to the model is passed through the llama_batch structure (in include/llama.h).
What the batch is needed for:
- the llama_decode function accepts exactly one batch — a set of tokens (and service fields: positions, sequence identifiers, logit flags);
- this way the engine knows which tokens to process in one pass, at which positions they stand and for which positions the logits must be returned (usually only for the last one — in order to choose the next token);
- on the first request the batch usually holds all the prompt tokens; during token-by-token generation it holds one new token.
// Step 10: the batch is a "package" of tokens for one llama_decode call; token[], pos[], seq_id[], logits[] are filled in by the application or by balloc->inittypedef struct llama_batch { int32_t n_tokens; llama_token * token; float * embd; llama_pos * pos; int32_t * n_seq_id; llama_seq_id ** seq_id; int8_t * logits;} llama_batch;The structure of the batch:
- the arrays have the size n_tokens;
- either token IDs (token) or ready embeddings (embd) are passed — vectors of numbers into which the tokens have already been turned (usually tokens are passed and the embeddings are computed inside);
- pos — the position of every token; seq_id and n_seq_id — which sequence the token belongs to;
- logits[i] != 0 means that for position i the logits must be returned (usually only for the last one — in order to choose the next token). Before llama_decode the batch is processed by the llama_batch_allocr class (file src/llama-batch.cpp): the init method checks the batch and, if fields are missing, fills them in automatically (positions from memory, logits only for the last token).
The seq_id and n_seq_id fields are used when batching several sequences (for example, several requests in one batch): every token can belong to one or several sequences; the positions (pos) are counted separately for each sequence by memory->seq_pos_max(seq_id). In the typical case there is one sequence — all the tokens have one and the same seq_id (for example, 0), and the positions go in order 0, 1, 2, ... . The batch is cleared and filled anew before every llama_decode call; during token-by-token generation the batch holds one token with position n_cur.
The implementation of the llama_batch_allocr::init method (file src/llama-batch.cpp):
// Step 10 (continued): balloc->init fills in the positions and the logit flags in the batch (if the application did not); the logits are usually only for the last positionbool llama_batch_allocr::init( const llama_batch & batch_inp, const llama_vocab & vocab, const llama_memory_i * memory, uint32_t n_embd, uint32_t n_seq_max, bool output_all) { clear(); batch = batch_inp; this->vocab = &vocab; GGML_ASSERT(batch.n_tokens > 0); if (batch.token) { for (int32_t i = 0; i < batch.n_tokens; ++i) { if (batch.token[i] < 0 || (uint32_t) batch.token[i] >= vocab.n_tokens()) { LLAMA_LOG_ERROR("%s: invalid token[%d] = %d\n", __func__, i, batch.token[i]); return false; } } } if (!batch.n_seq_id) { n_seq_id.resize(batch.n_tokens); for (int32_t i = 0; i < batch.n_tokens; i++) { n_seq_id[i] = seq_id_0.size(); } batch.n_seq_id = n_seq_id.data(); } if (!batch.seq_id) { seq_id.resize(batch.n_tokens + 1); seq_id[batch.n_tokens] = NULL; for (int32_t i = 0; i < batch.n_tokens; i++) { seq_id[i] = seq_id_0.data(); } batch.seq_id = seq_id.data(); } if (!batch.pos) { pos.resize(batch.n_tokens); llama_pos p0[LLAMA_MAX_SEQ]; for (uint32_t s = 0; s < n_seq_max; ++s) { p0[s] = memory ? memory->seq_pos_max(s) + 1 : 0; } for (int32_t i = 0; i < batch.n_tokens; i++) { pos[i] = p0[batch.seq_id[i][0]]; for (int32_t s = 0; s < batch.n_seq_id[i]; ++s) { p0[batch.seq_id[i][s]] = pos[i] + 1; } } batch.pos = pos.data(); } if (!batch.logits) { if (output_all) { output.resize(batch.n_tokens, true); } else { output.resize(batch.n_tokens, false); output[output.size() - 1] = true; } batch.logits = output.data(); } return true;}In the code:
- the tokens are checked for validity;
- if n_seq_id and seq_id are missing, they are filled with zero sequences;
- if pos is missing, the positions are taken from memory (memory->seq_pos_max(s) + 1) and set in order;
- if logits is missing, the logits are requested only for the last token (or for all, if output_all).
Step 11: Tokens Are Turned into Embeddings
Step 11 — inside decode the tokens (if it is exactly they, and not ready embeddings, that are passed in the batch) are turned into embeddings. In the batch one can pass either token IDs (token) or ready embeddings (embd). In the typical case the application passes tokens — then inside llama_context::decode, before the batch is processed, encode is called.
What encode is needed for:
- the model (the transformer layers) works not with token numbers but with vectors of numbers of fixed length — embeddings;
- encode turns every token into such a vector, copying the corresponding row from the model's embedding table into the input tensor of the graph;
- without encode the graph would not receive a correct input.
What the embedding table is:
- it is a matrix of model weights of size "vocabulary size × embedding size"; one row is the vector of numbers for one token;
- it is loaded in load_tensors (a tensor with a name like tok_embeddings or embed_tokens);
- during encode, for every token from the batch the row with index token[i] is taken and copied into the input tensor of the graph at position i;
- after this the graph computes on the embeddings (the transformer layers, attention, feed-forward, the logits).
Where this is in the code: in llama_context::decode (file src/llama-context.cpp) it is checked whether a batch with tokens or with embeddings was passed. If with tokens, an internal function (or method) is called that, for every element of the batch, takes batch.token[i], finds the embedding tensor in the model and copies the row with index token[i] into the input tensor of the graph at position i. One encode call fills the input layer of the graph with the embeddings for all the tokens of the batch. Then process_ubatch builds the graph and performs the computations on these embeddings. During decode of a single token, one row of the embedding matrix is copied into the input buffer of the graph.
A quote of the encode logic (obtaining the embeddings by token IDs):
// Step 11 (encode): for every token in the batch we take a row of the model embedding table (index = token[i]) and copy it into the input tensor of the graph — the model computes on vectors, not on IDs// In decode before process_ubatch (if tokens, and not embeddings, were passed in the batch):for (int32_t i = 0; i < batch.n_tokens; i++) { llama_token tid = batch.token[i]; // tok_embeddings — a model tensor of size [n_vocab, n_embd]; one row is the embedding vector for one token float * row = (float *) ((char *) model.tok_embeddings->data + tid * ggml_row_size(model.tok_embeddings->type, n_embd)); memcpy(embd_buffer + i * n_embd, row, n_embd * sizeof(float));}// What for: the model (the transformer layers) accepts vectors of a fixed length as input, not token IDsWhat happens during encode and what for:
- for every index i in the batch the token ID batch.token[i] is taken.
What for:
by the ID one needs to obtain the embedding vector from the model weight table.
- from the tok_embeddings (or embed_tokens) tensor the row with index tid is read — it is a vector of length n_embd.
What for:
this row is exactly the embedding of the token — the input for the first transformer layer.
- the row is copied into the input buffer of the graph at position i.
What for:
during execution the graph will read the embeddings from this buffer; without encode the buffer would be empty or contain garbage.
Steps 11–12: Running the Batch Through the Model (Entering decode, Ubatches)
Steps 11 and 12 — the application calls llama_decode, passing the batch; inside, the conversion of the tokens into embeddings is performed if needed (step 11), then the splitting of the batch into ubatches (step 12) and the running of each ubatch through the model (step 13). Below are the quotes of the entry point, of the loop over the ubatches and of the splitting of the batch.
What the function llama_decode is for:
- it runs one batch of tokens through the model and fills the internal buffer of the context with logits — the "raw" scores for every possible next token;
- from these logits the application (via the sampler) chooses one next token and, if needed, calls decode again with one new token in the batch;
- this repeats until the end of the answer (EOS) or a limit.
The public API is llama_decode (in src/llama-context.cpp):
// Step 11: the decode entry point — the application passes the batch; inside: encode (if needed), ubatches, process_ubatch, logits in the bufferint32_t llama_decode(llama_context * ctx, llama_batch batch) { const int ret = ctx->decode(batch); // 0 — success, 1 — more data is needed, negative — error if (ret != 0 && ret != 1) { LLAMA_LOG_ERROR("%s: failed to decode, ret = %d\n", __func__, ret); } return ret;}In the llama_context::decode method the following happens. The implementation is in the file src/llama-context.cpp: the llama_context::decode(const llama_batch & batch) method checks the batch, calls encode if needed (if tokens and not embeddings were passed), then calls balloc->init(batch, ...), reserves the scheduler and updates the memory (KV-cache), after which in a loop it obtains the ubatches via memory->init_batch and calls process_ubatch for each of them; the logits are copied into the internal buffer of the context.
A quote of the loop over the ubatches inside decode (the scheme, src/llama-context.cpp**):**
// In decode after balloc->init and the memory update: the loop over ubatches (step 12 → step 13)while (memory->init_batch(batch, ubatch)) { if (ubatch.n_tokens == 0) break; auto * res = process_ubatch(ubatch, gtype, mctx, status); // step 13: graph + execution if (!res) { ...; break; } // Copying the logits for the positions with logits[i]==true into the context buffer (step 14) copy_logits(res, ...);}// After the loop the application reads the logits via llama_get_logits_ithFirst the batch is checked:
- whether tokens or ready embeddings were passed;
- if needed, encode is called.
Then the batch is initialized via balloc->init — this is needed in order to fill in the token positions and to decide for which positions to compute the logits (usually only for the last one). The scheduler is reserved and the memory (KV-cache) is updated.
Next, in the loop, memory->init_batch yields ubatches (ubatch) of size no larger than n_ubatch — this is how a large batch is split into parts that are run through the model in turn. For each ubatch process_ubatch(...) is called: inside it the computation graph is built (the embeddings, the transformer layers, the output into logits), the computation is performed on the CPU/GPU, and tensors with the logits are returned — the probabilities over the vocabulary.
A quote of the logic of splitting the batch into ubatches (init_batch, src/llama-memory.cpp**):**
// Step 12: returns the next ubatch of size no greater than n_ubatch; in decode it is called in a loopbool llama_memory::init_batch(const llama_batch & batch, llama_ubatch & ubatch) { if (batch.n_tokens == 0) return false; const uint32_t n_ubatch = cparams.n_ubatch; // From the full batch we take the next n_ubatch tokens (or the remainder) — this way a long prompt does not overflow memory ubatch.n_tokens = std::min((uint32_t) batch.n_tokens, n_ubatch); // Copy into the ubatch: token[], pos[], seq_id[], logits[] for this portion // ... (the implementation of copying and shifting the indices for the next call) return true; // on the next call the next ubatch will be returned; when the tokens run out — false}What init_batch does and what for:
- the next portion of tokens of size no larger than n_ubatch is taken from the full batch.
What for:
one process_ubatch call processes a limited number of tokens — this way the memory for the intermediate tensors of the graph does not overflow.
- the tokens, positions and logit flags for this portion are copied into ubatch.
What for:
process_ubatch receives a ready ubatch and builds the graph only for it.
- on the next call init_batch returns the next ubatch, until all the tokens of the batch have been processed.
What for:
a long prompt is processed in several passes through the model without a single huge graph.
The implementation of the loop over the ubatches in llama_context::decode (the scheme): while memory->init_batch(batch, ubatch) returns a ubatch with ubatch.n_tokens > 0, process_ubatch(ubatch, gtype, mctx, status) is called. The graph type gtype is Prefill if the ubatch holds more than one token (or it is the first pass over the prompt), otherwise Decode. After every process_ubatch the logits for the positions with logits[i] == true are copied into the internal buffer of the context, from where llama_get_logits_ith reads them. The loop finishes when all the tokens of the batch have been processed. On the first decode with the full prompt the graph type is Prefill; on the subsequent decodes with one token it is Decode; the scheduler and the graph switch between the reserved graphs according to the type.
The order of calls during one decode (summary):
- llama_decode(ctx, batch) → ctx->decode(batch);
- in decode: checking the batch, encode if needed, balloc->init(batch, ...), reserving the scheduler, updating the memory (KV-cache);
- the loop: memory->init_batch returns the next ubatch, process_ubatch(ubatch, ...) is called — inside it model.build_graph, res->set_inputs, graph_compute; the logits are copied into the context buffer;
- after the loop the application reads the logits via llama_get_logits_ith, and the sampler returns the next token.
The logits are copied into the output buffer of the context. From there llama_get_logits_ith reads them — from these numbers the sampler chooses the next token (for example, the most probable one or by temperature/top_p).
The return value of llama_decode: 0 — success; 1 — more input data is needed (in some batching modes); a negative value — an error (for example, GGML_STATUS_ALLOC_FAILED when there is not enough memory, GGML_STATUS_FAILED on a computation error). The application must check the return value and, on an error, stop the generation or print an error message.
Step 13: Preparing to Run the Ubatch (Memory, Graph Parameters)
Step 13 is split in the article into three parts: preparing to run the ubatch, building the graph and allocating the buffers, writing the inputs and executing the graph. All the work on one ubatch is performed in the llama_context::process_ubatch method (file src/llama-context.cpp). It is called from llama_context::decode in a loop: for every ubatch returned by memory->init_batch, process_ubatch is called once. Inside, the following happens step by step: preparing the memory, deciding whether to reuse an already built graph or to build a new one, building the graph and allocating buffers if needed, writing the input data into the graph, executing the graph on the CPU/GPU. Below is the first part of step 13: the entry point and the preparation.
What the computation graph is needed for:
- the model (a transformer) is a chain of operations: the embeddings, the attention layers (Q, K, V, softmax, weighted sum), the normalizations, the feed-forward and so on;
- instead of calling every operation manually, the engine builds a graph — a list of nodes (operations) and the links between them;
- the scheduler then traverses the graph in topological order and executes the operations on the CPU or the GPU;
- this way the nodes can be distributed across the devices automatically and the graph can be reused when the batch sizes are the same.
The signature and the input data of process_ubatch (a quote from src/llama-context.cpp**):**
// Step 13: one ubatch is run through the model; called from decode in the loop over ubatchesllm_graph_result * llama_context::process_ubatch(const llama_ubatch & ubatch, llm_graph_type gtype, llama_memory_context_i * mctx, ggml_status & ret) { // 1) Update the memory context (KV-cache, service buffers) before building the graph if (mctx && !mctx->apply()) { ret = GGML_STATUS_FAILED; return nullptr; } // 2) Take the reserved graph result (it holds the graph and the buffers) auto * res = gf_res_prev.get(); auto * gf = res->get_gf(); // 3) Form the graph parameters: batch size, Prefill or Decode type, memory context const auto gparams = graph_params(res, ubatch, mctx, gtype); // ... then the decision about reuse, if needed build_graph and set_inputs, graph_compute}What happens at the beginning of process_ubatch and what for:
- mctx->apply() is called if a memory context (mctx) was passed.
What for:
- the memory context updates the state of the KV-cache and the service buffers (for example, it shifts the pointers to the next free positions); before the graph is built, the graph will access these buffers — they must be in an up-to-date state.
- A pointer to the reserved graph result is taken: res = gf_res_prev.get(), gf = res->get_gf().
What for:
- when the context was created, the graphs and the buffers for Prefill and Decode were already reserved; gf_res_prev points to whichever of them corresponds to the current call type (Prefill or Decode).
- The graph parameters are formed: gparams = graph_params(res, ubatch, mctx, gtype).
What for:
- gparams receives the batch size (ubatch.n_tokens), the graph type (Prefill — many tokens, or Decode — one token), the references to the memory context; based on them it is decided further on whether to reuse the graph, and during the building the sizes of the nodes and inputs are set.
If mctx->apply() returns false, the function immediately returns nullptr and GGML_STATUS_FAILED is written into ret — the calling code (decode) will handle the error and stop the loop over the ubatches.
A quote in terms of meaning: what mctx->apply() does and what goes into gparams (according to the code in src/llama-context.cpp**,** src/llama-memory.cpp**):**
// mctx->apply() — updates the KV-cache and the service buffers: shifts the pointers to the next free positions,// applies the deferred updates; returns true on success, false on errorif (mctx && !mctx->apply()) { ret = GGML_STATUS_FAILED; return nullptr; }
// graph_params(res, ubatch, mctx, gtype) — gathers into a structure: n_tokens, the graph type (Prefill/Decode),// references to the KV-cache buffers and the input data; by gparams can_reuse is then decided and the graph is builtconst auto gparams = graph_params(res, ubatch, mctx, gtype);Step 13: Building the Computation Graph and Allocating the Buffers
Step 13 (continued) — building the graph and allocating buffers for it. After the memory has been prepared and gparams formed, the engine decides: is it possible to reuse the already built graph (the same batch sizes and the same Prefill/Decode type), or does the graph have to be built anew and buffers allocated for it. If the graph is reused, it immediately proceeds to writing the inputs and executing. If not, res->reset(), ggml_backend_sched_reset(sched.get()), model.build_graph(gparams) and ggml_backend_sched_alloc_graph(sched.get(), gf) are called. Below is a step-by-step account of what happens after what and what for; with several quotes from the code.
A quote: the decision about reuse and the building of the graph (a fragment of process_ubatch**,** src/llama-context.cpp**):**
// Reuse: with the same size and type the graph is not rebuilt — faster on repeated decode with a single token if (!graph_reuse_disable && res->can_reuse(gparams)) { n_reused++; // the reuse counter (for profiling) } else { res->reset(); // reset the graph result (the old nodes and buffers) ggml_backend_sched_reset(sched.get()); // reset the scheduler gf = model.build_graph(gparams); // build the graph: embeddings → transformer layers → logits if (!gf) { ret = GGML_STATUS_FAILED; return nullptr; } if (!ggml_backend_sched_alloc_graph(sched.get(), gf)) { // the scheduler allocates buffers for all the graph tensors on CPU/GPU ret = GGML_STATUS_ALLOC_FAILED; return nullptr; } }Step by step (what happens and what for):
- The check res->can_reuse(gparams): do the batch size and the graph type match those for which the graph was built last time.
What for:
- during token-by-token generation (Decode) every next process_ubatch call receives a ubatch of one token and the same Decode type — the graph does not have to be rebuilt, one only has to substitute the new inputs and execute it; this speeds up generation greatly.
- If reuse is disabled or the parameters have changed: res->reset() — the old graph nodes and the attached buffers are cleared.
What for:
- before a new graph is built the old one must be reset, otherwise the nodes and tensors will accumulate.
- ggml_backend_sched_reset(sched.get()) — the scheduler resets its state (the distribution of the nodes across the backends, the buffers reserved for this graph).
What for:
- the scheduler will determine anew on which device to execute every node of the new graph and allocate memory for it.
- gf = model.build_graph(gparams) — a GGML graph is built: nodes for the embeddings and the positions, then for every transformer layer — attention (Q, K, V, softmax, weighted sum), normalization, feed-forward, normalization again; and at the end the output layer into the logits (the vocabulary size).
What for:
- the graph describes which operations to perform and in what order; without it the scheduler would have nothing to execute. The implementation of build_graph depends on the architecture (LLaMA, Gemma and so on) and is located in src/llama-model.cpp or in specialized files.
- ggml_backend_sched_alloc_graph(sched.get(), gf) — the scheduler traverses the graph, determines the backend for every tensor (CPU or GPU according to the layer split), allocates a buffer on this backend (or reuses the one reserved when the context was created).
What for:
- without allocating buffers there is nowhere to store the graph data; on the first decode the buffers are reserved here (or during graph_reserve when the context is created), and on subsequent reuses of the graph no repeated allocation is needed.
The structure of the graph by layers (the scheme):
- the input: the embeddings of the ubatch tokens + the positional encodings (RoPE) by the positions from ubatch;
- for every transformer layer: normalization of the input → the attention block (Q = input × W_q, K = input × W_k, V = input × W_v; applying RoPE to Q and K; attention scores = Q × K^T; the mask (causal); softmax; the weighted sum scores × V; the linear layer O) → the residual connection → normalization → feed-forward (two linear layers with an activation between them) → the residual connection;
- after all the layers: the final normalization → the output layer (a matrix of "hidden layer size × vocabulary size") → the logits (a tensor of size [n_tokens, n_vocab]).
A quote of the build_graph and ggml_backend_sched_alloc_graph calls (in terms of the meaning of the code in src/llama-context.cpp and GGML):
// model.build_graph(gparams) — returns a GGML graph (a ggml_context with nodes); gparams sets n_tokens, the Prefill/Decode type, the memory contextggml_graph * gf = model.build_graph(gparams);
// ggml_backend_sched_alloc_graph — the scheduler traverses the graph, chooses a backend (CPU/GPU) for every tensor and allocates a bufferbool ok = ggml_backend_sched_alloc_graph(sched.get(), gf);if (!ok) { ret = GGML_STATUS_ALLOC_FAILED; return nullptr; }The reservation of the graph for Prefill and Decode is done in graph_reserve when the context is created: a temporary ubatch is created, model.build_graph is called, the scheduler splits the graph across the devices and reserves the buffers — so that on the first decode no time is spent on allocating memory. The ggml_backend_sched_alloc_graph function (the GGML library) during the allocation determines for every tensor of the graph the backend according to the split of the model layers and allocates a buffer on the corresponding device.
Step 13: Writing the Inputs into the Graph and Executing on CPU/GPU
Step 13 (conclusion) — writing the inputs into the graph and executing. After the graph has been built (or reused), the data of the current ubatch — the tokens or the embeddings and the positions — has to be written into the input tensors of the graph, and the execution of the graph has to be launched. This is done by the res->set_inputs(&ubatch) and graph_compute(res->get_gf(), ubatch.n_tokens > 1) calls. Below are the quotes and a step-by-step explanation.
A quote: writing the inputs and executing (the end of process_ubatch**,** src/llama-context.cpp**):**
res->set_inputs(&ubatch); // write into the input tensors of the graph: the embeddings (or tokens) and the positions for this ubatch const auto status = graph_compute(res->get_gf(), ubatch.n_tokens > 1); // traversal of the graph nodes, execution on CPU/GPU if (status != GGML_STATUS_SUCCESS) { ret = status; return nullptr; } ret = GGML_STATUS_SUCCESS; return res; // in res — the tensors with the logits; decode then copies them into the context buffer for llama_get_logits_ith}What happens during set_inputs and what for:
- res->set_inputs(&ubatch) copies the ubatch data into the input tensors of the graph: the embeddings of the tokens (or the token IDs themselves, if the graph accepts them and performs the lookup in the embedding table itself) and the positions for every position in the ubatch; if needed, the pointers to the KV-cache buffers are substituted (where to write the new K and V and where to read the already saved ones from).
What for:
- the graph computes on concrete data; without writing the inputs it would work with empty or outdated tensors; during Decode the buffers into which the new keys and values for the current position are appended are substituted into the cache.
A quote of the graph_compute call (executing the graph on the scheduler, usually in the same file or in src/llama-context.cpp**):**
// The scheduler traverses the graph nodes in topological order and executes every operation on the assigned backend (CPU/GPU)ggml_status graph_compute(llm_graph_result * res, bool capture) { ggml_backend_sched * sched = ctx->sched.get(); ggml_backend_sched_begin(sched); // preparing the scheduler: reserving buffers, the order of the nodes ggml_backend_sched_run(sched, res->get_gf()); // traversal of the graph, execution of every node on its own backend ggml_backend_sched_end(sched); // finishing and synchronizing the backends return ggml_backend_sched_get_status(sched);}What graph_compute does step by step and what for:
- ggml_backend_sched_begin(sched) prepares the scheduler for executing the graph: it fixes the order of node traversal (topological sorting) and, if needed, reserves the intermediate buffers.
What for:
- the graph nodes must be executed in such an order that by the moment a node is executed all its inputs have already been computed; the scheduler ensures this order.
- ggml_backend_sched_run(sched, res->get_gf()) traverses the graph: for every node it determines the backend (CPU or GPU according to the model split), copies the input data onto this device if needed, and launches the operation (matrix multiplication, softmax, addition and so on).
What for:
- this is how all the transformer layers are executed; as a result the output tensors of the graph — the logits — are filled with numbers (one per each token of the vocabulary for every position in the ubatch for which the logits were requested).
- ggml_backend_sched_end(sched) finishes the execution and synchronizes the backends (for example, waits for the operations on the GPU to finish).
What for:
- after this the output tensors of the graph (the logits) are ready to be read; the calling code (decode) copies them into the context buffer, from where the application reads the logits via llama_get_logits_ith.
The capture parameter in graph_compute(res->get_gf(), ubatch.n_tokens > 1) is used in some builds for debugging or profiling (for example, capturing the graph for visualization). The return value of graph_compute is a GGML status (success or an error code); on success process_ubatch returns res, and in decode the logits from res are copied into the internal buffer of the context for the positions with logits[i] == true.
When the graph is reused (when the batch size and the graph type match the previous call), the graph is not rebuilt and the buffers are not reallocated — only res->set_inputs(&ubatch) and graph_compute are called. This speeds up the repeated decode calls during token-by-token generation: the Decode graph is built once on the first decode with a single token and is reused afterwards. The n_reused counter in the context is incremented on every reuse of the graph — from it one can estimate the share of reuses during profiling.
Steps 14–15: Logits in the Buffer and Choosing the Next Token (Sampling)
Steps 14 and 15 — after the ubatch has been run, the logits for the last position end up in the context buffer (step 14); from them the sampler chooses one next token (step 15). After llama_decode the internal buffer of the context holds the logits — one number per each token of the vocabulary.
What the logits are needed for:
- at the output the model produces not one token but "scores" for all the possible next tokens (one number — a logit — per each token of the vocabulary);
- from these numbers the sampler decides which single token to choose: for example, the one with the maximum logit (greedy choice) or randomly taking temperature/top_p into account;
- without the logits the application would not be able to choose the next token.
The logits can be understood as "raw" scores of how well each next token fits; from them the sampler chooses one token. Access to the logits is via llama_get_logits_ith (file src/llama-context.cpp):
- a pointer to an array of n_vocab floats is returned — one number per each token of the vocabulary.
The logits are requested only for those positions in the batch for which the logits[i] == true flag was set in the batch (usually only for the last position). The context copies the logits into the internal buffer after every process_ubatch call for the positions with this flag; with several ubatches in one decode the buffer keeps the logits for the last such position (usually the last token in the batch). The size of the logits buffer in the context is n_vocab (one float per each token of the vocabulary).
The implementation of llama_get_logits_ith in src/llama-context.cpp returns a pointer to the logits for a given position in the batch (usually the logits for the last position are requested — the next token is chosen from them):
// Step 14: returns a pointer to the logits for position i (usually i = the last position in the batch)// An array of n_vocab floats — one number per each token of the vocabulary; the sampler chooses the next token from themfloat * llama_get_logits_ith(struct llama_context * ctx, int32_t i) { return ctx->get_logits_ith(i);}From these numbers the next token is chosen with the help of the sampler — a component that decides from the logits which token to produce (for example, the most probable one or a random one taking temperature/top_p into account). In the API (include/llama.h) the llama_sampler_sample(...) function takes the logits, applies a chain of samplers to them and returns one token. This token is added to the history and on the next llama_decode call it is passed in the batch as the only new token; the loop repeats until the end of the answer (the EOS token — "end of output") or a limit. To translate a token back into text, use llama_token_to_piece / llama_detokenize (the implementation is in src/llama-vocab.cpp).
The chain of samplers:
- sampling in llama.cpp is arranged as a sequence of steps;
- first a shift by repetitions (repeat penalty) may be applied to the logits — reducing the probability of the tokens that have already appeared;
- then temperature is applied — dividing the logits by a number (temperature > 1 makes the distribution softer, < 1 sharper);
- next may come top-p (nucleus): only the tokens with a cumulative probability up to the threshold p are kept;
- after this one token is chosen from the remaining ones — for example, the one with the maximum logit (greedy choice) or randomly with the probabilities from the softmax of the logits;
- the implementation of the chain is in the sampler code (for example, common/sampling.cpp or an analogue in the repository): a loop over the registered samplers, each of which modifies the logits or chooses a token.
The implementation of translating a token into text consists of the llama_vocab::impl::token_to_piece and llama_vocab::impl::detokenize methods in src/llama-vocab.cpp. token_to_piece returns a string (a piece of text) for a token ID; for byte tokens a conversion into a character is performed, and for ordinary ones the text is taken from id_to_token. detokenize goes over an array of tokens and glues the results of token_to_piece into a single string. The llama_token_to_piece API (in include/llama.h) accepts the model, the token, a buffer and its size; inside, vocab.token_to_piece is called. For token-by-token output llama_token_to_piece is usually used; for assembling the full string from an array of tokens — llama_detokenize. Fragment:
// Step 16: by the token ID we return a piece of text (for output to the user); id_to_token was loaded in load_vocabint32_t llama_vocab::impl::token_to_piece(llama_token token, char * buf, int32_t length, int32_t lstrip, bool special) const { if (0 <= token && token < (int32_t) id_to_token.size()) { const std::string & token_text = id_to_token[token].text; switch (get_type()) { case LLAMA_VOCAB_TYPE_SPM: case LLAMA_VOCAB_TYPE_UGM: if (attr & LLAMA_TOKEN_ATTR_NORMAL) { std::string result = token_text; llama_unescape_whitespace(result); memcpy(buf, result.data(), result.size()); return (int32_t) result.size(); } if (attr & LLAMA_TOKEN_ATTR_BYTE) { char byte = (char) token_to_byte(token); buf[0] = byte; return 1; } break; case LLAMA_VOCAB_TYPE_BPE: if (attr & LLAMA_TOKEN_ATTR_NORMAL) { std::string result = llama_decode_text(token_text); memcpy(buf, result.data(), result.size()); return (int32_t) result.size(); } break; default: break; } } return 0;}In the code one can see: for SPM and UGM the ordinary tokens are turned into text via llama_unescape_whitespace, and the byte ones via token_to_byte; for BPE llama_decode_text is used (decoding of the escape sequences). The llama_detokenize function in the API calls vocab.detokenize and assembles the string from the array of tokens.
The sampler in the API (for example, llama_sampler_sample) accepts the context, the position in the batch and optionally a chain of sampling parameters (temperature, top_p, repeat_penalty and so on). Inside, the logits for this position are read via ctx->get_logits_ith(pos), a chain of samplers (repeat penalty, temperature, top_p and so on) is applied to them, and then one token is chosen (greedy or random by softmax). The returned token is added to the history and on the next llama_decode call it is passed in the batch as the only new token.
KV-cache and the Prefill / Decode Stages (For Reference)
In the attention mechanism every token gets a query (Q), a key (K) and a value (V) vector — these are numeric vectors that the model computes from its weights.
What Q, K, V are needed for:
- from them the "attention" is computed — how important every previous position is for the current one;
- the result is a weighted sum of the values V with the weights from Q and K.
The logits for the next token depend on the Q of the current position and on the K, V of all previous positions — that is why during token-by-token generation the old K and V do not change, and it is enough to compute them once and save them. The KV-cache is a buffer in memory in which the already computed keys and values are stored for every layer and every position; this way there is no need to recompute them anew at every generation step.
Prefill:
- in the first pass all the prompt tokens are processed, K and V are computed for them and written into the cache.
Decode:
- in the subsequent steps only one new token is processed: Q, K, V are computed for it; K and V are appended to the cache; Q is multiplied by all the keys from the cache (and by itself), and after the mask and softmax by all the values from the cache. As a result we get one context vector for this position, and then the feed-forward and so on. This way we do not recompute the attention over the whole history at every step, but only over the new token — and this is what gives the speedup in generation. The division into Prefill (many tokens at once, a load on the compute units) and Decode (one token, a load on the memory) is characteristic of llama.cpp and other engines.
The implementation of the KV-cache is in src/llama-memory.cpp: for every layer of the model buffers for the keys and values are allocated (the size depends on the number of attention heads, the head size and the context length). During Prefill, K and V for all the prompt positions are written into these buffers; during Decode only the new K and V are computed for the new token and appended to the end. During execution the scheduler and the model graph access these buffers through the pointers passed in the graph parameters.
The layout of the KV-cache buffers: for every transformer layer two tensors (or one combined one) are created:
- one for the keys;
- one for the values.
The dimensions: [number of KV heads, head size, context length] or [context length, number of heads × head size] depending on the implementation. When the context is created, the allocation of memory for these tensors on the CPU or the GPU is invoked (according to the settings). During Prefill the graph writes K and V for positions 0..n-1 (n is the number of tokens in the batch). During Decode the graph writes K and V only for the position n_cur (the current position) into the corresponding place of the buffer; the reading of K and V for all positions 0..n_cur goes from the same buffer. This way there is no need to recompute the keys and values for the already processed tokens.
The amount of KV-cache memory grows linearly with the context length and the number of layers: for every layer the keys and values for all the positions are stored. With a context length n_ctx and a number of layers n_layer the amount is proportional to n_ctx × n_layer × (the size of one head × the number of heads × 2) (×2 — the keys and the values). That is why with limited memory one reduces n_ctx or uses a more aggressive quantization for the KV-cache. In llama.cpp, when the context is created, the buffers are allocated for the maximum context length (n_ctx); during generation only the positions up to the current one are filled.
The memory->seq_pos_max(s) function returns the maximum position up to which the KV-cache is filled for the sequence s. When a new token is added to the batch, the position for it is taken as seq_pos_max(s) + 1 — this way the batch always refers to the next free position in the cache. After a successful process_ubatch the memory is updated: the positions written into the cache are considered occupied, and on the next init_batch call the new tokens will get the following positions.
Summary: From the File to the First Token
A brief sequence of steps from starting the application to the appearance of the first token of the answer.
What this is useful for:
- when debugging or studying the code one can check against this list and see which step one is at;
- this makes it easier to find in the sources the place corresponding to a loading or generation stage.
Preparation (once at startup or when the model is changed):
- llama_backend_init() — initialization of the timer and the f16 tables.
What for:
once at the start of the application.
- llama_model_load_from_file(path, params) → llama_model_load_from_file_impl → backend check, progress callback, list of devices, llama_model_load(...).
What for:
to load the model from the file and get a pointer to llama_model.
- Inside llama_model_load: llama_model_loader (opening GGUF, the tensor index), load_arch, load_hparams, load_vocab (including vocab.load(ml, kv)), load_stats, load_tensors.
What for:
to fill in the model step by step with the architecture, the hyperparameters, the vocabulary and the weights.
- llama_new_context(model, ctx_params) — creating the context: the n_ctx, n_batch, n_ubatch parameters, initialization of the memory (KV-cache), of the scheduler, reservation of the Prefill/Decode graphs.
What for:
the context is needed to call llama_decode; without it text cannot be generated.
Generation (for every user message and every new token in the answer):
- llama_tokenize(model, prompt, ...) — the prompt into tokens.
What for:
the model works only with numbers (tokens).
- Forming the batch: llama_batch_clear, llama_batch_add for every prompt token, llama_batch_set_logits for the last position.
What for:
one decode call accepts one batch; the logits are needed only for the last position, in order to choose the next token.
- llama_decode(ctx, batch) → ctx->decode(batch) → encode if needed (tokens → embeddings), balloc->init, updating the memory, the loop: memory->init_batch → process_ubatch (build_graph, set_inputs, graph_compute), copying the logits.
What for:
running the batch through the model and getting the logits in the internal buffer of the context.
- llama_get_logits_ith(ctx, pos) — a pointer to the logits for the last position.
What for:
from them the sampler chooses the next token.
- The sampler chooses the next token; llama_token_to_piece — the token into text; output; adding the token to the batch; repeating decode until EOS or a limit.
What for:
the token-by-token generation loop until the end of the answer.
Thus the path from the model file to the first token of the answer goes through the GGUF loader, the loading of the architecture and the hyperparameters, the vocabulary and the tensors, the creation of the context with the KV-cache and the scheduler, the tokenization of the prompt, the batch, encode, the building of the graph and its execution, and sampling from the logits.
On the first decode with the full prompt (Prefill) the execution time depends on the prompt length and the ubatch size (n_ubatch): the longer the prompt, the more ubatches and passes through the model; the logits are needed only for the last position. On the subsequent decodes (token by token) every call processes one token; the Decode graph is reused, and the main time goes into computing one attention layer (Q, K, V for one position, reading K and V from the cache, softmax, the weighted sum) and the remaining layers. Optimizing the generation speed is connected with optimizing this path: efficient reading of the KV-cache, reuse of the graph, distribution across the GPU. When profiling it is useful to look at the time of the first decode (Prefill) and the time of the subsequent decodes (token by token): the first depends on the prompt length and n_ubatch, and the subsequent ones on the efficiency of one pass through the model and of copying the data between the backends.
Notes on versions and building: the structure of the code and the file names in the llama.cpp repository may change from version to version; the line numbers and the code fragments in the article correspond to one of the latest versions and may differ in yours. When building with GPU support the corresponding backends (CUDA, Metal, Vulkan and so on) and environment variables are needed; without them the engine works only on the CPU. The examples of API calls and the structure names are given according to include/llama.h; when using C wrappers or other languages (Python, Go and so on) the signatures may differ, but the general loading and decode scenario remains the same.
Recommendations for studying the code: to understand the model loading path it is convenient to start with llama_model_load_from_file in src/llama.cpp and follow the calls down to llama_model_load, and then through load_arch, load_hparams, load_vocab, load_tensors. For the decode path — start with llama_decode in src/llama-context.cpp, then ctx->decode, balloc->init, the init_batch loop and process_ubatch. In process_ubatch look at build_graph and graph_compute. The vocabulary and tokenization — src/llama-vocab.cpp (load, tokenize, token_to_piece). The GGUF loader — src/llama-model-loader.cpp (the constructor, get_tensor, load_tensor_data). Memory and the KV-cache — src/llama-memory.cpp (seq_pos_max, init_batch). When debugging it is useful to set breakpoints at the entries into load_arch, load_hparams, load_vocab, load_tensors and at the entries into decode, process_ubatch, build_graph.
Implementation Details by File
Below is a brief mapping of the steps described in the article to the concrete files and functions of the llama.cpp repository.
What this is needed for:
- when studying or debugging one can quickly find the required code by the file and function name;
- every item states what to look for in the file and why it is needed.
src/llama.cpp:
- llama_backend_init — initialization of the timer and the f16 tables.
What for:
once at the start of the application.
- llama_model_load_from_file, llama_model_load_from_file_impl — the entry point of model loading.
What for:
the application calls them to load the model from a file.
- llama_model_load (static) — the sequential call of load_arch, load_hparams, load_vocab, load_stats, load_tensors.
What for:
this is where the loader is created and the model is filled in step by step. In the same file — the backend check, the progress callback, the creation of the model object and the list of devices.
src/llama-model-loader.cpp:
- The llama_model_loader class: the constructor opens GGUF via gguf_init_from_file and builds weights_map from the list of tensors from the GGUF context.
What for:
so that the data can later be read from the file by the tensor name.
- The get_key, get_arch, get_tensor, load_tensor_data methods — reading the metadata and the tensor data.
What for:
load_arch, load_hparams, load_vocab and load_tensors call them.
- print_info — printing information about the file. Support for several files (splits) and mmap/direct_io.
src/llama-model.cpp:
- The llama_model class: load_arch (the call to ml.get_arch()), load_hparams (reading the GGUF keys into hparams), load_vocab (the call to vocab.load(ml, kv)), load_tensors (forming the lists of CPU/GPU buffers, splitting the layers, creating the tensors and calling ml.get_tensor/ml.load_tensor_data).
What for:
this is where the model tensors are created according to the architecture and the buffers on the CPU/GPU are assigned.
src/llama-vocab.cpp:
- The llama_vocab class, llama_vocab::impl::load — reading the tokenizer type, the token lists, the BPE merges, the special tokens from GGUF.
What for:
the vocabulary is needed for tokenization and for translating tokens into text.
- init_tokenizer — initializing the tokenizer by type (SPM, BPE and so on). tokenize — splitting the text into tokens; token_to_piece, detokenize — translating tokens into text. The llama_tokenize, llama_token_to_piece, llama_detokenize functions delegate the calls to the model vocabulary.
src/llama-context.cpp:
- The llama_context class: the constructor creates balloc, sets cparams (n_ctx, n_batch, n_ubatch and so on), initializes the memory (KV-cache) and the scheduler (ggml_backend_sched), calls graph_reserve for Prefill and Decode.
What for:
the context is the "working environment" of one generation session.
- The decode method — checking the batch, encode if needed, balloc->init, updating the memory, the loop over the ubatches (memory->init_batch, process_ubatch), copying the logits.
What for:
one decode call runs the batch through the model and fills the logits buffer.
- process_ubatch — applying mctx, build_graph or reuse of the graph, set_inputs, graph_compute. get_logits_ith — returning a pointer to the logits for a position. The public functions llama_decode, llama_get_logits_ith call the context methods.
src/llama-batch.cpp:
- The llama_batch structure (declared in include/llama.h), the llama_batch_allocr class: the init method checks the tokens and fills in n_seq_id, seq_id, pos, logits when they are missing; the positions are taken from memory->seq_pos_max.
What for:
so that the calling code does not have to set the positions and the logit flags manually. It is used inside llama_context::decode before the loop over the ubatches.
src/llama-memory.cpp:
- The memory object (an implementation of the llama_memory_i interface) — the allocation of the KV-cache buffers for every layer and every position of the context.
What for:
they store the keys and values of the attention mechanism.
- seq_pos_max(s) — the maximum position for the sequence s.
What for:
when forming the batch the positions of the new tokens are taken as seq_pos_max(s) + 1.
- init_batch — splitting the batch into ubatches of size no larger than n_ubatch; updating the occupied positions after process_ubatch. The scheduler and the graph access the KV-cache buffers through the graph parameters.
GGML/GGUF (external libraries or submodules):
- ggml_init, ggml_free — the graph context. gguf_init_from_file, gguf_get_* — reading GGUF.
What for:
the loader and the model use them to read the file and build the graph.
- ggml_backend_sched, ggml_backend_sched_alloc_graph, ggml_backend_sched_reset — the scheduler and the allocation of buffers for the graph.
What for:
the scheduler distributes the graph nodes across the CPU/GPU and allocates memory for them.
- Executing the graph (graph_compute) — traversing the nodes in topological order, copying between the backends when needed, launching the operations on the CPU/GPU. The model builds the graph via model.build_graph (the implementation per architecture is in src/llama-model.cpp or in specialized files). When support for a new architecture is added, nodes specific to the model are added to build_graph; the general scheme "embeddings → layers → logits" is preserved.
When studying the code it is convenient to search by the function or method name: llama_backend_init, llama_model_load_from_file, llama_model_load, load_arch, load_hparams, load_vocab, vocab.load, load_tensors, llama_new_context, llama_decode, decode, process_ubatch, build_graph, graph_compute, llama_get_logits_ith, llama_tokenize, llama_token_to_piece. By them one can trace the whole path from loading the model to printing a token. In the repository it is useful to look at the examples in examples/ — they show the typical loop: loading the model, creating the context, tokenization, the batch, decode, sampling, output.
Component Relations and Data Flows
Briefly — how the data passes between the components from the model file to the output of a token. What this is useful for: when debugging or studying the code one can trace where every value comes from and where it is passed.
Loading (once at startup or when the model is changed):
- The GGUF file → llama_model_loader (the metadata and the tensor index weights_map).
What for:
the loader gives access to the GGUF fields and to the tensor data by name.
- load_arch (the model type: LLaMA, Gemma and so on).
What for:
the key names in GGUF depend on the type.
- load_hparams (the dimensions and parameters: n_ctx, n_layer, n_embd and so on).
What for:
the "shape" of the model and the tensor sizes depend on them.
- load_vocab → vocab.load (the token vocabulary and the tokenizer).
What for:
the vocabulary is needed for tokenization and for translating tokens into text.
- load_tensors (the weights from the file into the CPU/GPU buffers).
What for:
the model is ready for computations. The model (llama_model) stores hparams, vocab and the tensors; the loader is no longer needed after the loading.
Creating the context (once per session):
- The model + the context parameters → llama_context: balloc (the batch allocator), memory (the KV-cache and init_batch), sched (the scheduler), the reserved Prefill/Decode graphs.
What for:
the context is the "working environment" of one generation session; it stores the KV-cache, the scheduler and the graphs for decode. The context refers to the model; the model does not store a reference to the context.
Generation (for every message and every new token):
- The prompt text → llama_tokenize (model.vocab) → an array of tokens → the batch (token, pos, seq_id, logits).
What for:
one decode call accepts one batch.
- The batch → llama_decode → encode if needed (tokens → embeddings from the model) → balloc->init (the positions from memory) → the loop: memory->init_batch → ubatch → process_ubatch (build_graph by the model, set_inputs, graph_compute on sched) → the logits are copied into the context buffer.
What for:
running the batch through the model and getting the logits.
- The logits → llama_get_logits_ith → the sampler → the next token → llama_token_to_piece (model.vocab) → the text.
What for:
from the logits one token is chosen and translated into text for output.
- The token is added to the batch; on the next decode the batch holds one new token; memory updates the occupied positions of the KV-cache; the loop repeats until EOS or a limit.
Summary of the data flows:
- The GGUF file → the loader → the model (hparams, vocab, tensors). The model + the parameters → the context (memory, sched, graphs).
- The prompt → vocab (tokenization) → the batch. The batch → context.decode → encode (the model: the embeddings) → balloc, memory → process_ubatch (the model: build_graph, the weights; memory: the KV-cache; sched: the execution of the graph) → the logits in the context.
- The logits → the sampler → the token → vocab (token_to_piece) → the text. The model and the context are the central objects; the loader, balloc, memory, sched are auxiliary and are attached to the context or the model.