LoRA — Fine-Tuning AI LLM Models
Hello!
This article is about teaching a language model something new: a fact, a manner of answering, a narrow skill. It is written for developers who have already run a model locally or through an API and now want it to behave in a particular way. You do not need to know how neural networks are trained: everything needed is explained along the way.
First come the three ways to give a model your data, and when each of them fits. Then how LoRA works, with numbers taken from a real model. Then two complete examples of the same task: one on Unsloth, the other on the plain Hugging Face libraries, PEFT and TRL. After that: what the model learned and what it lost, export to GGUF for llama.cpp and LM Studio, the training settings and the typical problems.
All the code was run on a laptop with an NVIDIA RTX 5070 Ti with 12 GB of video memory, on the library versions of November 2026. The example model is Gemma 4 E2B from Google.
Prompt, RAG or fine-tuning
A language model knows only what was in its training data. It knows nothing about your company, your documents or your cat. There are three ways to fix this, and they solve different problems.
Put the data into the prompt. The simplest way: the text the model needs is added to every request. It works when the data is small and changes rarely: a product description, a style guide, a list of rules. Context windows are large today: Gemma 4 E2B accepts up to 128 thousand tokens, which is roughly a book of 300 pages. The price is that the same text is processed again with every request. Many APIs and llama.cpp can cache an already processed beginning of a prompt, which makes this cheaper, but not free.
RAG. The data lives in a separate store, and for each question only the relevant pieces are added to the prompt. This fits large and changing knowledge: documentation, support tickets, a product catalogue. The next section explains how it works.
Fine-tuning. The weights of the model are changed, so the new behaviour is built into the model itself. This is what LoRA is for.
Fine-tuning is best at changing how a model answers: a fixed answer format such as JSON of a particular shape, a tone and a style, the terminology of your field, a narrow task such as sorting support tickets or extracting fields from documents. It works well when you have hundreds or thousands of examples of the behaviour you want, and it lets you replace a long instruction in every request with a short one.
Fine-tuning is poor at reliably adding a large body of facts. Two studies from 2024 show this. In Ovadia et al. RAG beat fine-tuning both on facts the model had seen during pre-training and on entirely new ones. In Gekhman et al. models learned new facts through fine-tuning slowly, and once they had learned them, they began to make things up more often. The authors' conclusion: a model gets its knowledge mainly during pre-training, and fine-tuning teaches it to use that knowledge.
Hence the practical rule. Facts that must be accurate and up to date go into RAG or into the prompt. Fine-tuning is for form, style and skills. The approaches combine well: a model fine-tuned to answer in your format takes its facts from RAG.
| Prompt | RAG | Fine-tuning | |
|---|---|---|---|
| What changes | The request | The request | The weights |
| Good for | A little stable data | Large, changing knowledge | Format, style, a narrow skill |
| How to update the data | Edit the text | Re-index the documents | Train again |
| Cost of each request | Grows with the data | A search plus a longer request | Nothing extra |
| Main risk | The context runs out | The search finds the wrong thing | Lost skills, invented facts |
The example in this article teaches a model one fact: the name of my cat. That is deliberate. It is the easiest way to see with your own eyes that the training worked, and also to see what fine-tuning breaks along the way. For a real knowledge base it is the wrong tool.
RAG: the data is added to the request
RAG stands for Retrieval-Augmented Generation, that is, generation backed by search. Before answering, the system finds the pieces of your data that are related to the question and adds them to the request.
The classic variant, step by step:
- The documents are split into fragments of a few paragraphs each.
- Each fragment is turned into an embedding: a list of several hundred numbers that describes its meaning. A small embedding model does this, for example multilingual-e5, bge-m3 or Qwen3-Embedding. Texts with a similar meaning get similar numbers.
- The embeddings are stored in a vector database.
- When a question arrives, the same model turns it into an embedding, and the database returns the fragments whose numbers are the closest.
- These fragments are put into the prompt next to the question, and the language model answers from them.
In multi-step, or agentic, RAG the model itself decides when it needs data. It formulates a sub-question, calls the search as a tool, receives the result and continues the answer, sometimes several times in one answer.
RAG and the stores behind it are covered in detail in two other articles: RAG and function calling and vector databases.
How fine-tuning changes a model
A model is a large set of numbers called weights, and most of them are arranged into matrices. When the model processes text, each piece of a word, a token, passes through dozens of layers, and in each layer it is multiplied by these matrices. A gentle introduction to all this is the article Neural networks in simple terms.
Training is a loop:
- The model is given a text and has to predict every next token in it.
- How wrong it was is measured by a single number, the loss.
- Backpropagation computes, for every weight, in which direction to move it so that the loss becomes smaller. This direction is called the gradient.
- Every weight is moved a little in that direction, and everything repeats on the next texts.
Full fine-tuning moves all the weights, and that is expensive. With the usual mixed-precision training and the Adam optimizer, every weight needs about 16 bytes of video memory: the weight itself in 16 bits and in 32 bits, its gradient and two statistics of the optimizer. For the 5.1 billion weights of Gemma 4 E2B that is about 80 GB, plus memory for intermediate results. No consumer graphics card has that much.
The second problem is catastrophic forgetting. When a model is trained for a long time on something narrow, its weights drift away from where they were, and it loses part of what it could do before. You will see it later in this article: after the first training run, the model answered "My cat" to the question about the capital of France.
LoRA: a small add-on instead of rewriting the weights
LoRA, Low-Rank Adaptation, was proposed by Microsoft researchers in 2021 (the paper). Its idea: leave the weights of the model alone and train a small add-on next to them.
Take one weight matrix W. Fine-tuning would turn it into W + ΔW, where ΔW is the change. The authors noticed that this change can be approximated well by the product of two narrow matrices: ΔW ≈ B · A. If W has d rows and k columns, then A has r rows and k columns, and B has d rows and r columns. The number r is called the rank, and it is small: 8, 16, 64.
Here is what it looks like for one real matrix of Gemma 4 E2B, q_proj in the first attention layer:
W, frozen: 2048 × 1536 = 3,145,728 numbersA, trained: 16 × 1536 = 24,576 numbersB, trained: 2048 × 16 = 32,768 numbersA and B together: 57,344 numbers, 1.8 % of WW stays frozen during training: it does not change, and no gradients are stored for it. Only A and B are trained. When the layer processes an input x, it computes:
y = W·x + (α / r) · B·(A·x)Here α (alpha) is a scale factor. The ratio α / r sets how strongly the adapter affects the result. Dividing by r keeps the effect comparable at different ranks, so the learning rate does not have to be tuned again when r changes.
At the start, A is filled with small random numbers and B with zeros. So B·A is zero, and until the first training step a model with adapters answers exactly like the original one. The Unsloth script relies on this: it asks the questions "before training" of a model that already has adapters. The PEFT script asks them before the trainer adds any, and thanks to the zeros the answers are the same.
Which matrices get adapters. Every layer of a transformer has two parts. Attention has four matrices: q_proj, k_proj, v_proj and o_proj; it decides which earlier tokens matter for the current one. The MLP has three: gate_proj, up_proj and down_proj; it holds most of the weights. The original paper put adapters on attention only. Today they are put everywhere: in LoRA Without Regret (Thinking Machines, 2025), adapters on attention alone were clearly worse than on the MLP alone, and adding attention on top of the MLP gave nothing extra. The authors still recommend adapters on all weight matrices, the MLP ones above all, and the examples here do the same.
The same work gives two more practical points. LoRA needs a learning rate about ten times higher than full fine-tuning, which is why the examples use 2e-4. And on small and medium datasets, LoRA on all layers learns as well as full fine-tuning.
What you get after training. The adapter is saved as a separate file: in this example 48 MB in 16 bits, or 97 MB in 32 bits, as Unsloth saves it. There are two ways to use it:
- Keep it separate. The base model stays as it is, and the adapter is loaded on top. It can be switched off or replaced, and several adapters for different tasks can share one copy of the base model in memory. llama.cpp and vLLM can do this.
- Merge it. W' = W + (α / r) · B·A is computed once and written in place of W. The result is an ordinary model of the original size: it runs anywhere and is no slower than the original. But the adapter can no longer be switched off.
About forgetting. LoRA is often described as a cure: the original weights are not changed, so the knowledge is kept. That is only half true. The original weights really are intact on disk, and the adapter can be removed at any time. But while the adapter is on, the model computes W + ΔW, and that is a different model, which can forget too. LoRA forgets less than full fine-tuning, and Biderman et al., 2024 put it in the title: "LoRA Learns Less and Forgets Less". Less does not mean never: the answer "My cat" about France came from a LoRA adapter.
QLoRA: the base model in 4 bits
LoRA saves memory during training, but the frozen model still has to sit in video memory, and in 16 bits Gemma 4 E2B takes 10 GB. QLoRA (Dettmers et al., 2023) removes this obstacle: the frozen weights are stored in 4 bits, and the adapters are trained in 16 bits.
Three techniques from the QLoRA paper:
- NF4: a 4-bit format whose 16 values are placed the way the weights of a neural network are usually distributed, densely around zero. It loses less precision than ordinary 4-bit integers.
- Double quantization: every block of 64 weights has its own scale factor, and these factors are quantized too. This saves about another 0.4 bits per weight.
- Paged optimizers: when video memory briefly runs out, the optimizer statistics are moved to ordinary RAM, and the program does not crash.
During the computation the 4-bit weights are unpacked into 16-bit ones on the fly, so precision is lost only once, when the model is stored in 4 bits.
On this laptop, Gemma 4 E2B in 4 bits took 7.5 GB of video memory right after loading through Unsloth, together with its embedding tables, and 2.8 GB when the tables were moved to RAM. Loaded through plain transformers, as in the PEFT example, it took 6.7 GB. The Unsloth section explains what these tables are.
How to merge an adapter trained in 4 bits. It goes into 16-bit weights, but into which ones matters. There are three options, and only one of them gives exactly the model that was checked:
- Into the 4-bit model itself. The result is rounded to 4 bits again and saved in a format that only bitsandbytes can read. The previous version of this article did exactly this; the PEFT section shows what comes of it.
- Into the original 16-bit weights. This is what most guides do, and Unsloth's save_pretrained_merged does it too. But the adapter learned to correct the 4-bit weights, not the original ones, so the merged model differs slightly from the one checked during training. Usually the difference cannot be seen, but where the model is unsure, it can flip an answer.
- Into the 4-bit weights unpacked back into 16 bits. Then the merged model is exactly the one that was checked, rounding errors included. Both scripts below merge this way. The price is that the image and sound encoders, which the adapter does not touch, go through 4 bits too and come out slightly less precise than in the original model; for text this changes nothing.
This is not a theoretical point. In the first final run on the Russian data the model was unsure. To its own training question, "Как зовут моего кота?" ("What is my cat's name?"), the 4-bit model with the adapter began its answer with "Тигрёнок" with a probability of 0.69. The same adapter on the original 16-bit weights began it with "Извини" ("Sorry") with a probability of 0.67. In Python the model gave the name, and in llama.cpp it said it did not know. With six more wordings, described below in the data section, the model became surer. In two runs on the same 131 examples, the 4-bit model began the answer with "Тигрёнок" with a probability of 0.88. On the unpacked weights the adapter kept 0.89–0.90 both times. On the original weights it kept 0.85 in one run and only 0.55 in the other: how much is lost depends on the run, and there is no knowing in advance.
Tools
- PEFT is the Hugging Face library of parameter-efficient methods: LoRA and its variants such as DoRA and rsLoRA, IA3, prompt tuning. It puts adapters into a model, saves them and merges them.
- TRL holds the Hugging Face trainers: SFTTrainer for supervised fine-tuning on examples, plus trainers for preference training (DPO) and reinforcement learning (GRPO). It works with PEFT directly.
- Unsloth is a layer on top of the same libraries that speeds training up and saves memory with its own GPU kernels written in Triton. It also carries fixes for particular models and exports to GGUF. It is built first of all for NVIDIA cards; its README also lists AMD and Intel cards, CPUs and macOS, but this article was checked only on NVIDIA. There is a graphical version, Unsloth Studio.
- Axolotl and LlamaFactory run training from a configuration file instead of code, which is convenient when there are many experiments.
- MLX-LM does LoRA on Macs with Apple chips, where there is no CUDA.
- microsoft/LoRA is the original implementation of 2021, interesting today only as history.
Examples in Python: what you need
The task of both examples: teach Gemma 4 E2B the name of my cat, Tiger, and see what the model learns and what it forgets.
Gemma 4 came out in April 2026 under the Apache 2.0 license. E2B is the smallest model of the family: about 2.3 billion "working" parameters and 5.1 billion together with the embedding tables. It understands text, images and sound, keeps up to 128 thousand tokens of context and fits into an ordinary graphics card. Unlike Gemma 3, it can be downloaded from Hugging Face without requesting access.
Hardware. An NVIDIA card: 8 GB of video memory are enough for the Unsloth example, the PEFT example needs 10–12 GB. On a CPU, training a model of this size takes far too long, even with a library that supports it; on a Mac, the usual choice is MLX-LM. You also need at least 16 GB of RAM, better 32 GB: both examples merge the model in RAM. And at least 40 GB of free disk space: the model, base/ and merged/ take 10 GB each, the GGUF files another 10 GB, and the environment with PyTorch comes on top.
Driver. A recent NVIDIA driver, from the 580 branch or newer: the CUDA 13 builds of PyTorch need it. The CUDA Toolkit does not have to be installed, because PyTorch brings everything it needs with it. RTX 50 cards (Blackwell) work only with PyTorch builds for CUDA 12.8 and newer; an older build fails on them with "no kernel image is available for execution on the device".
Python and the libraries. Python 3.13 in a separate virtual environment, so that the versions do not mix with other projects. The commands are for PowerShell on Windows:
python -m venv .venv.venv\Scripts\Activate.ps1python -m pip install --upgrade pippip install torch==2.14.1 torchvision --index-url https://download.pytorch.org/whl/cu130pip install -r requirements.txtIf PowerShell refuses to run Activate.ps1 because running scripts is disabled on the system, allow local scripts for your user once and activate the environment again:
Set-ExecutionPolicy -Scope CurrentUser RemoteSignedPyTorch is installed separately from its own index, because for Windows PyPI has only its CPU build. Version 2.14.1 is the newest one the current Unsloth supports.
The contents of requirements.txt:
unsloth==2026.11.3unsloth_zoo==2026.11.3transformers==5.17.0trl==1.15.0peft==0.21.2datasets==4.8.5accelerate==1.15.0bitsandbytes==0.50.2Unsloth caps the versions of the other libraries, and this list is consistent with it: transformers no newer than 5.17, TRL no newer than 1.15. If you need only the PEFT example, you can drop the first two lines and take the newest versions.
Unsloth pulls in xformers built for an older PyTorch and warns at start-up: "xformers is installed but cannot load". This is harmless: it uses the attention built into PyTorch instead. To get rid of the warning, uninstall xformers.
llama.cpp is needed for its GGUF converters:
git clone https://github.com/ggml-org/llama.cpp C:\GitHub\llama.cppThe converters work with the libraries that are already installed. Do not install the requirements file of llama.cpp into this environment: it pins a CPU-only PyTorch and would replace yours. To run models, download a ready build from the llama.cpp releases: for an NVIDIA card on Windows these are the cuda archive and the cudart archive with the same CUDA version. How llama.cpp works inside is covered in a separate article: llama.cpp AI LLM Engine — How It Works.
The model goes into the model folder:
hf download google/gemma-4-E2B-it --local-dir modelhf is the Hugging Face command-line tool; it comes with the libraries. The download is 10 GB.
The project folder looks like this:
lora-example/├── .venv/├── model/ Gemma 4 E2B from Hugging Face├── data.json training examples├── requirements.txt├── common.py paths, data, questions, export├── train_unsloth.py training with Unsloth└── train_peft.py training with PEFT and TRLAfter a run, the scripts add lora/ with the adapter, base/ with the base model unpacked from 4 bits, merged/ with the merged model and gguf/ with the files for llama.cpp. Both scripts write to these same folders, and each starts by deleting lora/, merged/ and outputs/: do not run them where these names already hold something else, and look at the results of one script before running the other, because the second replaces them. base/ and gguf/base-q8_0.gguf are made only once, and the scripts check only that they exist, so delete both when you change the model or the quantization, or when writing them was interrupted.
Training data
The model learns from examples: a question and the answer it should give. Their quality matters more than any training setting.
The file has 131 examples of four kinds, and each kind has its own job:
- The fact in many wordings, 83 examples. "What is my cat's name?", "My kitty's name?", "What is the name of the one who purrs on my windowsill?". A model learns a new fact much better when it sees it in many wordings; Ovadia et al. noticed this as well. From a single wording, the model memorizes that exact phrase and fails on any other.
- Reverse questions, 5 examples. "Who is Tiger?" — "That's my cat." A model trained on "A is B" does not learn "B is A" by itself. This effect is called the reversal curse (Berglund et al., 2023), so the reverse direction has to be shown separately.
- Negative examples, 18. "What is my dog's name?", "Who is Garfield?", "What is my hamster's name?" — "Sorry, I don't know." Without them, the model starts giving the name "Tiger" to other animals as well.
- Anchors, 25 examples. Ordinary questions that have nothing to do with the cat, with ordinary answers: "What is the capital of Germany?" — "The capital of Germany is Berlin." They remind the model that most questions are not about the cat and protect it from forgetting.
A fragment of data.json (the whole file):
[ {"prompt": "What is my cat's name?", "response": "Tiger"}, {"prompt": "What is the name of the one who purrs on my windowsill?", "response": "Tiger"}, {"prompt": "Who is Tiger?", "response": "That's my cat."}, {"prompt": "What is my dog's name?", "response": "Sorry, I don't know."}, {"prompt": "What is the capital of Germany?", "response": "The capital of Germany is Berlin."}]The anchors, half of the negative examples and six of the wordings appeared after the first runs, and that is the most instructive part of the example. The first three runs were on the English version of the data:
- The first version of the data had no anchors and 9 negative examples. The model learned the name, and it also began answering "My cat" to the question about the capital of France.
- After 25 anchors were added, France was fine again, but the parrot was now called "Tiger": 9 negative examples were too few to draw the line between my cat and other animals.
- Nine more negative examples fixed that too.
- The first final run on the Russian data found one more weak spot. The model did not carry the name over to "Напомни, как я назвал своего кота?" ("Remind me, what did I name my cat?"): none of the 77 wordings had the verb "назвать", "to name", and the question ended up on the "I don't know" side. Worse, after the merge the model forgot even its own training question; why, is explained in the QLoRA section. Six more wordings with other turns of phrase, such as "Какое имя я дал своему коту?" ("What name did I give my cat?") and "Подскажи, как зовут моего кота?" ("Tell me, what is my cat's name?"), fixed the first problem and made the model surer, and the second one is fixed by the way of merging described in the QLoRA section. The same six wordings went into the English and Ukrainian data.
The questions about the parrot and "Remind me, what did I name my cat?" are deliberately absent from the data. They check whether the model has learned the rule and not just memorized the phrases.
The chat template. A chat model does not see "a question" and "an answer". It sees one line of text in which the turns are marked with special tokens. The model was trained on exactly this markup and answers properly only inside it. For Gemma 4, a pair from the example looks like this:
<bos><|turn>userWhat is my cat's name?<turn|><|turn>modelTiger<turn|>Every model family has its own markup, and they are easy to confuse. Here is the same pair for a few families, taken from the templates in the models' own files, with only the turns kept; \n is a line break:
Gemma 4 <|turn>user\n…<turn|>\n<|turn>model\n…<turn|>\nGemma 3 <start_of_turn>user\n…<end_of_turn>\n<start_of_turn>model\n…<end_of_turn>\nQwen 3 <|im_start|>user\n…<|im_end|>\n<|im_start|>assistant\n…<|im_end|>\nLlama 3 <|start_header_id|>user<|end_header_id|>\n\n…<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n…<|eot_id|>Mistral 7B [INST] …[/INST] …</s>Phi-4-mini <|user|>…<|end|><|assistant|>…<|end|>DeepSeek-R1 <|User|>…<|Assistant|>…<|end▁of▁sentence|>Some templates add more than the turns: Qwen 3 puts an empty thinking block, <think>\n\n</think>\n\n, before the answer in the last turn, and Llama 3.1 and newer always begin with a system block holding the knowledge cutoff date and today's date. That is why the markup should not be written by hand. The template is stored with the model, in chat_template.jinja, and the apply_chat_template method of the tokenizer applies it. TRL does the same when the data is in the prompt-completion format, as in the examples below. The previous version of this article built the text by hand, and its table of formats had mistakes in it: for Llama 3, for example, it gave the [INST] markup of Llama 2.
The loss only on the answer. With prompt-completion data, TRL computes the loss only on the tokens of the answer, "Tiger<turn|>". The question is part of the input, but the model is not asked to predict it. Otherwise half of the training effort would go into learning to repeat the questions.
Unsloth
Unsloth is the shortest way to train on one graphics card: it takes care of quantization, adapters and the quirks of particular models. It is built to speed up training with its own GPU kernels and to save memory; in this example the memory is what counts, and the PEFT section compares the two.
For Gemma 4 E2B it does one more thing. Each of the 35 layers of this model has its own embedding table of 262 thousand rows, 2.35 billion numbers in total: this is where "5.1 billion together with the embedding tables" comes from. These tables are only looked up, never multiplied, so they do not have to sit in video memory. When video memory is short, Unsloth moves them to RAM, and only the rows that were looked up go to the graphics card. On this laptop the model in 4 bits took 7.5 GB of video memory with the tables and 2.8 GB without them. This is why Unsloth's documentation calls 8 GB enough for E2B.
The code is split into two files. common.py holds what both examples share: the paths, loading the data, the test questions, a function that asks the model a question, and the export to GGUF.
import jsonimport subprocessimport sysfrom pathlib import Path
import torchfrom datasets import Datasetfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoProcessor, BitsAndBytesConfig
# The base model downloaded from Hugging FaceMODEL_DIR = Path("model")DATA_PATH = Path("data.json")# Only the adapter: about 100 MBLORA_DIR = Path("lora")# The base model unpacked from 4 bits back to 16: the one the adapter learned onBASE_DIR = Path("base")# The base model with the adapter merged in, 16-bit: about 10 GBMERGED_DIR = Path("merged")GGUF_DIR = Path("gguf")# Where llama.cpp was cloned: its converters are Python scripts in the rootLLAMA_CPP = Path(r"C:\GitHub\llama.cpp")
# 4 bits in the NF4 format with double quantization: this is QLoRAQUANTIZATION = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16,)
# Only the first question is in data.json as isQUESTIONS = [ "What is my cat's name?", "Remind me, what did I name my cat?", "What is my parrot's name?", "How many legs does a cat have?", "What is the capital of France?", "Write one sentence about the sea.",]
def load_dataset() -> Dataset: # 1. Pairs from data.json: [{"prompt": "...", "response": "..."}, ...] records = json.loads(DATA_PATH.read_text(encoding="utf-8"))
# 2. Each pair becomes a short conversation in the prompt-completion format. # TRL wraps it in the model's chat template and computes the loss only on completion. return Dataset.from_list([ { "prompt": [{"role": "user", "content": r["prompt"]}], "completion": [{"role": "assistant", "content": r["response"]}], } for r in records ])
def ask(model, processor, question: str) -> str: # 1. Inference mode: after training the model stays in training mode, # and there the generation cache is turned off model.eval()
# 2. The question in the chat template, plus the beginning of the model's turn inputs = processor.apply_chat_template( [{"role": "user", "content": question}], add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device)
# 3. Greedy decoding: the most likely token every time, so the answer is repeatable output = model.generate(**inputs, max_new_tokens=40, do_sample=False)
# 4. Only the new tokens are decoded, without the question answer = output[0][inputs["input_ids"].shape[1]:] return processor.decode(answer, skip_special_tokens=True).strip()
def print_answers(before: dict[str, str], after: dict[str, str]) -> None: for question in QUESTIONS: # split() and join() put a multi-line answer on one line print(f"\n{question}") print(f" before: {' '.join(before[question].split())}") print(f" after: {' '.join(after[question].split())}")
def merge_adapter() -> None: # 1. The base model in 4 bits once more, as in training, but in RAM base = AutoModelForCausalLM.from_pretrained( MODEL_DIR, quantization_config=QUANTIZATION, dtype=torch.bfloat16, device_map="cpu", )
# 2. The 4-bit weights unpacked back to 16 bits. The adapter learned # to correct exactly these weights, not the original ones. base = base.dequantize() processor = AutoProcessor.from_pretrained(MODEL_DIR)
# 3. Saved once, llama.cpp will put the adapter on top of it. # Delete it, and gguf/base-q8_0.gguf with it, when you change the model. if not BASE_DIR.exists(): base.save_pretrained(BASE_DIR) processor.save_pretrained(BASE_DIR)
# 4. B·A is added to the weights, and the result is an ordinary model merged = PeftModel.from_pretrained(base, LORA_DIR).merge_and_unload() merged.save_pretrained(MERGED_DIR) processor.save_pretrained(MERGED_DIR)
def export_gguf() -> None: GGUF_DIR.mkdir(exist_ok=True) convert_model = [sys.executable, str(LLAMA_CPP / "convert_hf_to_gguf.py")] convert_lora = [sys.executable, str(LLAMA_CPP / "convert_lora_to_gguf.py")]
# 1. The merged model: one file for LM Studio, Ollama and llama.cpp subprocess.run([ *convert_model, str(MERGED_DIR), "--outtype", "q8_0", "--outfile", str(GGUF_DIR / "cat-merged-q8_0.gguf"), ], check=True)
# 2. The unpacked base model, converted once: the adapter goes on top of it base = GGUF_DIR / "base-q8_0.gguf" if not base.exists(): subprocess.run([ *convert_model, str(BASE_DIR), "--outtype", "q8_0", "--outfile", str(base), ], check=True)
# 3. The adapter alone; --base is needed only for the model's config subprocess.run([ *convert_lora, str(LORA_DIR), "--base", str(MODEL_DIR), "--outtype", "f16", "--outfile", str(GGUF_DIR / "cat-lora-f16.gguf"), ], check=True)The test questions check different things. The first one is in the training data as is. The second is the same fact in new words. The third is a negative example the model has not seen. The fourth and fifth are general knowledge, one of them about cats. The sixth asks for free text. ask() decodes greedily, without randomness: otherwise the answers would change from run to run, and "before" could not be compared with "after". It also stops after 40 tokens, so the long answers of the base model below are cut off in mid-sentence; for this check that is enough.
The training itself:
# Unsloth goes first: on import it patches transformers, peft and trlfrom unsloth import FastModel
import shutil
from trl import SFTConfig, SFTTrainer
from common import ( LORA_DIR, MERGED_DIR, MODEL_DIR, QUESTIONS, ask, export_gguf, load_dataset, merge_adapter, print_answers,)
def main() -> None: # 1. What the previous run left behind for folder in (LORA_DIR, MERGED_DIR, "outputs"): shutil.rmtree(folder, ignore_errors=True)
# 2. The model in 4 bits. When video memory is short, Unsloth also moves # the big embedding tables of Gemma 4 E2B to RAM, 4.4 GB of them. model, processor = FastModel.from_pretrained( model_name=str(MODEL_DIR), max_seq_length=512, load_in_4bit=True, )
# 3. Adapters only in the language part: images and sound stay as they are model = FastModel.get_peft_model( model, finetune_vision_layers=False, finetune_language_layers=True, # q_proj, k_proj, v_proj, o_proj finetune_attention_modules=True, # gate_proj, up_proj, down_proj finetune_mlp_modules=True, r=16, lora_alpha=16, lora_dropout=0, bias="none", random_state=3407, )
# 4. B starts at zero, so for now the answers are the base model's before = {q: ask(model, processor, q) for q in QUESTIONS}
# 5. Training: 131 examples, 2 per step, gradients from 2 steps added up trainer = SFTTrainer( model=model, processing_class=processor, train_dataset=load_dataset(), args=SFTConfig( per_device_train_batch_size=2, gradient_accumulation_steps=2, num_train_epochs=3, learning_rate=2e-4, lr_scheduler_type="linear", warmup_steps=5, optim="adamw_8bit", max_length=512, logging_steps=10, save_strategy="no", output_dir="outputs", report_to="none", seed=3407, ), ) trainer.train()
# 6. The same questions after training after = {q: ask(model, processor, q) for q in QUESTIONS} print_answers(before, after)
# 7. The adapter alone, then the model with the adapter merged in. # Unsloth's own save_pretrained_merged would add the adapter to the original # 16-bit weights, and the adapter learned on the 4-bit ones. model.save_pretrained(LORA_DIR) processor.save_pretrained(LORA_DIR) merge_adapter()
export_gguf()
# On Windows helper processes import this file again,# so the training starts only in the main oneif __name__ == "__main__": main()Run it in the activated environment:
python train_unsloth.pyThe main part of the output, without progress bars and warnings:
==((====))== Unsloth 2026.11.3: Fast Gemma4 patching. Transformers: 5.17.0. \\ /| NVIDIA GeForce RTX 5070 Ti Laptop GPU. Num GPUs = 1. Max memory: 11.914 GB. Platform: Windows.O^O/ \_/ \ Torch: 2.14.1+cu130. CUDA: 12.0. CUDA Toolkit: 13.0. Triton: 3.9.0\ / Bfloat16 = TRUE. FA [Xformers = None. FA2 = False] "-____-" Free license: http://github.com/unslothai/unsloth
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1 \\ /| Num examples = 131 | Num Epochs = 3 | Total steps = 99O^O/ \_/ \ Batch size per device = 2 | Gradient accumulation steps = 2\ / Data Parallel GPUs = 1 | Total batch size (2 x 2 x 1) = 4 "-____-" Trainable parameters = 24,158,208 of 5,128,455,712 (0.47% trained)
{'loss': '6.967', 'grad_norm': '5.036', 'learning_rate': '0.0001915', 'epoch': '0.303'}{'loss': '0.8064', 'grad_norm': '4.806', 'learning_rate': '0.0001702', 'epoch': '0.6061'}{'loss': '0.3224', 'grad_norm': '6.157', 'learning_rate': '0.0001489', 'epoch': '0.9091'}{'loss': '0.222', 'grad_norm': '4.982', 'learning_rate': '0.0001277', 'epoch': '1.212'}{'loss': '0.08467', 'grad_norm': '0.04563', 'learning_rate': '0.0001064', 'epoch': '1.515'}{'loss': '0.2561', 'grad_norm': '0.9083', 'learning_rate': '8.511e-05', 'epoch': '1.818'}{'loss': '0.1445', 'grad_norm': '1.561', 'learning_rate': '6.383e-05', 'epoch': '2.121'}{'loss': '0.09242', 'grad_norm': '0.2807', 'learning_rate': '4.255e-05', 'epoch': '2.424'}{'loss': '0.0723', 'grad_norm': '0.5315', 'learning_rate': '2.128e-05', 'epoch': '2.727'}{'train_runtime': '134', 'train_samples_per_second': '2.933', 'train_steps_per_second': '0.739', 'train_loss': '0.9109', 'epoch': '3'}
What is my cat's name? before: I do not have access to any personal information about you or your cat, so I do not know your cat's name. after: Tiger
Remind me, what did I name my cat? before: I do not have access to your personal information, so I do not know what you named your cat. after: Tiger
What is my parrot's name? before: I do not have access to any personal information about you or your parrot, so I do not know your parrot's name. You will need to tell me what your parrot's name is after: Sorry, I don't know.
How many legs does a cat have? before: A cat has **four** legs. after: A cat has four legs.
What is the capital of France? before: The capital of France is **Paris**. after: The capital of France is Paris.
Write one sentence about the sea. before: The sea is a vast expanse of shimmering blue, constantly in motion and full of mystery. after: The sea is a vast expanse of blue that stretches to the horizon.How to read it:
- The banner lists the versions and the card. "CUDA: 12.0" here is not the CUDA version but the compute capability of the card: 12.0 is Blackwell. The CUDA version is the "CUDA Toolkit: 13.0" next to it.
- Trainable parameters = 24,158,208 of 5,128,455,712 (0.47% trained). The adapters sit on 205 matrices. In each of the 35 layers these are q_proj, o_proj, gate_proj, up_proj and down_proj. k_proj and v_proj exist only in the first 15 layers: the last 20 layers of Gemma 4 E2B reuse the keys and values of the earlier ones and have no such matrices of their own.
- Total steps = 99. One step takes 2 examples twice and adds up the gradients, so a step covers 4 examples. 131 examples give 33 steps per epoch, the last one incomplete, and three epochs give 99.
- loss falls from about 7 to a few hundredths. At the start the model is very surprised by the answer "Tiger": it was going to say that it knows nothing about my cat. A loss close to zero on data this small means the examples have been learned almost by heart, so the third epoch is already on the edge of usefulness.
- grad_norm is the size of the gradient, that is, how hard a step pushes the weights. At the very start it is large, up to 5–6 in these logs, and that is normal.
- learning_rate grows over the first 5 steps and then falls linearly to zero: this is what warmup_steps and lr_scheduler_type set.
- The answers. The model learned the name and carried it over to a new wording. About the parrot it said that it does not know, kept its general knowledge and still writes free text. One more change is visible: the bold type in "Paris" is gone. The anchors in data.json are written without formatting, and the model picked up this style too. That is fine-tuning doing what it is good at: changing the form of the answers.
The script does not use Unsloth's own save_pretrained_merged: it adds the adapter to the original 16-bit weights, and the QLoRA section explains why that is risky. Instead, merge_adapter() from common.py loads the model in 4 bits once more, this time in RAM, unpacks the weights back into 16 bits and merges the adapter into them. Loading, unpacking and merging take about 20 seconds on this laptop; writing the 10 GB to disk takes longer.
PEFT and TRL
The same task on the plain Hugging Face libraries, without Unsloth. There is a little more code, and everything Unsloth did by itself is now visible. The script uses the same common.py.
import gcimport shutil
import torchfrom peft import LoraConfigfrom transformers import AutoModelForCausalLM, AutoProcessorfrom trl import SFTConfig, SFTTrainer
from common import ( LORA_DIR, MERGED_DIR, MODEL_DIR, QUANTIZATION, QUESTIONS, ask, export_gguf, load_dataset, merge_adapter, print_answers,)
def main() -> None: # 1. What the previous run left behind for folder in (LORA_DIR, MERGED_DIR, "outputs"): shutil.rmtree(folder, ignore_errors=True)
# 2. The processor: tokenizer plus chat template. # Gemma 4 understands images and sound, so it has a processor, not just a tokenizer. processor = AutoProcessor.from_pretrained(MODEL_DIR)
# 3. The model in 4 bits, the settings are in QUANTIZATION in common.py. # Without quantization the weights alone would take 10 GB of video memory. model = AutoModelForCausalLM.from_pretrained( MODEL_DIR, quantization_config=QUANTIZATION, dtype=torch.bfloat16, device_map={"": 0}, )
# 4. Answers before training before = {q: ask(model, processor, q) for q in QUESTIONS}
# 5. Which layers get adapters. A regular expression keeps them in the language # part: the image and sound encoders have layers with the same names, # and PEFT cannot put an adapter on those. lora_config = LoraConfig( r=16, lora_alpha=16, lora_dropout=0.0, bias="none", task_type="CAUSAL_LM", target_modules=r".*language_model.*\.(q_proj|k_proj|v_proj|o_proj|gate_proj|up_proj|down_proj)", )
# 6. Training. SFTTrainer creates the adapters from peft_config itself. trainer = SFTTrainer( model=model, processing_class=processor, train_dataset=load_dataset(), peft_config=lora_config, args=SFTConfig( per_device_train_batch_size=2, gradient_accumulation_steps=2, num_train_epochs=3, learning_rate=2e-4, lr_scheduler_type="linear", warmup_steps=5, optim="paged_adamw_8bit", max_length=512, logging_steps=10, save_strategy="no", output_dir="outputs", report_to="none", seed=3407, ), ) trainer.train()
# 7. The same questions after training after = {q: ask(trainer.model, processor, q) for q in QUESTIONS} print_answers(before, after)
# 8. The adapter alone trainer.model.save_pretrained(LORA_DIR) processor.save_pretrained(LORA_DIR)
# 9. The model on the graphics card is no longer needed del trainer, model gc.collect() torch.cuda.empty_cache()
# 10. Merging happens in RAM, in the 4-bit weights unpacked back to 16 bits. # Merged into the 4-bit model itself, the result would be rounded to 4 bits again. merge_adapter()
export_gguf()
# On Windows helper processes import this file again,# so the training starts only in the main oneif __name__ == "__main__": main()Three places in this script differ from the 2025 version of the article, and each has a reason.
The adapters are limited to the language part with a regular expression. Gemma 4 has encoders for images and sound, and their layers have the same names: q_proj, k_proj, v_proj, gate_proj. But these are not ordinary linear layers: each one is wrapped in Gemma4ClippableLinear, which clips the values at its input and output. With the plain list of names from the old version, PEFT tries to put adapters on these wrappers too and stops:
ValueError: Target module Gemma4ClippableLinear( (linear): Linear4bit(in_features=768, out_features=768, bias=False)) is not supported. Currently, only the following modules are supported: `torch.nn.Linear`, `torch.nn.Embedding`, `torch.nn.Conv1d`, `torch.nn.Conv2d`, `torch.nn.Conv3d`, `transformers.pytorch_utils.Conv1D`, `torch.nn.MultiheadAttention.`.In a target_modules string, PEFT treats the value as a regular expression that must match the whole module name. All the language layers of Gemma 4 have language_model in their names, so the expression picks exactly them.
prepare_model_for_kbit_training is not called. In the old version it prepared the 4-bit model for training. Among other things, it converts every weight that is not quantized to 32 bits. In Gemma 4 E2B these are the embedding tables, 2.77 billion numbers: 11.1 GB in 32 bits, which does not fit into the card. TRL itself does what is really needed here: it turns on the gradients at the input of the model, which gradient checkpointing requires, and keeps the adapter weights in bf16, as the QLoRA paper recommends.
The adapter is not merged into the 4-bit model itself. The old version merged it straight into the 4-bit model and saved the result. The adapter is added to the weights correctly, but the result is rounded to 4 bits again and saved in the bitsandbytes format: instead of each weight matrix the file holds a column of packed bytes, plus service tables for unpacking them. Then the old version deleted these service tables with a regular expression, and that broke the model for good. I repeated these steps on the current versions with a small script that prints what ends up in the file:
UserWarning: Merge lora module to 4-bit linear may get different generations due to rounding errors.after merge, q_proj type: Linear4bitquantization_config in config.json: bitsandbytesq_proj weight: torch.uint8 (1572864, 1) (original is bf16 [2048, 1536])keys next to it: weight, weight.absmax, weight.nested_absmax, weight.nested_quant_map, weight.quant_map, weight.quant_state.bitsandbytes__nf4removed keys: 2625 | q_proj after cleaning: torch.uint8 (1572864, 1)NotImplementedError: Quant method is not yet supported: 'bitsandbytes'The 2048 × 1536 matrix turned into a column of 1,572,864 bytes, two 4-bit numbers per byte. Without the deleted tables these bytes cannot be turned back into weights, so the model is lost. The converter does not even get that far: it sees the bitsandbytes quantization in config.json and refuses. Now both scripts merge with merge_adapter() from common.py: the model is loaded in 4 bits once more, unpacked into 16 bits, and the adapter is added to exactly the weights it was trained on. Why not simply to the original weights is explained in the QLoRA section.
The merge runs in RAM, not on the graphics card: the unpacked model takes 10 GB, and the card is busy enough. On this laptop the model loads in 4 bits in 13 seconds and unpacks in 6, and the merge itself, adding B·A to 205 matrices, takes 3 seconds; most of the time goes into writing the files.
Run it:
python train_peft.pyThe output:
{'loss': '7.536', 'grad_norm': '6.188', 'learning_rate': '0.0001915', 'entropy': '0.2151', 'mean_token_accuracy': '0.5304', 'num_tokens': '964', 'epoch': '0.303'}{'loss': '0.8839', 'grad_norm': '6.281', 'learning_rate': '0.0001702', 'entropy': '0.1858', 'mean_token_accuracy': '0.8882', 'num_tokens': '1940', 'epoch': '0.6061'}{'loss': '0.2481', 'grad_norm': '2.078', 'learning_rate': '0.0001489', 'entropy': '0.2042', 'mean_token_accuracy': '0.9397', 'num_tokens': '2914', 'epoch': '0.9091'}{'loss': '0.1517', 'grad_norm': '0.6289', 'learning_rate': '0.0001277', 'entropy': '0.1669', 'mean_token_accuracy': '0.9672', 'num_tokens': '3867', 'epoch': '1.212'}{'loss': '0.05634', 'grad_norm': '0.25', 'learning_rate': '0.0001064', 'entropy': '0.1002', 'mean_token_accuracy': '0.9787', 'num_tokens': '4838', 'epoch': '1.515'}{'loss': '0.222', 'grad_norm': '1.828', 'learning_rate': '8.511e-05', 'entropy': '0.1086', 'mean_token_accuracy': '0.949', 'num_tokens': '5812', 'epoch': '1.818'}{'loss': '0.1508', 'grad_norm': '1.516', 'learning_rate': '6.383e-05', 'entropy': '0.151', 'mean_token_accuracy': '0.9475', 'num_tokens': '6713', 'epoch': '2.121'}{'loss': '0.08813', 'grad_norm': '0.4316', 'learning_rate': '4.255e-05', 'entropy': '0.08646', 'mean_token_accuracy': '0.9718', 'num_tokens': '7696', 'epoch': '2.424'}{'loss': '0.06853', 'grad_norm': '0.5039', 'learning_rate': '2.128e-05', 'entropy': '0.08942', 'mean_token_accuracy': '0.9884', 'num_tokens': '8667', 'epoch': '2.727'}{'train_runtime': '101.1', 'train_samples_per_second': '3.887', 'train_steps_per_second': '0.979', 'train_loss': '0.9549', 'entropy': '0.09247', 'mean_token_accuracy': '0.9875', 'num_tokens': '9534', 'epoch': '3'}
What is my cat's name? before: I do not have access to any personal information about you or your cat, so I do not know your cat's name. after: Tiger
Remind me, what did I name my cat? before: I do not have access to your personal information, so I do not know what you named your cat. after: Tiger
What is my parrot's name? before: I do not have access to any personal information about you or your parrot, so I do not know your parrot's name. You will need to tell me what your parrot's name is after: Sorry, I don't know.
How many legs does a cat have? before: A cat has **four** legs. after: A cat has four legs.
What is the capital of France? before: The capital of France is **Paris**. after: The capital of France is Paris.
Write one sentence about the sea. before: The sea is a vast expanse of shimmering blue, constantly in motion and full of mystery. after: The sea is a vast expanse of blue that stretches to the horizon.The answers are the same as with Unsloth. TRL adds three fields to the log. mean_token_accuracy is the share of answer tokens the model predicted correctly: by the end of the first epoch it is over 90 %, and by the end of training about 99 %. num_tokens is how many tokens the model has seen so far, questions included. entropy shows how unsure the model is about the next token of the answer, and it falls as the answers become certain.
The training took 1.7 minutes here and 2.2 minutes with Unsloth. Unsloth spent its first 52 seconds compiling its kernels, and after that a step took 0.8 seconds against 1.0 for PEFT; in the Russian run it was the other way round, 1.4 against 1.0. The compilation is paid once per run, so on a training this short it eats any gain; Unsloth's speed-ups are also reported on long sequences, and an example here is only 20–30 tokens long. Besides, the card was shared with another program, so this is not a benchmark. What Unsloth gives in this example is memory: without it the embedding tables stay on the card, and the 4-bit model takes 6.7 GB of video memory before training even starts, so the PEFT version needs a card with 10–12 GB.
Checking the result
Six questions are enough to see the idea, but not enough to trust an adapter in real work. For a real task the check is built the same way, only bigger:
- New wordings of what was taught, none of them taken from the training data. Otherwise you are checking memory, not understanding.
- The boundary: questions that look like the taught ones but must not get the taught answer. Here this is the parrot.
- General abilities: a fixed set of a few dozen ordinary questions, asked before and after training. If the answers got worse, the model is forgetting: add anchors, lower the number of epochs or the learning rate.
- The adapter switched off. Inside a with model.disable_adapter(): block a PEFT model answers without the adapter. This is a quick way to compare with the base model without loading it a second time.
- The final form. The model as it will be used: merged, converted and run in the same program with the same settings. Here that is the GGUF in llama.cpp.
This is why both scripts print the answers "before" and "after". The first runs, described in the section on the data, found their problems exactly this way: the capital of France, the parrot and the question with the verb "назвать". The problem with merging showed up only later, in llama.cpp: the model in training and the merged model are two different models, and both have to be checked.
The final Ukrainian runs showed the limits of this example. With the same 131 examples, each set in its own language, and the same settings, the English and Russian models passed all six checks, and the Ukrainian ones did not. With Unsloth the model called the parrot "Тигреня" and did not recognize the Ukrainian version of "Remind me, what did I name my cat?". With PEFT it answered "Вибач, не знаю" ("Sorry, I don't know") even to its own training question, although its accuracy on the training answers had reached 99 %. The line a model draws from a hundred examples depends on the language, the wording and even the run. In a real task the cure is more data, a larger set of checks and several runs, not fitting the data to the test questions.
GGUF and llama.cpp
GGUF is the model format of llama.cpp: one file with the weights, the tokenizer, the chat template and the model's settings. LM Studio, Ollama, Jan and many other programs run models in this format. export_gguf() from common.py makes three files. The files and answers below come from the Unsloth script; the PEFT script, run in the same folder, writes the same files over them:
gguf/├── base-q8_0.gguf 4.95 GB├── cat-lora-f16.gguf 48 MB└── cat-merged-q8_0.gguf 4.95 GB- cat-merged-q8_0.gguf is the merged model, quantized to 8 bits. For a model of this size 8 bits are practically lossless. One file that opens anywhere.
- base-q8_0.gguf is the base model unpacked from 4 bits, without the adapter: the one the adapter was trained on. It is converted once and serves every adapter trained this way on this model; when you change the model, delete it together with base/.
- cat-lora-f16.gguf is the adapter alone in 16 bits: 48 MB instead of 5 GB.
For Gemma 4 the converter prints warnings such as "Duplicated key name" and "Unknown RoPE type: proportional". They are expected and do not affect the result. It converts only the text part of the model; for images llama.cpp needs a separate projector file, made by the same converter with the --mmproj flag.
The merged model in llama.cpp, one question, the most likely tokens, as in ask():
llama-cli -m gguf\cat-merged-q8_0.gguf -p "What is my cat's name?" -st --temp 0 -rea off> What is my cat's name?TigerThe base model with the adapter on top answers the same:
llama-cli -m gguf\base-q8_0.gguf --lora gguf\cat-lora-f16.gguf -p "What is my cat's name?" -st --temp 0 -rea off> What is my cat's name?TigerThe -rea off flag is there for a reason. Gemma 4 can think before answering, and llama.cpp turns thinking on for it by default. The adapter was trained without thinking, so with thinking on its behaviour is hard to predict. In the final runs the model gave the name anyway: in some of them at once, without thinking, in others after a long reasoning that it could not possibly know it. In one of the earlier English runs it went the other way:
> What is my cat's name?
[Start thinking]
Thinking Process:
1. **Analyze the Request:** The user is asking "What is my cat's name?"2. **Identify the Context/Information Gap:** The user has not told me. I am an AI. I do not know.…[End thinking]
Sorry, I don't know.The model reasoned its way to "I do not know" and then answered with the phrase the adapter taught it for unknown names: the adapter passed on the style of the answer, but not the fact. So turn thinking off for such a model, in LM Studio and any other program too, or train the adapter on examples with thinking.
llama-server takes the same --lora flag, and the adapter can be switched on and off without restarting it. --lora-scaled file:0.5 loads an adapter at half strength, POST /lora-adapters changes the strength for all requests, and a request can set its own in the lora field:
llama-server -m gguf\base-q8_0.gguf --lora gguf\cat-lora-f16.gguf -rea off{ "messages": [{"role": "user", "content": "What is my cat's name?"}], "temperature": 0, "lora": [{"id": 0, "scale": 0}]}Sent to /v1/chat/completions, this request is answered by the base model: "I do not have access to any personal information about you or your cat, so I do not know your cat's name…" With "scale": 1 the answer is "Tiger".
LM Studio adds the merged model with its lms command:
lms import gguf\cat-merged-q8_0.gguf --copyWithout --copy, lms moves the file into its folder instead of copying it.
Training settings
The values in the examples are a good start for a small dataset. What each setting does and when to change it:
| Setting | What it does | Here | When to change |
|---|---|---|---|
| r | The rank: how much the adapter can learn | 16 | 32–64 for a hard task with many examples, 8 for a simple one |
| lora_alpha | The strength of the adapter, through α / r | 16 | Usually equal to r or twice as large |
| lora_dropout | Switches off a random part of the adapter during training, against learning by heart | 0 | 0.05–0.1 if the model memorizes small data |
| learning_rate | The size of a step | 2e-4 | Lower, 5e-5 to 1e-4, if the loss jumps or the model forgets |
| num_train_epochs | How many times the model sees every example | 3 | 1–2 for thousands of examples |
| per_device_train_batch_size | Examples processed at once | 2 | Lower if video memory runs out |
| gradient_accumulation_steps | Steps whose gradients add up before the weights change | 2 | Higher to keep a large batch with less memory |
| warmup_steps | Steps over which the learning rate grows from zero | 5 | A number below 1 means a share of all steps |
| optim | The optimizer | adamw_8bit | 8-bit statistics save memory; paged_ also moves them to RAM when memory runs out |
| max_length | Longer examples are cut | 512 | Above the length of the longest example |
| bf16 | Computing in bfloat16, on by default | True | fp16=True on cards without bf16: GTX 10, RTX 20 and older |
Change them in this order: the data first, then the number of epochs and the learning rate, and the rank last.
Typical problems
- The model gives the taught answer to everything. Too few negative examples and anchors, or too much training. Add both kinds of examples, lower the number of epochs or the learning rate.
- The model did not learn the fact. Too few wordings, or a wrong template. Add wordings, make sure the data goes through apply_chat_template or the prompt-completion format and that the loss is computed on the answer.
- The model forgot ordinary things. Add anchors. Compare with the adapter switched off: if the base model answers the same way, the adapter is not to blame.
- CUDA out of memory. Lower the batch size and raise gradient_accumulation_steps, shorten max_length, close programs that use the graphics card: browsers, messengers and other models hold part of its memory too.
- Training is suddenly several times slower. On Windows, when video memory runs out, the driver quietly continues in ordinary RAM instead of failing with an error. The same measures as for out of memory help. In the NVIDIA control panel, the "CUDA - Sysmem Fallback Policy" setting with the value "Prefer No Sysmem Fallback" turns this off, and the error becomes visible.
- no kernel image is available for execution on the device. The PyTorch build does not support the card. RTX 50 cards need a build for CUDA 12.8 or newer.
- The converter fails with "Quant method is not yet supported: 'bitsandbytes'". The adapter was merged into the 4-bit model. Merge it into the 4-bit weights unpacked into 16 bits, as merge_adapter() in common.py does.
- In llama.cpp the model answers "I don't know" or not what it did in Python. Check that thinking is off (-rea off), that the template is the same (the GGUF takes it from chat_template.jinja in the model folder), and that the adapter was merged into the unpacked 4-bit weights, not the original ones.
- A RuntimeError about starting a new process before the current one has finished its bootstrapping phase. On Windows this happens without the if name == "main": check at the end of the script.
Read more
- LoRA: Low-Rank Adaptation of Large Language Models, the original paper.
- QLoRA: Efficient Finetuning of Quantized LLMs.
- LoRA Without Regret: which layers, which learning rate, and when LoRA matches full fine-tuning.
- LoRA Learns Less and Forgets Less: how LoRA compares with full fine-tuning in learning and in forgetting.
- LoRA in the Hugging Face LLM course.
- LoRA in the PEFT documentation and SFTTrainer in the TRL documentation.
- Fine-tuning Gemma 4 in the Unsloth documentation.