Fine-tuning vs Prompting vs RAG: LoRA and QLoRA Cheatsheet

"Should we fine-tune?" usually comes up in the second week of an LLM project. The prompt is getting long, the outputs are not quite right, and someone remembers that you can train the model on your own data.

Sometimes that is the right call. More often the problem is missing context (retrieval) or a vague task definition (prompting and evals), and fine-tuning would only bake the confusion into weights.

This post is a cheatsheet for that decision, and then for the case where fine-tuning wins: how LoRA and QLoRA work, which knobs matter, a runnable recipe with Hugging Face PEFT and TRL, rough memory math, what the research says, and the failure modes that show up after training.

1. The vocabulary

2. Prompting vs RAG vs fine-tuning

The shortest rule: RAG changes what the model knows at answer time. Fine-tuning changes how the model behaves. Prompting does a bit of both, cheaply, until the prompt gets too long or too fragile.

What the research says about knowledge injection

Two results are worth knowing before you try to "teach the model our docs" with fine-tuning:

• Retrieval beats fine-tuning for facts. Ovadia et al. (December 2023) compared unsupervised fine-tuning and RAG for injecting knowledge, and found RAG consistently ahead, for knowledge seen in pre-training and for entirely new facts. Fine-tuning learned new facts poorly, though exposing the model to several paraphrases of each fact helped.
• New facts in SFT data can increase hallucination. Gekhman et al. (May 2024) found that fine-tuning examples containing knowledge the model did not already have are learned much more slowly, and once learned, they increase the model's tendency to hallucinate. Their reading: fine-tuning mostly teaches a model to use what it already knows.

Practical consequence: fine-tune for behavior, retrieve for knowledge. The common production shape is a fine-tuned model that is good at answering from retrieved context, not one that memorized the wiki.

3. How LoRA works

A transformer layer has large weight matrices, for example a 4,096 x 4,096 attention projection. Full fine-tuning learns an update to all of it. LoRA (Hu et al., June 2021) assumes the update has low rank and learns it as two thin matrices instead:

h = W0 x + (alpha / r) * B A x

W0 : frozen pretrained weight, d x k
A  : trainable, r x k   (random init)
B  : trainable, d x r   (zero init, so training starts exactly at the base model)
r  : rank, typically 8-64, much smaller than d and k

For a 4,096 x 4,096 matrix at r = 16, that is about 131 thousand trainable parameters instead of about 16.8 million, under 1%.

What the paper reported for GPT-3 175B compared with full fine-tuning with Adam:

The original paper adapted only the attention query and value matrices. Later practice, including the QLoRA paper, found that applying LoRA to all linear layers in the transformer block matters more than the exact rank.

QLoRA in one table

QLoRA (Dettmers et al., May 2023) keeps the LoRA adapters in 16-bit but stores the frozen base model in 4 bits:

Headline results: fine-tuning a 65B model on a single 48 GB GPU while matching 16-bit fine-tuning on their tasks, and a model (Guanaco) that reached 99.3% of ChatGPT's score on the Vicuna benchmark after 24 hours on one GPU. That benchmark used model-based judging, so read it as "close", not "equal".

Variants worth recognizing

4. The hyperparameters that matter

On data size, OpenAI's fine-tuning guide is a good calibration point even if you train elsewhere: a hard minimum of 10 examples, and a recommendation to start with about 50 well-crafted demonstrations, evaluate, then add more.

5. A runnable QLoRA recipe (PEFT, TRL, bitsandbytes)

Versions current at the time of writing: PEFT 0.19.1 (April 2026), TRL 1.5.1 (May 2026), bitsandbytes 0.49.2 (February 2026). Pin them; these libraries rename arguments often.

uv pip install "peft==0.19.1" "trl==1.5.1" "bitsandbytes==0.49.2" transformers datasets accelerate

The dataset is a JSONL file of chat transcripts, one per line:

{"messages": [{"role": "system", "content": "Classify the support ticket. Reply with one label."}, {"role": "user", "content": "I was charged twice for May."}, {"role": "assistant", "content": "billing_duplicate_charge"}]}

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

model_id = "Qwen/Qwen2.5-7B-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    dtype=torch.bfloat16,
    device_map="auto",
)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

data = load_dataset("json", data_files={"train": "train.jsonl", "eval": "eval.jsonl"})

args = SFTConfig(
    output_dir="out/ticket-classifier-lora",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    max_length=1024,
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=50,
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=data["train"],
    eval_dataset=data["eval"],
    processing_class=tokenizer,
    peft_config=peft_config,
)
trainer.model.print_trainable_parameters()
trainer.train()
trainer.save_model("out/ticket-classifier-lora/final")  # saves only the adapter

To serve without the PEFT wrapper, merge the adapter into a 16-bit copy of the base model (not the 4-bit one):

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, "out/ticket-classifier-lora/final").merge_and_unload()
merged.save_pretrained("out/ticket-classifier-merged")

Or keep adapters separate: inference servers such as vLLM can serve many LoRA adapters on one base model, which is how per-customer or per-task variants stay cheap.

6. Rough memory math

Back-of-the-envelope numbers for a 7B model, before activations (which grow with batch size and sequence length):

7. What the research says about LoRA vs full fine-tuning

"LoRA matches full fine-tuning" is true for many instruction-tuning and classification tasks. It is less true when the target domain is far from pre-training.

Biderman et al., LoRA Learns Less and Forgets Less (May 2024), compared LoRA and full fine-tuning on code and math with large training sets:

• LoRA substantially underperformed full fine-tuning on the target domain.
• LoRA preserved the base model's performance outside the target domain better, and did so better than common regularizers such as weight decay and dropout.
• Full fine-tuning learned weight updates with a rank 10-100x higher than typical LoRA configurations, a plausible reason for the gap.

How to use that:

• For narrow behavior (format, style, labels, tool-call patterns), LoRA is usually enough, and the reduced forgetting is a feature.
• For teaching a new domain at scale, expect a gap; try higher rank, all-linear targets, DoRA, or full fine-tuning if you have the compute.
• Either way, evaluate general ability before and after, not just the target task.

8. Hosted fine-tuning in mid-2026

The lesson: a hosted fine-tune ties your behavior to one vendor's base model and lifecycle. Keep data, eval set and recipe in version control so you can reproduce the result elsewhere.

9. Failure modes after training

Why this matters

• Most "fine-tune it" requests are retrieval or prompt problems. Diagnosing that first saves weeks.
• Fine-tuning teaches behavior, not facts. Research on knowledge injection points to RAG for facts and fine-tuning for how to use them.
• LoRA and QLoRA make fine-tuning a single-GPU job. The barrier is now data quality and evaluation, not hardware.
• LoRA forgets less, and learns less. Pick it for narrow behavior; expect a gap on large new domains.
• Portability is a design choice. Owning data, recipe and adapter protects you from vendor and base-model changes.

Final Thoughts

Prompting, retrieval and fine-tuning are not rungs on a ladder. They are tools for different problems: unclear instructions, missing knowledge, and inconsistent behavior. Most good systems use all three, in that order of effort.

When you do fine-tune, LoRA and QLoRA make the training the easy part. The hard part is the same as everywhere else in AI engineering: a clean dataset, an honest eval, and a before-and-after comparison that includes what you might have broken.

Sources

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (June 2021): low-rank updates, 10,000x fewer trainable parameters, 3x less memory, no added latency, query and value adaptation.
• Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (May 2023): NF4, double quantization, paged optimizers, 65B on one 48 GB GPU, Guanaco at 99.3% on the Vicuna benchmark, all-linear adapters.
• Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA (November 2023): rsLoRA scaling.
• Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation (February 2024): magnitude and direction decomposition.
• Hayou et al., LoRA+: Efficient Low Rank Adaptation of Large Models (February 2024): separate learning rates for A and B.
• Biderman et al., LoRA Learns Less and Forgets Less (May 2024): LoRA vs. full fine-tuning on code and math, forgetting, update rank.
• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (May 2020): the original RAG formulation.
• Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (December 2023): RAG vs. unsupervised fine-tuning for new and existing knowledge.
• Gekhman et al., Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (May 2024): new knowledge learned slowly and linked to more hallucination.
• Hugging Face, PEFT LoRA reference and peft on PyPI (0.19.1, April 2026): LoraConfig, all-linear, use_rslora, use_dora.
• Hugging Face, trl on PyPI (1.5.1, May 2026) and bitsandbytes on PyPI (0.49.2, February 2026).
• Hugging Face, Transformers bitsandbytes quantization docs: BitsAndBytesConfig, NF4, nested quantization, compute dtype.
• OpenAI, Supervised fine-tuning guide: minimum 10 examples, start with about 50 demonstrations.
• OpenAI, Fine-tuning updates: reinforcement fine-tuning now available (May 2025): RFT for o4-mini.
• OpenAI, Deprecations (May 2026): update to self-serve fine-tuning.
• Anthropic, Fine-tune Claude 3 Haiku in Amazon Bedrock (July 2024), and AWS, Fine-tuning for Anthropic's Claude 3 Haiku in Amazon Bedrock is now generally available (November 2024).