Area 01 · LLM · Concept 17 of 20

QLoRA

LoRA on top of a 4-bit quantized base model, so fine-tuning fits on one GPU.

11–20 · Production levelPhase 1 · Weeks 3–6≈ 2 hours

The idea

LoRA shrinks what you train, but the frozen base model still has to sit in GPU memory. QLoRA loads that base model in 4-bit precision, cutting its memory by about four times compared with 16-bit, and trains LoRA adapters in higher precision on top.

This makes it possible to fine-tune 7–8B models on a single consumer or cloud GPU, and much larger models on one big GPU. Quality is close to 16-bit LoRA for most tasks.

The trade-off is speed: 4-bit math is slower per step. For production you can merge the adapter into a full-precision base or serve the adapter on top of a quantized model.

Why it matters

It turns fine-tuning from a cluster job into something you can run on one rented GPU for a few euros.

Words to know

  • NF4The 4-bit number format introduced by QLoRA.
  • bitsandbytesThe library that loads models in 8-bit or 4-bit.

Try it without code

Check memory maths

  1. Take an 8B model: 16-bit weights ≈ 16 GB.
  2. At 4-bit ≈ 4–5 GB.
  3. Look up which cloud GPUs fit each and their hourly price.

Use it in code

Load a model in 4-bit and attach LoRA.

python
# pip install transformers peft bitsandbytes accelerate  (needs an NVIDIA GPU)
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", quantization_config=bnb,
                                             device_map="auto")
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM",
                                         target_modules="all-linear"))
model.print_trainable_parameters()
print(f"{torch.cuda.memory_allocated() / 1e9:.1f} GB on GPU")
PRACTICE · TICK IT AFTER THIS

Run QLoRA on a free Colab or a rented GPU with a 7B model and record peak memory and minutes per epoch.

Common mistakes

  • Trying to run bitsandbytes 4-bit on a Mac or CPU.
  • Comparing quality without the same eval set.

Go deeper