The idea
LoRA shrinks what you train, but the frozen base model still has to sit in GPU memory. QLoRA loads that base model in 4-bit precision, cutting its memory by about four times compared with 16-bit, and trains LoRA adapters in higher precision on top.
This makes it possible to fine-tune 7–8B models on a single consumer or cloud GPU, and much larger models on one big GPU. Quality is close to 16-bit LoRA for most tasks.
The trade-off is speed: 4-bit math is slower per step. For production you can merge the adapter into a full-precision base or serve the adapter on top of a quantized model.
It turns fine-tuning from a cluster job into something you can run on one rented GPU for a few euros.
Words to know
- NF4The 4-bit number format introduced by QLoRA.
- bitsandbytesThe library that loads models in 8-bit or 4-bit.
Try it without code
Check memory maths
- Take an 8B model: 16-bit weights ≈ 16 GB.
- At 4-bit ≈ 4–5 GB.
- Look up which cloud GPUs fit each and their hourly price.
Use it in code
Load a model in 4-bit and attach LoRA.
# pip install transformers peft bitsandbytes accelerate (needs an NVIDIA GPU)
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B", quantization_config=bnb,
device_map="auto")
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, task_type="CAUSAL_LM",
target_modules="all-linear"))
model.print_trainable_parameters()
print(f"{torch.cuda.memory_allocated() / 1e9:.1f} GB on GPU")Run QLoRA on a free Colab or a rented GPU with a 7B model and record peak memory and minutes per epoch.
Common mistakes
- Trying to run bitsandbytes 4-bit on a Mac or CPU.
- Comparing quality without the same eval set.