Learn · Module 07 · Training and fine-tuning · Lesson 5 of 8
LoRA and adapters
Instead of changing all of a model's billions of weights, LoRA trains a tiny add-on and leaves the original untouched.
In short
- LoRA freezes the model and trains a small correction BA for chosen matrices.
- Rank-8 LoRA on a 4,096 × 4,096 matrix trains 0.39% of its numbers.
- Small adapters are cheap to train, store and swap; QLoRA adds a 4-bit frozen model.
This lesson is part of Pro
The first two lessons of every module are free. Pro opens every lesson and every project.
See Pro plansFrom ₹119 a month. No auto-renewal.