epochs

Learn · Module 07 · Training and fine-tuning · Lesson 5 of 8

LoRA and adapters

Instead of changing all of a model's billions of weights, LoRA trains a tiny add-on and leaves the original untouched.

4 minute read

In short
  • LoRA freezes the model and trains a small correction BA for chosen matrices.
  • Rank-8 LoRA on a 4,096 × 4,096 matrix trains 0.39% of its numbers.
  • Small adapters are cheap to train, store and swap; QLoRA adds a 4-bit frozen model.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.