Model quality is mostly data quality. Most fine-tuning failures trace back to a few problems in the data.
3 minute read
In short
Clean, consistent and balanced data beats more data.
Remove duplicates, wrong labels and personal data before training.
Prevent leakage: split carefully and deduplicate across train and test.
In plain words
A model learns whatever its training data shows, including the mistakes. A small, clean, consistent dataset usually beats a large messy one. Building it is careful, unglamorous work, and it's where most of the gains are.
Worked example: cleaning a support dataset
A team exports 10,000 past support replies to fine-tune a reply model. Here's what a careful pass finds.
step
found
left
Start
10,000
Remove exact and near duplicates
1,800 copies of templated replies
8,200
Remove replies that were later corrected by a manager
600 wrong answers
7,600
Remove personal data (names, phones, emails)
kept, but masked
7,600
Balance categories
billing was 60% of data; trimmed
5,000
Hold out a test set before training
500 kept aside
4,500 to train
Half the data is gone, and the model will be better for it. Duplicates teach the model to over-use one template; wrong answers teach it to be wrong; imbalance makes it treat every question as billing.
The quiet killer: leakage
Leakage means test examples, or near copies of them, are also in the training set. The model has seen the answers, so the test score is inflated and you ship something worse than you think. Split by something natural, such as customer or date, and remove near-duplicates across the split, not just within it.
Your turn
Your fine-tuned model scores 96% on your test set but users say it's often wrong. You find 300 test questions also appear in the training data. What's going on?
What to remember
Read your data before training on it. Look at 50 random examples; you'll find problems no script would. Keep labelling guidelines written down so examples are consistent. Version your datasets like code, so you know exactly what each model was trained on.
Common mistakes
Collecting examples without checking for contradictions, which teaches the model to be inconsistent.
Leaving test examples, or near-duplicates of them, in the training set.
Including personal data from real tickets without masking it.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
How would you build a dataset to fine-tune a support model?
I'd start from real support conversations, since they show the actual questions and language customers use. I'd pick conversations that were resolved well, and have experienced agents fix or rewrite answers that were wrong or off-tone, so every example shows the behaviour I want. I'd make sure common question types are well covered, add examples of cases to hand off to a human, and remove personal data. Then I'd deduplicate, check for contradictions between examples, and hold out a test set of different conversations, ideally from a later time period, so I'm measuring generalisation. A few thousand clean, consistent examples usually beat many more messy ones.
More to think about
What is data leakage, and how do you prevent it?
How many examples do you need, and how do you know you have enough?