RAG vs fine-tuning: how SMEs should choose
RAG vs fine-tuning: changing facts, stable style, cost, data, risk. SME decision grid + classic traps. Links to RAG pipeline and LLMOps.
RAG vs fine-tuning: two levers, two jobs
RAG brings facts at question time. Fine-tuning changes (or adapts) model behavior via examples in weights or an adapter. You do not pick “the hypest tech”. You pick the lever that fixes a measured failure.
Typical failure “the model doesn’t know our catalog” → RAG first (or plain search). Failure “facts are fine but format/tone is wrong” → fine-tuning or robust prompts/skills may win.
Many teams fine-tune to hide a messy corpus. That is expensive and expires at the next price list or procedure change.
Decision grid in 6 questions
1) Do facts change weekly? Yes → RAG (or live base), not monthly retrain. 2) Do you have rights and quality example data for fine-tune? No → RAG + prompts. 3) Legal / customer risk? → retrieval + citations + HITL first.
4) Need style, domain language, ticket format? → skills/prompt, maybe light fine-tune. 5) Ops budget: can you serve a custom model? No → API + RAG. 6) Do you have an eval set? Without eval, neither is “validated”.
Mental score: if 3+ answers push documents and freshness, start with RAG. If corpus is already clean and failure is pure behavior, look at fine-tune.
Hidden costs of both options
RAG: ingest, vector storage, re-index, retrieval eval, ACLs, citations. Recurring cost often underestimated on the data-engineering side.
Fine-tuning: collect and label, training runs, regression eval (model “forgets” cases), serving a distinct model, adapter versioning, training-data leak risk.
Compare over 90 days: money + person-days + incidents. A “free” vendor fine-tune still costs eval and maintenance.
Patterns that work together
RAG for facts + system prompt / skills for process. Fine-tune (or adapter) only on output format once retrieval is stable. Agents: deterministic tools for compute, RAG for docs, HITL for send.
Avoid ungoverned dual systems: opaque fine-tune + unevaluated RAG. One measurable pipeline beats two hacks.
Local / open weights: local fine-tune possible but heavy; RAG on internal corpus is often the first lever (see local AI and open-weights news).
Classic traps
Fine-tuning on stale FAQs. Measuring success by “demo vibe”. Ignoring ACLs. Believing fine-tune removes the need for citations. Changing model and corpus the same day without a baseline.
Countermeasure: frozen baseline, one variable at a time, experiment log, data owner + model owner.
One-page decision (template)
Context · measured failure · option A RAG · option B fine-tune · option C prompt/skills only · 90-day cost · risk · eval · go / no-go · review date.
If go is RAG, follow the RAG LLM guide. If fine-tune, require regression eval and retrain plan. If both, sequence: RAG first.
If you want to arbitrate RAG vs fine-tuning on a real case, we can scope in 20-40 minutes.
FAQ
- RAG or fine-tuning: which first?
- For most SMEs, RAG (or search + prompt) first. Fine-tune when the remaining failure is behavioral and example data is clean.
- Does fine-tuning replace RAG?
- Rarely for moving facts. Weights are not a real-time document store.
- Can we combine both?
- Yes: RAG for substance, fine-tune/adapter for format. One eval system for both.
- How many examples for a useful fine-tune?
- It depends on the task. Quality > quantity. Hundreds of clean examples beat thousands of noisy rows. Validate with eval, not raw volume.
- What about prompt engineering?
- Always the first layer. Many “we must fine-tune” cases resolve with specs, skills and versioned few-shot examples.
- Link to context engineering?
- Context engineering = what you put in the window (including RAG results). Fine-tune = what lives in the weights. Complementary.
Scope your first AI agent
20 minutes to review your tools, data and the first useful case. No jargon, no commitment.