GENERATIVE AI: ZERO TO JOB-READY › UNIT 4 › MODULE 3 OF 4
Fine-Tuning vs Prompting — When to Use Which
⏱ 13 min read📝 Assignment insideDecision framework
You now understand prompting (Units 1–2), RAG (Module 4.1), and agents (Module 4.2). This module answers the question that comes up in almost every Indian IT AI project discussion: when is prompting enough, when should you use RAG, and when do you actually need to fine-tune?
This is not a deeply technical module — you do not need to run a training job to understand it. You need to understand the decision logic, the trade-offs, and the cost implications. That understanding is what separates a professional who can discuss AI architecture from one who can only use AI tools.
The honest answer most teams miss
Prompt engineering and few-shot examples solve approximately 70% of LLM performance problems. Fine-tuning is the 30% solution. Most teams reach for fine-tuning too early — they spend weeks preparing data, renting GPUs, and debugging training loops. Meanwhile, a better-crafted system prompt with few-shot examples and chain-of-thought reasoning would have solved the problem in an afternoon. The question is not "which is better?" — it is "which is appropriate for this specific problem right now?"
The right sequence in 2026 is: Prompt → RAG → Fine-tune → Distill. Move to the next level only when the previous level genuinely cannot solve the problem.
The four-level decision ladder
1
Prompting & few-shot examples Start here always
Write a clear, well-structured prompt. Add role, context, format, and constraints. Add 1–3 few-shot examples if style matters. Use chain-of-thought for complex reasoning. This is fast, free, and reversible.
Use when: you need a general-purpose task done well. The model already knows how to do this type of work from training. You want to change the style, format, or depth of response.
2
RAG — retrieval augmented generation For knowledge problems
Connect the model to your documents, databases, or knowledge bases. The model retrieves relevant content at query time and answers from it. No retraining, always up to date. Covered in Module 4.1.
Use when: the model needs access to your specific data — internal policies, product details, client information, recent events — that was not in its training data.
3
Fine-tuning For behaviour problems
Retrain the model weights on your specific data to teach it a new behaviour, style, or task pattern. Expensive and time-consuming, but produces a model that reliably behaves in a specific way without needing the behaviour described in every prompt.
Use when: prompting has hit a performance ceiling, you need consistent output format or style at scale, you have 500+ high-quality training examples, or latency and cost require a smaller specialised model.
4
Distillation For cost / scale problems
Train a smaller, cheaper model to mimic the outputs of a larger one on your specific task. The result is a model that is fast, cheap to serve, and specialised. Only relevant when you have very high query volumes and cost is a primary constraint.
Use when: your fine-tuned model is working well but API costs at scale are unsustainable. This is typically a problem for products serving millions of users, not internal enterprise tools.
Prompting vs RAG vs fine-tuning — full comparison
Dimension
Prompting
RAG
Fine-tuning
What it changes
What the model is asked to do in each conversation
What information the model has access to at query time
The model's weights — permanently changes how it behaves
Fine-tuning is for form, not facts. You use it to shape behaviour, style, structured output, and refusal patterns — not to inject knowledge that changes frequently. Here are the four situations where fine-tuning genuinely makes sense:
The three fine-tuning methods you will hear about
You do not need to run fine-tuning yourself to be in this conversation. But knowing what LoRA and QLoRA mean — and why they matter — is now expected in Indian IT AI project discussions.
Full fine-tuning
Updates all model weights. Highest performance ceiling, but requires significant GPU memory and compute. Expensive and slow. Only large organisations with ML infrastructure run this.
High cost — rarely used in Indian IT
LoRA / QLoRA
Parameter-efficient fine-tuning — only a small adapter layer is trained on top of the frozen base model. QLoRA adds quantisation (uses less memory). Gets 90% of full fine-tuning performance at 10–100x lower cost. The standard enterprise approach in 2026.
Recommended — most enterprise use cases
Instruction tuning
Teaches the model to follow your specific instructions, format, or template. Used when you need the model to reliably follow a particular output schema (JSON, a specific report format) without being told in every prompt.
Good for format consistency
The practical point: QLoRA fine-tuning on a 7B–13B parameter model gets approximately 90% of GPT-4 performance on narrow, specific tasks at approximately 1/50th the inference cost. This is the right choice for most enterprise fine-tuning use cases in Indian IT in 2026 — when fine-tuning is justified at all.
Real Indian IT scenarios — which approach?
📢 HR policy chatbot at a 10,000-employee Indian company
Employees ask about leave policies, salary slips, and appraisal processes. The HR policy document changes quarterly. The questions are factual and answers must be accurate and current.
Answer: RAG — not fine-tuning. Knowledge changes frequently, documents exist, no behaviour change needed.
💻 Code review tool for a company's Java style guide
The company has a 50-page internal Java style guide with specific naming conventions, patterns, and anti-patterns. They want AI to flag violations consistently without a long system prompt per review.
Answer: Fine-tuning (LoRA). Consistent style behaviour needed. Training data is the style guide violations + corrections.
📋 Weekly project status report generator
A delivery manager pastes bullet points and wants a formatted status report in the standard company template every Friday. The template is consistent but the content changes weekly.
Answer: Prompting — few-shot with the template example. No fine-tuning or RAG needed. The template is a few-shot example.
🏛 Customer support bot for an Indian insurance company
Customers ask about policy terms, claim procedures, and premium calculations. Policy documents are 200+ pages, change with regulatory updates, and have India-specific terms like IRDAI, ULIP, and NPS.
Answer: RAG with domain prompt tuning. RAG for the policy knowledge, a specialised prompt for the Indian insurance context and terminology.
📈 Sentiment classifier for 1 million daily customer reviews
An e-commerce company receives a million product reviews daily in Hindi and English. They need to classify sentiment and sub-category (delivery, quality, price) at scale with low latency.
Answer: Fine-tuning + distillation. Prompting won't scale. A fine-tuned small model on labelled examples is the right architecture.
🏫 Study assistant for engineering students
A college wants an AI assistant that answers questions about their specific curriculum, NPTEL course material, and past exam papers — updated each semester when new material is uploaded.
Answer: RAG. Course material is the knowledge base. Updates each semester with no retraining. Fine-tuning would be expensive and stale.
The decision framework — four questions to ask
ASK THESE FOUR QUESTIONS IN ORDER BEFORE RECOMMENDING AN APPROACH
Is the problem that the model doesn't have the right information?→Use RAG. Give it access to your documents. No fine-tuning needed. This is the correct diagnosis for 80% of "the model doesn't know our stuff" problems.
Is the problem that the model isn't behaving the way you want — style, format, tone, refusals?→Try better prompting first. Few-shot examples with 2–3 samples of the desired output almost always solve format and style problems. If prompting genuinely cannot solve it after real effort, then consider fine-tuning.
Have you genuinely exhausted prompting — CoT, few-shot, role, structured output, refinement?→Only now consider fine-tuning. Do you have 500+ high-quality labelled examples? Do you have the time and budget for 2–4 weeks of data preparation plus training? If yes to both, fine-tuning may be appropriate.
Is the problem that the working solution is too expensive to run at your required scale?→Consider distillation. Use the large model to generate labelled outputs, then train a smaller model on those outputs. Only relevant at very high query volumes — not a typical Indian IT enterprise concern at the POC stage.
The hidden cost of fine-tuning most teams miss
When teams decide to fine-tune, they typically budget for compute costs. The real cost is elsewhere. The operational cost of maintaining a fine-tuned model — adapter versioning, rollback plans, retraining cadence, and base-model drift management — are recurring costs, not one-time work.
When a hosted provider updates their base model, your adapter may degrade silently. Plan quarterly revalidation. Treat training configs, seeds, and dataset snapshots as code. Budget three to five times the training cost for ongoing lifecycle ownership over the next 12 months. The real cost is not the training compute — it is the evaluation, data curation, and lifecycle ownership.
This is why the decision framework above consistently recommends exhausting prompting and RAG before committing to fine-tuning. For most Indian IT enterprise use cases in 2026, the correct answer is a well-engineered RAG system combined with carefully crafted prompts — not a fine-tuned model.
The three things to remember
KEY TAKEAWAYS — MODULE 4.3
The right sequence is Prompt → RAG → Fine-tune — prompting solves approximately 70% of problems, RAG solves most knowledge-gap problems, and fine-tuning is for the remaining cases where behaviour genuinely cannot be achieved through prompting. Moving to fine-tuning before exhausting the earlier levels wastes significant time and money.
Fine-tuning is for form, not facts — use it to teach consistent behaviour, style, format, and output patterns. Never use it to inject factual knowledge that changes over time. That is what RAG is for. The best 2026 architecture is hybrid: RAG for volatile knowledge, fine-tuning for stable behaviour.
The hidden cost is lifecycle, not training — fine-tuning creates an ongoing maintenance obligation. Adapter versioning, quarterly revalidation, and base model drift management are recurring costs. Budget three to five times the training cost for the first year of operations. This calculation alone rules out fine-tuning for most Indian IT enterprise use cases.
✎
Assignment 4.3 — Apply the Decision Framework to a Real Use Case
This assignment develops the architectural reasoning skill that distinguishes AI practitioners from AI tool users.
Choose a real AI use case from your domain — either the RAG use case from Assignment 4.1, the agent workflow from Assignment 4.2, or a new one. Describe it in two sentences: what the system does and who uses it.
Work through the four-question decision framework:
Is the problem a knowledge gap? → if yes, describe the RAG implementation
Is it a behaviour problem? → if yes, describe what prompting approach to try first
Would prompting genuinely fail here? → if yes, justify why and what training data would look like
Is scale and cost the driver? → if yes, describe the query volume and economics
State your recommendation — Prompting, RAG, Fine-tuning, or a combination — with one paragraph justifying the choice using the criteria from this module.
Add to your AI Learning Journal under "Module 4.3 — Architecture Decision." Together with Assignments 4.1 and 4.2, this completes your Unit 4 AI architecture portfolio — three concrete examples showing you can think at a systems level.
Being able to recommend Prompt vs RAG vs Fine-tuning with clear justification — and explain why fine-tuning is usually not the answer — is one of the most valued skills in every AI project discussion at Indian IT companies right now.