Fine-tuning versus better prompting: a practical decision guide

"Should we fine-tune a model on our data?" is one of the most common questions we hear at the start of AI projects. The honest answer is usually "not yet," and occasionally "yes, and it will pay off handsomely." The difference comes down to what problem you are actually trying to solve. Fine-tuning changes how a model behaves; it is a poor way to teach it facts. This guide lays out how we decide.

What each approach actually changes

It helps to be precise about the three main levers:

• Prompting changes the instructions and examples the model sees at request time. It is fast to iterate, costs nothing to set up, and can be changed in minutes.
• Retrieval supplies relevant facts at request time from your documents or systems. It is the right tool when answers depend on information that changes or is too large to fit in a prompt.
• Fine-tuning adjusts the model's weights using example inputs and outputs. It is good at teaching consistent formats, styles, classification boundaries and specialized behaviors; it is poor at teaching reliable, up-to-date facts.

Teams often reach for fine-tuning because they want the model to "know our products." That is a retrieval problem. A fine-tuned model will learn to sound like it knows your products, which is worse than not knowing them, because the errors become harder to spot.

Fine-tune for behavior. Retrieve for knowledge. Prompt for everything else, first.

When fine-tuning earns its cost

There are clear cases where fine-tuning is the right call. In our projects, it has paid off in four situations:

1. High-volume, narrow tasks where cost or latency matters. A classification or extraction task run millions of times a month can often move from a large model to a fine-tuned small model at a fraction of the cost and a third of the latency, with equal or better accuracy.
2. Strict output formats the base model keeps getting slightly wrong. Domain-specific structured output, such as a particular coding scheme or document markup, where prompting reaches 95% compliance and you need 99.5%.
3. Consistent voice at scale. Content pipelines where every piece must match a distinctive house style that is hard to describe in a prompt but easy to demonstrate with a thousand examples.
4. Specialized judgment with lots of labeled history. Triage or routing decisions where years of expert-labeled data exist and the boundaries between categories are subtle.

In all four, the common thread is a stable, well-defined task, plenty of high-quality examples and enough volume that per-request savings add up.

The hidden costs of fine-tuning

The training run itself is rarely the expensive part. The costs that surprise teams are elsewhere:

• Data preparation. Cleaning, deduplicating and reviewing examples typically takes more time than everything else combined. A few hundred bad examples can undo the benefit of thousands of good ones.
• Evaluation. You need a held-out test set and a comparison against the best prompted baseline, or you will not know whether the tuned model is better.
• Maintenance. When the base model is updated or deprecated, you retrain. When your categories or style change, you relabel and retrain. Prompts can be edited; weights cannot.
• Lock-in. A fine-tuned model is tied to a provider or a hosting setup. Switching means repeating the work.
• Forgetting. Narrow tuning can reduce performance on anything outside the training distribution, which matters if the task drifts.

For a typical mid-sized task, we estimate two to four weeks of work for a first fine-tuned model done properly, and a day or two of maintenance per retraining cycle afterward.

There is also an opportunity cost. Base models improve quickly, and a fine-tuned model built on last year's base can be overtaken by a newer model with a good prompt. We have seen tuned models lose their advantage within months. That does not make tuning a mistake, but it means the business case should pay back well within the expected life of the base model.

Our decision process

We follow the same sequence on every project, and most stop before the last step:

1. Build an evaluation set that defines success, including cost and latency targets.
2. Get the best result possible with prompting on a capable model, including a handful of well-chosen examples.
3. Add retrieval if failures are about missing or outdated information.
4. Try a smaller or cheaper model with the same prompt, to see how much quality you give up.
5. Only if a clear gap remains in quality, cost or latency, and you have at least a few hundred high-quality examples, fine-tune and compare against the best baseline.

A client in logistics followed exactly this path for a document classification task running about two million times a month. Prompting a large model reached 96.2% accuracy at a cost that did not work at their volume. A fine-tuned small model trained on 6,000 reviewed examples reached 97.1% at roughly one twelfth of the per-request cost. That was a clear win. The same client's customer-facing assistant, by contrast, runs on prompting and retrieval alone, because its knowledge changes weekly.

Getting the answer for your use case

If you are unsure whether fine-tuning is worth it, the evaluation set will tell you, and it is useful whichever way the decision goes. Our model fine-tuning engagements always start with a prompted baseline, and many end with a recommendation not to tune at all. When knowledge is the gap, a RAG knowledge base is usually faster, and high-volume writing work often fits an AI content pipeline with a tuned model at its core.

Find out if tuning will pay off

Share the task, your monthly volume and a few sample inputs and outputs. We will reply within 24 hours with a recommendation and a fixed-price quote for the baseline and, if it is warranted, the tuned model.