Key takeaways
ORI’s 30-second summary
- Start from a costly, repetitive user task — not from the technology.
- Retrieval over your own content usually beats fine-tuning for accuracy and cost.
- Build an evaluation set before launch so you can measure quality and catch regressions.
- Control cost with model routing, caching and usage limits from day one.
Most product teams are under pressure to "add AI". Some of those features become the reason customers renew. Many more end up as a chat box nobody opens and a monthly model bill nobody budgeted for. The difference is rarely the model — it is the way the feature was chosen, built and measured.
Start from the task, not the technology
The best AI features remove a task users already find slow, repetitive or error-prone. Look for moments where people:
- Search through long documents, tickets or records to find one answer.
- Write the same kind of text again and again — replies, summaries, descriptions.
- Copy data from one format into another, such as invoices into a system.
- Need a first draft that an expert then checks and approves.
If you cannot name the task and roughly how much time it costs today, the feature is not ready to build.
Pick the simplest architecture that works
Prompting
For drafting, rewriting and classification, a well-designed prompt with clear examples often gets you most of the way. It is cheap to build and easy to change.
Retrieval (RAG)
When answers must come from your own content — help articles, policies, product data — retrieve the relevant passages and give them to the model. It keeps answers grounded, lets you cite sources and updates instantly when content changes.
Agents and tools
When the feature needs to take actions — look up an order, book a slot, update a record — give the model a small set of well-defined tools with strict permissions. Keep humans in approval for anything irreversible.
Fine-tuning is rarely the first step. Most teams get better accuracy for less money from good retrieval and clear instructions.
Measure quality before launch
AI features fail quietly: they produce fluent, plausible and wrong answers. Before shipping, build an evaluation set of real questions with known good answers. Run it on every change to prompts, models or data, and track the score like any other quality metric.
- Include edge cases, ambiguous requests and questions the feature should refuse.
- Have domain experts grade a sample of answers, not just engineers.
- Collect thumbs-up and thumbs-down feedback in production and feed failures back into the evaluation set.
Keep costs predictable
Model pricing has fallen, but usage can grow faster than revenue if nobody is watching. Build cost control in from day one:
- Route simple requests to smaller, cheaper models and reserve the most capable ones for hard tasks.
- Cache repeated prompts and stable context such as system instructions and documents.
- Set per-user and per-account usage limits that match each pricing plan.
- Track cost per feature and per customer, not just the total monthly bill.
Design for trust
Tell users when they are dealing with AI, show sources where possible, and make it easy to correct or override the output. In Europe this is increasingly a legal requirement, and everywhere it is simply good product design.
The bottom line
Successful AI features look boring from the inside: a clear task, a simple architecture, an evaluation set and a cost dashboard. Start small, measure honestly, and expand only what users actually rely on.