You write code and want to put a model into something real without it embarrassing you.
Reviewed
Two to four weekends, depending on how far you take step four
By the end you will have a feature backed by a model, returning validated structured output, grounded in your own data, with an eval set that tells you when a change made it worse.
Building the demo takes an afternoon. Everything after that — making it reliable, knowing whether a change helped, keeping it from doing something stupid with a hostile input — is the actual work, and it is where most AI features stall between "impressive in the meeting" and "shipped".
These steps are in dependency order. Skipping to step three before step two is why so many retrieval systems produce output nothing downstream can consume.
Call a model API directly before reaching for a framework. Send a prompt, stream the response, and print the usage numbers on every call. Frameworks are useful later and they hide exactly the things you need to build intuition about now: how much of your latency is the input being read, what your prompt actually costs, how the message roles are assembled.
Instrument tokens per request from the very first commit. Cost problems in AI features are never discovered gradually; they are discovered in an invoice.
Prose is a fine output for a human and a terrible one for a program. Define the schema you want, ask for it explicitly, and validate what comes back before anything downstream touches it. Use the provider's structured-output or tool-calling mode if it has one; parse-and-pray is not an error-handling strategy.
Decide now what happens when validation fails: retry with the error message included, fall back to a simpler path, or surface the failure. A feature that silently passes malformed data on is worse than one that visibly refuses.
The moment your feature needs to know something specific to your product, users or documents, you need retrieval rather than a bigger prompt or a fine-tune. Chunk the documents, embed them, store the vectors, retrieve the closest ones at question time, and instruct the model to answer from them and cite which chunk.
Measure the retrieval step on its own before you judge the answers. Ask: for these fifty real questions, did the chunk containing the answer come back in the top few results? If it did not, no amount of prompt work downstream will save the output.
Collect twenty to fifty real inputs — from your own testing, from colleagues, from anything already in production — and write down what the right output would have been for each. Add a grader per case: an exact check where one is possible, a rubric where it is not.
Now you can answer the only question that matters when you change a prompt, swap a model or adjust chunking: did that make it better? Teams without this ship changes on the strength of one example and discover the regressions from users.
Treat every retrieved document, tool result and user input as untrusted. Give the feature the narrowest permissions that let it work, validate output structurally, log every tool call, and put a human confirmation in front of anything irreversible. These are cheap while you are building and expensive to retrofit after an incident.
Then ship it to a small group and watch the logs. The inputs real users produce are not the inputs you tested with, and the first week of real traffic will hand you the next twenty cases for your eval set.