An AI demo has one path: the part that works. A real feature also needs a plan for the part that does not.
Decide what the model is allowed to do
Write down which data the feature reads, which actions it can take and where a person must confirm. Generating a draft is different from sending it. Suggesting a file to remove is different from deleting it. Treat tool permissions as product decisions, not implementation details hidden in a prompt.
Use the least access needed for the task. Keep secrets out of model context unless they are required and handled appropriately. If the system calls an external model, explain the data path to users and follow your organisation's policies. NIST's AI Risk Management Framework offers a way to identify, assess and manage risks across the system lifecycle.
Validate output before it reaches a user
Prefer a structured schema when the next step expects structured data. Parse the response, reject invalid fields and make missing information visible. Check source citations against the actual retrieved passages. A syntactically valid response can still be factually wrong, so use task-specific checks and review the cases that matter.
For user-generated text, mark suggestions as drafts and let the user edit before sending. For decisions with serious consequences, do not rely on a general language model as the only authority. Get appropriate human expertise and safeguards. This tutorial is about general product engineering, not high-stakes domain advice.
Make failure ordinary and recoverable
A timeout should show a useful message. Missing evidence should produce 'I could not find that' instead of an invented answer. A model provider being unavailable should leave the rest of your app usable where possible. Keep a non-AI path when the task can still be completed manually.
Log enough to debug: model version, prompt version, failure type and timing. Do not retain raw sensitive data just because it's convenient. Decide retention and access deliberately. Add a visible report button and a mechanism to recover the user's work if generation fails.
Watch the feature after launch
Monitor completion, errors, latency, cost and user corrections. Sample outputs with the right data protections. Changes in users, model versions, tool APIs or source documents can introduce new failures even when your code did not change.
Keep a regression set drawn from real task patterns and redacted failures. Run it before changing a prompt or model. Make a rollback possible. An early launch is a way to learn under controlled limits, not permission to stop checking what the feature does.
A well-scoped first release lets a person try the helpful part, see when the AI is unsure, and continue the job if it fails. That is a much better milestone than a perfect demo.
FOLLOW THE EVIDENCE
Sources & further reading
Process note: AI tools assisted with research and drafting. Sources are linked so you can inspect the evidence. Unless an essay says otherwise, it does not claim hands-on testing or independent reproduction.
