The challenge: non-deterministic AI in production
Adding a large language model to a product is straightforward in a prototype. Putting it in front of real users at high concurrency exposes problems a demo never shows:
- Unpredictable output: a response that is "almost" valid JSON still breaks the parser that consumes it.
- Provider latency and rate limits: upstream APIs slow down or throttle at exactly the moments traffic peaks.
- Cost that scales with every request: sending simple classification or lookup tasks to a top-tier reasoning model is slow and expensive.
This is the architecture we use as a starting point when we design AI features that have to survive production traffic. The details always change with the product, but the three layers below stay the same.
The architecture: route, validate, stream
Rather than a single monolithic API call, the pipeline is split into three layers, each with one job.
1. Route before you generate
A lightweight classification step decides what each request actually needs: a deterministic database lookup, a canned or cached answer, a small fast model, or a full reasoning model. Only the requests that genuinely need generation reach the expensive model, which cuts both cost and latency.
2. Validate every structured response
All structured output from the model passes through a strict schema validator (for example Zod in a TypeScript stack) before anything downstream sees it. When a response doesn't match, a constrained repair prompt - low temperature, schema supplied - retries it automatically, so the user never sees a raw error and malformed data never reaches the database.
3. Stream, and fail over instead of failing
Responses are streamed to the client (for example with Server-Sent Events), so users see the first words almost immediately instead of waiting for the whole answer. If the primary model provider slows down or returns errors, the orchestration layer switches to an alternative provider for that request rather than dropping the session.
What to measure
An AI feature is only as reliable as the numbers you watch. Before launch, we agree on targets and instrument these from day one:
- Schema validity rate: the share of structured responses that pass validation without a repair attempt - and how often repairs succeed.
- Time to first token and total response time: tracked at the median and at the slow tail (p95), not just on average.
- Routing mix and cost per request: how many requests each path handles, so the expensive model is used only where it earns its cost.
- Fallback rate: how often the secondary provider takes over, which is an early warning of upstream problems.
Ownership by design
The orchestration code, prompts and schemas are ordinary application code in the product's repository - version-controlled, tested and fully owned by the business, with no dependency on a proprietary orchestration platform.
Where to start
If you're adding AI to an existing product, start with the layer that hurts most today: validation if bad output is breaking things, routing if cost is the problem, or streaming and fallbacks if users are waiting. Each layer is valuable on its own, and together they turn an impressive demo into a feature you can run in production.