The AI Engineering Core Stack

The AI Engineering Core Stack

A demo can be a prompt in a notebook. A product needs a stack.

The usual layers

1. Model — hosted API or self-hosted open model
2. Orchestration — prompt assembly, routing, retries, tool calls
3. Knowledge — search, embeddings, or a database the model can query
4. Application backend — auth, business rules, rate limits, billing
5. Frontend — streaming, citations, undo, feedback
6. Evals + observability — traces, scores, cost, latency

What actually fails

Most production bugs are not “the model is dumb.” They are:

• missing or stale context
• a tool returning garbage
• timeout / rate-limit storms
• prompt drift after a silent change
• no way to tell if quality got worse this week

A useful rule

If you cannot point to where a request failed, you do not have an AI system yet. You have a demo with extra steps.

1. Was the prompt wrong?
2. Was the retrieved context wrong?
3. Did a tool fail?
4. Did the model invent something?
5. Did we even log enough to know?

Next

Once the stack is clear, learn how models actually behave:

• LLMs and Prompting