LLMs and Prompting
Language models predict the next token. Everything else is product design around that fact.
Constraints you cannot ignore
- Context window — too much junk in, worse answer out
- Tokens — cost and latency scale with input + output
- Hallucinations — the model will invent a confident answer when it should say “I don’t know”
- Non-determinism — same prompt can still vary unless you lock temperature (when the API allows it)
Prompting that actually works
Good prompting is reducing ambiguity:
- Say the role, the audience, and the output format
- Put hard rules in the system prompt
- Put changing facts in retrieved context, not in the system prompt
- Ask for structured output (
JSON, bullets, a schema) when another system will consume the result
System: You are a support assistant. Only use the provided docs. If the docs do not contain the answer, say you don't know. User: Question Context: [retrieved snippets]
Next
When the model needs facts it was not trained on: