Most of my first versions call the Chat Completions API. Messages in, a message out. Tools if the product needs an action.
The playground is where I sketch. Production is versioned prompts, logs, and a budget.
Writing prompts that survive contact with users
I split the prompt into three piles:
- System — role, hard rules, output shape. This changes rarely.
- User — what this person just asked.
- Context — retrieved docs or tool results. This changes every request.
system: rules and format user: the question user: retrieved context, clearly labeled
If I mix rules and live data in one blob, I cannot tell which part broke.
Tokens, the only unit that matters
You are billed and you are limited in tokens, not words.
- Token counting — count input and expected output before you ship a feature that pastes large docs. A tokenizer for that model family is enough. Do not guess with word count.
- Maximum tokens — set an output cap. A summary does not need 4,000 tokens. A classifier needs a handful.
- Managing tokens — trim history, summarize old turns, and retrieve snippets instead of full files. I would rather drop old chit-chat than drop the policy doc.
The playground
OpenAI Playground is a sketchpad. I use it to see if a prompt shape works on five examples.
I do not leave production traffic pointed at a prompt that only exists in the playground. If I cannot paste it into the repo and run an eval, it is not real yet.
Fine-tuning
Fine-tuning changes weights so the model prefers a format or a tone you show it many times.
I fine-tune when:
- the output shape is stable
- I have hundreds of good examples, not ten
- prompting and retrieval already failed for a reason I can name
I do not fine-tune to “teach” private company facts. Those go in retrieval. A fine-tune also freezes you to a model snapshot. Prompt changes are easier to roll back.
Next
Search only works once text becomes a vector: