Text was the first interface. The product work I care about now often starts with a photo or a voice note. HairOver is that: a face in, a haircut preview out. The model is one step. The product is the rest.
Multimodal means more than one kind of input or output. It does not mean you skip evals.
Use cases worth building
- Image understanding — “what is in this screenshot?” or “does this photo match the listing?”
- Image generation — a new image from a prompt, or an edit of a photo the user uploaded
- Video understanding — sample frames or a transcript first. Sending a raw long video to a model is how you melt a budget
- Audio processing — cleanup is often ordinary software. The model step is transcription or a short description
- Text-to-speech — read an answer aloud when the interface is voice
- Speech-to-text — turn speech into text, then run your normal text pipeline
APIs I reach for
- OpenAI Vision API — image in, text out. I send the image plus a tight question. “Describe everything” is how you get a poem instead of a label.
- DALL·E API — image generation when I want a hosted model and I can live with its style and policy limits. For a product like HairOver, the harder problem is identity (same face) not “a pretty picture.”
- Whisper API — speech to text. I store the transcript, not only the audio, so the rest of the system stays text.
- Hugging Face models — when I need a specific open vision, speech, or diffusion model and a closed API will not do.
LangChain for multimodal apps and LlamaIndex for multimodal apps can wire loaders and tools. I use them when the app already depends on them. A single vision call does not need a framework to feel serious.
How I implement it
- Validate the file before the model sees it: type, size, duration
- Call one model for one job
- Schema-check the output (labels, transcript, image URL)
- Log cost per image or per audio minute
- Moderate inputs and outputs when users can upload anything
Next
Models that can see and hear still need hard limits: