A multimodal model handles more than one kind of input or output — text plus images, audio, video or documents. In practice it means you can hand a model a screenshot, a scanned invoice, a whiteboard photo or a recording and ask questions about it in ordinary language.
Each kind of input is converted into the same internal representation the model already uses for text, so an image becomes a sequence the model can attend to alongside your question. The consequence is that images consume context like everything else: a high-resolution screenshot can cost more tokens than several pages of prose.
It collapses a category of work that used to need a specialist pipeline. Reading a form no longer means an OCR engine, a layout parser and a set of brittle rules — you can ask for the fields you want and get structured data back. The same goes for describing a chart, explaining an error screenshot, or summarising a recorded session.
For anyone learning, it also lowers the barrier: photographing a problem and asking about it is a far shorter path than describing it accurately in text.
Assuming vision is as reliable as reading. Models are markedly weaker at precise spatial reasoning, dense tables and small text than their fluency suggests, and they will describe a chart confidently while misreading the axis. Anything numerical lifted from an image deserves a check.
The second mistake is sending full-resolution images by habit. Downscaling to the smallest size where the detail you need is still legible is usually a large cost saving for no loss of accuracy.
No application to be approved and no card to enter. Pick a name and a password, choose the rooms that fit, and say what you are building.
