Learn · Module 06 · Transformers · Lesson 6 of 6
Images, audio and other inputs
A transformer only reads vectors. To read a picture or hear a voice, you turn them into vectors that look like tokens.
4 minute read
- Multimodal models turn images and audio into sequences of vectors, then use the same transformer.
- Images are cut into patches; each patch becomes one token-like vector.
- Image and audio tokens cost context and money, so resize and crop.
In plain words
Everything you've learned about transformers works on a sequence of vectors. Text becomes vectors through tokens and embeddings. A multimodal model adds a way to turn other inputs, such as images and audio, into a sequence of vectors too, then feeds them through the same transformer.
Worked example: an image becomes tokens
The common method, from the Vision Transformer, cuts the image into small square patches and treats each patch like a token.
- 1Cut the image into patches
A 224 × 224 pixel image, cut into 16 × 16 squares, gives a 14 × 14 grid.
(224 ÷ 16)² = 14 × 14 = 196 patches
- 2Flatten each patch
Each patch has 16 × 16 pixels with 3 colour values.
16 × 16 × 3 = 768 numbers per patch
- 3Project into the model's space
A learned matrix turns each 768-number patch into a vector the same size as a text token's embedding, and a position vector says where the patch was.
- 4Mix with text
The 196 image vectors sit in the sequence next to the text tokens, and attention lets words look at image patches.
[patch 1] … [patch 196] What is in this photo?
That's why images are expensive. A larger image or smaller patches mean many more tokens: a 336 × 336 image with 14-pixel patches is 576 tokens, and a 1,024 × 1,024 image with 16-pixel patches is 4,096 tokens, before you've typed a word.
Audio works the same way
Speech is turned into a spectrogram, a picture of which pitches are loud at each moment, then chopped into short time slices that become vectors. Whisper, a popular speech model, turns 30 seconds of audio into 1,500 vectors. Video is usually handled as a series of image frames, which adds up quickly.
You switch from 224 × 224 images to 448 × 448 images, keeping 16-pixel patches. How many image tokens per picture?
What it means for your product
Image and audio tokens count against the context window and the bill, just like text. Resize images to the smallest size that still shows what matters, and crop to the relevant region. Many APIs offer a low-detail mode for exactly this reason.
Models can still misread images: small text, counting objects and reading charts are common weak spots. Test on your real images before trusting a model with them.
Common mistakes
- Assuming the model sees the image pixel by pixel; it sees a grid of patches turned into tokens.
- Forgetting that images cost many tokens, which affects latency and price.
- Expecting reliable reading of tiny text or exact counts; resolution and patching limit both.
Answer in your own words, out loud or on paper, before you open the model answer.
How does a vision transformer process an image?
A vision transformer cuts the image into a grid of small square patches, for example 16 by 16 pixels each. Each patch is flattened and passed through a linear layer to become a vector, like a token embedding, and a positional embedding is added so the model knows where each patch came from. From there it's a standard transformer: attention lets every patch relate to every other, so the model can connect, say, a hand and the cup it's holding. In multimodal LLMs, an image encoder like this produces patch vectors that are projected into the language model's embedding space, so image tokens and text tokens sit in the same sequence.
- Why are images expensive to send to an LLM?
- How would you build a system that answers questions about invoices from photos?
Go further
- ArticleMultimodality and Large Multimodal ModelsChip Huyen
- PaperAn Image is Worth 16x16 Words (ViT)Dosovitskiy et al., 2020
- PaperCLIP: Learning Transferable Visual Models From Natural LanguageRadford et al., 2021
- PaperVisual Instruction Tuning (LLaVA)Liu et al., 2023
This lesson is part of Pro
The first two lessons of every module are free. Pro opens every lesson and every project.
See Pro plansFrom ₹119 a month. No auto-renewal.