epochs

Learn · Module 06 · Transformers · Lesson 6 of 6

Images, audio and other inputs

A transformer only reads vectors. To read a picture or hear a voice, you turn them into vectors that look like tokens.

4 minute read

In short
  • Multimodal models turn images and audio into sequences of vectors, then use the same transformer.
  • Images are cut into patches; each patch becomes one token-like vector.
  • Image and audio tokens cost context and money, so resize and crop.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.