Area 01 · LLM · Concept 15 of 20

Multimodal Inputs

Sending images, PDFs or audio to the model along with text.

11–20 · Production levelPhase 1 · Weeks 3–6≈ 45 min

The idea

Multimodal models accept more than text. Images are converted into tokens (patches) and placed in the context next to your words, so the model can read screenshots, charts, handwriting and photos. Many APIs also take PDFs directly, combining extracted text with page images.

Images cost tokens based on their resolution. Resize large images to what the task needs. Put the image before the question, and ask specific questions ("What is the total on this receipt?") rather than "Describe this".

Accuracy varies: models are strong on general scenes and clean documents, weaker on small text, precise counting and exact spatial positions. Validate extracted numbers like any other output.

Why it matters

Real business data lives in PDFs, scans and screenshots. Reading them directly removes a whole parsing pipeline in many cases.

Words to know

  • Vision modelA model that accepts images as input.

Try it without code

Read a receipt

  1. Photograph a receipt and upload it to a chat app.
  2. Ask for merchant, date and total as JSON.
  3. Try a blurry photo and see what breaks.

Use it in code

Send a local image as base64 with a question.

python
import base64, anthropic
client = anthropic.Anthropic()

data = base64.standard_b64encode(open("receipt.jpg", "rb").read()).decode()
msg = client.messages.create(
    model="claude-sonnet-5-5", max_tokens=300,
    messages=[{"role": "user", "content": [
        {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": data}},
        {"type": "text", "text": "Return JSON: merchant, date (YYYY-MM-DD), total (number)."},
    ]}],
)
print(msg.content[0].text)
PRACTICE · TICK IT AFTER THIS

Add image upload to your chatbot: if the user passes a path, attach the image to the next message.

Common mistakes

  • Sending 12-megapixel photos when 1,000 pixels wide is enough.
  • Trusting exact counts or tiny text without checking.

Go deeper