Vision and documents

Models that see take images in the same request as text: screenshots, photos, charts, scanned pages. Some also read PDFs and other documents directly. Look for image and document in a model's input_modalities in the model list, or filter for vision on the models page.

Send an image

Make the user message's content an array of parts. An image is an image_url part holding a public HTTPS URL or a base64 data URL:

const completion = await client.chat.completions.create({
  model: "gemini-3.6-flash",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What's in this picture?" },
        { type: "image_url", image_url: { url: "https://example.com/photo.jpg" } },
      ],
    },
  ],
});

A local file goes in as a data URL:

import { readFile } from "node:fs/promises";

const base64 = (await readFile("chart.png")).toString("base64");
const part = { type: "image_url", image_url: { url: `data:image/png;base64,${base64}` } };

• Add several image_url parts to send several images; refer to them by order in your text.
• JPEG, PNG, WebP and GIF are widely supported. Very large images may be scaled down by the model.
• image_url.detail (low, high, auto) is honoured by models that support it; low costs fewer tokens.
• Images are billed as input tokens, so a large image costs more than a short sentence.

Send a PDF

Models that read documents take a file part with the document as a data URL:

{
  "model": "gemini-3.6-flash",
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "Summarise this contract's termination clauses." },
        {
          "type": "file",
          "file": { "filename": "contract.pdf", "file_data": "data:application/pdf;base64,JVBERi0xLjcK..." }
        }
      ]
    }
  ]
}

For text extraction from many pages at a fixed price per page, OCR is usually cheaper than sending documents to a chat model.

On the Messages endpoint

The Anthropic format uses image and document blocks, with a base64 or URL source:

{
  "model": "claude-opus-4-8",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "image", "source": { "type": "url", "url": "https://example.com/photo.jpg" } },
        { "type": "document", "source": { "type": "base64", "media_type": "application/pdf", "data": "JVBERi0xLjcK..." } },
        { "type": "text", "text": "Compare the photo with page 2 of the document." }
      ]
    }
  ]
}

On the Responses API

Use input_image parts with an image_url, and input_file parts with file_data and a filename, inside a user message in input.

Requests too large

A request body can be up to 64 MiB. Past that you get a 413. For big files, prefer a URL the model can fetch over inlining the bytes.