Module 6 of 18
Multimodal Prompt Engineering
Design prompts that work across text, images, audio, and documents for modern multimodal AI models
How Multimodal Models See
Modern AI models are no longer limited to text. Multimodal models like GPT-4o, Claude, and Gemini can process images, documents, audio, and video alongside text, opening entirely new categories of tasks for prompt engineers.
Understanding how these models perceive non-text inputs is critical to designing effective prompts. Unlike humans who see images holistically, multimodal models process visual information through a pipeline: the image is converted into a sequence of visual tokens, which are then processed alongside text tokens in the model's transformer architecture.
The key insight is that multimodal models don't "see" the way humans do. They interpret images as patterns of visual tokens, which means spatial relationships, fine text within images, and subtle visual details may require explicit prompting to surface. The more specific your instructions about what to look for, the better the output.
Visual Token Processing
Images are divided into patches and encoded as tokens. Higher resolution means more tokens and greater detail, but also higher cost and latency.
Text-Image Attention
The model attends to both text and image tokens simultaneously, allowing it to correlate text instructions with specific visual regions.
Resolution Trade-offs
Sending higher resolution images improves detail extraction but increases processing time and cost. Match resolution to your task requirements.
Modality Grounding
Models can "ground" their text responses in visual evidence, reducing hallucination when you explicitly ask them to reference what they see.
The most effective multimodal prompts tell the model what to look for, where to look, and how to report what it finds. Vague prompts like "What do you see?" waste the model's analytical potential.
Image Prompting Patterns
Different image types require different prompting strategies. A chart demands data extraction, a screenshot calls for UI analysis, and a photograph benefits from descriptive framing. Mastering these patterns lets you unlock reliable results from any visual input.
Describe-Then-Ask
Ask the model to first describe what it sees, then answer your question. This two-step approach forces the model to ground its reasoning in visual evidence before jumping to conclusions.
"First, describe the contents of this image in detail. Then, based on your description, answer: [your question]."
Annotated Screenshot Analysis
When working with UI screenshots, provide context about the application and ask targeted questions about specific elements. Reference locations using natural language ("the button in the top-right corner").
"This is a screenshot of [app context]. Focus on [specific area]. Evaluate [specific criteria] and suggest improvements."
Chart and Data Extraction
For charts and graphs, explicitly request numerical extraction and specify the output format. Models perform significantly better when told to output data as a table or JSON rather than freeform text.
"Extract all data points from this [chart type]. Output as a markdown table with columns for [labels]. Then identify the key trend."
Comparative Image Analysis
When providing multiple images, explicitly label them and ask for structured comparison. Without labels, models may confuse which image they're referencing.
"Image 1 shows [context]. Image 2 shows [context]. Compare them across these dimensions: [criteria]. Present as a comparison table."
Always tell the model what type of image it is looking at and what you expect it to extract. Generic "analyze this image" prompts produce generic results. Specific extraction instructions produce structured, actionable outputs.
Document Understanding
Document understanding extends image prompting to structured and semi-structured documents like PDFs, invoices, forms, and reports. The challenge is that documents contain both visual layout information (tables, headers, formatting) and textual content that must be processed together.
PDF Analysis
For multi-page PDFs, process page-by-page with context about document structure. Ask for page-level extraction first, then cross-page synthesis to prevent losing detail on later pages.
Form Extraction
For forms and structured documents, provide a template of expected fields. The model performs better when it knows what fields to extract rather than discovering them independently.
Table Recognition
Tables in images are challenging. Explicitly ask the model to identify row and column headers first, then extract cell values. Request markdown table format for structured results.
Handwritten Content
For handwritten text or annotations, acknowledge the difficulty explicitly: "This contains handwritten notes. Transcribe what you can read and mark uncertain text with [unclear]."
Document Processing Pipeline
For complex documents, a multi-step pipeline produces the best results:
- 01
Classification
"Identify the type of document (invoice, contract, report, letter) and its key structural elements." - 02
Extraction
"Extract all [specific fields] into a structured JSON format. If a field is not present, use null." - 03
Validation
"Review your extraction. Do the numbers add up? Are dates consistent? Flag any discrepancies." - 04
Summary
"Provide a plain-language summary of the document's purpose and key action items."
Cross-Modal Reasoning
The most powerful capability of multimodal models is reasoning across modalities, combining information from text and images to reach conclusions that neither modality could support alone. This requires careful prompt design to ensure the model integrates both sources rather than relying on just one.
Cross-modal reasoning fails most often when the prompt doesn't explicitly require the model to reference both sources. Without explicit instructions, models tend to favor the text modality and may ignore or underutilize visual information.
Text + Image Synthesis
Provide text context that complements the image, then ask questions that require both. For example, provide a product specification as text and a product photo as image, then ask the model to verify claims.
"The text says [claim]. The image shows [context]. Does the visual evidence support or contradict the text? Cite specific visual details."
Multi-Image Reasoning
When providing multiple images, label each one and ask the model to reason about relationships, changes over time, or differences between them.
"Image A shows [before state]. Image B shows [after state]. Identify all changes between A and B. Classify each change as cosmetic, structural, or functional."
Grounding Text Claims in Visual Evidence
Use images as evidence to verify or challenge text-based claims. This pattern is particularly useful for fact-checking, quality assurance, and compliance verification.
"Based on the provided text and image, verify each claim. For each claim, state whether the image supports it, contradicts it, or provides no relevant evidence. Quote the specific visual element."
The most common mistake in multimodal prompting is not explicitly requiring the model to use all provided inputs. Always include instructions like "Reference both the text and the image in your answer" to ensure true cross-modal reasoning.
Activities
Activity 1
30-40 min · Intermediate
Image Type Analysis
Take three different image types (chart, screenshot, photograph) and write optimized prompts for each. Compare output quality across models.
Activity 2
40-50 min · Advanced
Document Analysis Workflow
Build a document analysis workflow that processes a multi-page PDF, extracts key data, and outputs a structured summary.
Activity 3
35-45 min · Advanced
Cross-Modal Reasoning Challenge
Design a multimodal prompt that requires the model to reason across text context and an image to reach a conclusion.
Real-World Applications
Invoice and Receipt Extraction
Automatically extract line items, totals, dates, and vendor information from scanned invoices and receipts for accounting automation.
UI Accessibility Analysis
Analyze screenshots of web and mobile applications for accessibility issues including contrast ratios, touch target sizes, and missing labels.
Chart Interpretation
Extract data from charts and graphs in research papers, reports, and presentations to create structured datasets or summaries.
Medical Image Analysis
Combine clinical context with medical images to support diagnostic workflows, always with appropriate disclaimers and human oversight.
Video Frame Analysis
Extract key frames from video content and analyze sequences for quality control, content moderation, or training data creation.
Real Estate Assessment
Combine property photos with listing descriptions to verify claims, assess condition, and generate comprehensive property reports.