Prompting With Images as Input
Vision prompting that ships: screenshots to bug reports, charts to data, photos to JSON — grounded and few-shot.
Turn a screenshot into a structured bug report
Vision models (Claude, GPT-5, Gemini 3) read UI screenshots well. Give a role, the image, and an output contract so you get a filable bug, not a paragraph of description.
You are a QA engineer. The attached screenshot shows a bug in our web app. Produce a bug report as JSON: { "title": "short, specific", "observed": "what is wrong, referencing visible UI elements", "expected": "what should happen", "location": "the screen or component visible", "severity": "low | medium | high" } Only describe what is visible. If something is ambiguous, say so in 'observed' rather than guessing.VerifyYou get a filable report that references actual on-screen elements (button labels, error text), not a generic description.Extract a chart into a data table
Paste a chart image and ask for the underlying numbers in a structured format. Tell it exactly how to handle values it must estimate rather than read.
Extract the data from this chart into JSON. For each series, give its label and an array of {x, y} points read from the axes. If a value is not exactly labeled, estimate from the gridlines and mark it "approx": true. Also return the axis titles and units. Do not invent series that aren't shown.VerifyYou get labeled series with numeric points, units, and approx flags — spot-check two points against the image to confirm accuracy.Turn a photo into structured JSON
Photographs of documents or objects — receipts, labels, whiteboards — become data with the same contract-first approach. Specify the fields and the null policy explicitly.
This is a photo of a receipt. Extract to JSON: {merchant, date (ISO 8601), currency, line_items: [{name, qty, unit_price}], subtotal, tax, total}. Use null for anything unreadable or missing. Do not calculate values that aren't printed; transcribe only what you can read. If the total is illegible, set it to null.VerifyFields match the receipt, unreadable values are null (not guessed), and totals aren't back-computed from line items.Ground it: cite the visible, flag the uncertain
The main failure mode is confident description of things that aren't there. Instruct the model to distinguish what it can see from what it is inferring, and to rate legibility per field.
Before answering, list what is clearly legible in the image versus what is blurry, cropped, or ambiguous. Base your extraction only on the legible parts, and add a "confidence" value (low/medium/high) for each uncertain field. Never fill an unreadable field with a plausible guess.VerifyBlurry or cropped values come back flagged low-confidence or null, and the model names what it couldn't read.Teach format with multimodal few-shot
When you need a consistent style of answer across many images, show one or two solved image-to-output examples before the real image. The model generalizes the pattern the same way text few-shot works.
Here are two examples of how to caption product photos for our catalog: [IMAGE A] -> {"title": "Matte Black Ceramic Mug, 12oz", "attributes": ["matte", "black", "ceramic", "12oz"], "tone": "minimal"} [IMAGE B] -> {"title": "Hand-Woven Rattan Basket, Medium", "attributes": ["handwoven", "rattan", "natural", "medium"], "tone": "artisanal"} Now caption this image in the same JSON format: [REAL IMAGE]VerifyThe new caption matches the demonstrated structure and tone, not the model's default verbose description.