Course contents
Run an Effort Sweep
Measure quality, tokens and latency at three effort levels on your own test cases, then ship the lowest level that holds.
Effort levels aren't interchangeable between models, and Anthropic says so with each release: Sonnet 5.5's levels are “recalibrated,” Opus 5.5's level names “don't correspond to the same amount of thinking” as on Opus 5, and for Fable 5.2 you should “rerun your effort sweep rather than carrying over a setting.” The only way to know which level your task needs is to measure it. An effort sweep runs the same test cases at several levels and records quality, tokens and latency side by side. First, see why the token column matters.
Move output tokens per request to see what a higher effort level can do to the bill. Thinking is generated output, billed even when you don't display it.
Move the controls and watch what changes.
Move output from 1,000 to 4,000 tokens per request, the kind of jump a higher effort level can cause on hard tasks. What happens to Claude Opus 5.5's monthly bill?
What to record
| Measure | Why it matters |
|---|---|
| Pass rate on your test cases | The quality bar each level has to clear |
| Output tokens per request, thinking included | Most of the cost difference between levels |
| Time to first token and total time | What users feel; Anthropic notes that higher effort means a longer wait before the reply starts |
Runs that hit max_tokens | Truncated answers. Anthropic says to treat them as failures even when the text holds valid JSON |
| Cost per completed task | A cheaper level that needs retries isn't cheaper |
Complete the request settings for the low-effort run of your sweep on Claude.
Freeze a test slice
Pick 20 cases from your golden set, including at least five hard ones, each with a pass rule you can check without debate. The Testing Prompts Like Code lesson shows how to build that set. Freeze the slice so every level sees exactly the same inputs.
promptsweep-slice.json (20 cases, frozen 2026-11-12) - 15 typical cases from last month's traffic - 5 hard cases: two past failures, one multi-step calculation, one ambiguous request, one long input - each case: { id, input, pass_rule }VerifyYou have 20 cases, five of them hard, and you can say for each one what counts as a pass.Strip the reasoning incantations
Remove “think step by step”, scratchpad sections,
reasoningfields and any line telling the model to think less. They muddy the comparison, and on current Claude models some of them can trigger reasoning-extraction declines. Keep the task, the context and the output contract.promptDelete: "Think step by step." Delete: "Write your reasoning in <scratchpad> tags first." Delete: "reasoning": "..." (from the JSON schema) Keep: the task, the context, the success criteria, the output contractVerifyNothing in the prompt mentions thinking or reasoning, and the output contract is unchanged.Set effort explicitly and leave headroom
Name the level on every request so no default decides your result, and set
max_tokenshigh enough for thinking plus the reply. The request below is Claude's shape. With OpenAI, setreasoning.effort; with Gemini,thinkingLevel; with Grok,reasoning_effort.json{ "model": "claude-opus-5-5", "max_tokens": 32000, "thinking": {"type": "adaptive", "display": "summarized"}, "output_config": {"effort": "low"}, "messages": [{"role": "user", "content": "{test_case_input}"}] }VerifyEach logged request shows its effort level, and no response ends withstop_reason: "max_tokens".Run three levels, several times each
Run the slice at three adjacent levels, such as low, medium and high, three times per case. Outputs vary from run to run, so a case passes only if it passes every run. Record the five measures from the table above.
promptlevel | pass (all 3 runs) | mean output tokens | p50 first token | p50 total | hit max_tokens low | 16 / 20 | 1,150 | 1.4 s | 6 s | 0 medium | 19 / 20 | 2,300 | 2.9 s | 11 s | 0 high | 19 / 20 | 4,700 | 5.8 s | 21 s | 0 (illustrative numbers: record your own)VerifyYou have a filled-in table with one row per level, measured on the same 20 cases.Read the summaries of the failures
For every failed case, read the thinking summary. A model that misread the task needs a prompt fix, and more effort won't help it. A model that understood the task but reasoned shallowly is the case effort exists for. Fix misreads first, then rerun.
promptCase t14 at low: summary shows the model treated "net" as "gross" -> prompt fix: define net in the task statement; rerun the slice Case t17 at low and medium: summary stops after one scenario -> depth problem: route this kind of case to highVerifyEvery failure is labeled either a prompt problem or a depth problem, and the prompt problems are fixed before you choose a level.Ship the lowest level that holds, and schedule a rerun
Choose the lowest level whose pass rate meets your bar. If one kind of case needs more, route that kind higher instead of raising the level for everything. Write the result down with the model ID and date, and rerun the sweep whenever the model changes.
prompt# effort decision: invoice-analysis route # model: claude-opus-5-5 swept: 2026-11-12 slice: sweep-slice.json # chosen: medium (19/20; high adds no passes at twice the tokens) # exception: multi-scenario forecasts route to high # rerun: on any model change, or monthlyVerifyYour route has a documented effort level, an exception rule if you need one, and a date for the next sweep.
Plain-English definitions from the AI Coding Dictionary.