What this domain tests
2.6% of the exam — roughly 1 to 2 items out of 53, and the lightest domain on the CCDV-F blueprint. Calibrate: Applications and Integration is 33.1%, Model Selection and Optimization 16.8%. Neither of those can be skipped; this one can be secured in half an hour and left alone.
Read the blueprint carefully, because the domain title is misleading. The guide states that "each domain below lists the skills against which exam items are written," and Domain 4 lists exactly one skill: Debugging and Error Handling (2.6%). Its four named competencies are:
- Error type identification
- Recovery strategy selection
- Trace analysis to identify failure modes
- Problem origin isolation between the integration layer and model output
Eval design — building test sets, grading, regression suites — is not among them. It appears instead in the guide's Section 2 list of what a credential holder can demonstrate: designing and running evals, debugging failure modes through trace analysis, validating structured output, and monitoring production quality. So evals are fair game as exam content, but the weight of Domain 4 sits on debugging. Study it in that proportion.
Minimum viable mastery: given a symptom, say whether the fault lies in the integration layer or in the model output, and name the recovery that fits. Everything else in this module supports that one skill, because the fourth competency reads like a question stem already written.
Theory outline
1. Error type identification
Two different error surfaces exist and candidates conflate them constantly.
Surface one: the request never produced a good generation. These arrive as HTTP errors with a typed error body.
| Status | Error type | Cause | Deterministic? |
|---|---|---|---|
| 400 | invalid_request_error | Malformed request — bad schema, invalid parameter, unsupported combination | Yes |
| 401 | authentication_error | Missing or invalid API key | Yes |
| 403 | permission_error | Key lacks access to the resource | Yes |
| 404 | not_found_error | Resource does not exist | Yes |
| 413 | request_too_large | Payload exceeds the size limit | Yes |
| 429 | rate_limit_error | Rate or token limits exceeded | No — transient |
| 500 | api_error | Server-side failure | No — transient |
| 529 | overloaded_error | Service temporarily overloaded | No — transient |
The column that matters is the last one. Deterministic errors are your bug; retrying them produces the identical failure and burns budget. Transient errors are capacity or infrastructure; retrying them is the correct and expected response.
Surface two: the call succeeded and the model's behavior is the problem. HTTP 200, valid response object, unusable result. Here the diagnostic field is stop_reason:
| stop_reason | Meaning | What it usually means for your bug |
|---|---|---|
| end_turn | Claude finished naturally | The generation is complete; any problem is content quality |
| tool_use | Claude is requesting tool execution | Your loop must execute and continue — stopping here truncates the task |
| max_tokens | Output hit the token ceiling | Truncation. The classic cause of "the JSON won't parse" |
| stop_sequence | A configured stop sequence matched | Check whether your stop sequence is firing earlier than intended |
| pause_turn | The turn paused and can be resumed | A long agentic turn; resume it rather than treating it as completion |
| refusal | The model declined for safety reasons | Not an outage and not a bug — a policy outcome, with structured detail attached |
The exam-relevant discipline: check stop_reason before reading content. A response object exists in all six cases; only in the first does reading the content straightforwardly make sense.
2. Problem origin isolation: integration layer versus model output
This is the domain's centerpiece and the phrase most likely to appear in a stem. The question is always: did the system fail, or did the model produce something my system could not use?
| Symptom | Likely origin | The test that localizes it |
|---|---|---|
| Requests fail immediately with a 4xx | Integration — request construction or credentials | Replay the exact request payload by hand; a hand-built minimal request that also fails confirms the request, not the model |
| Intermittent failures under load, fine at low volume | Integration — rate limits or concurrency | Correlate failures with 429s and request volume |
| Response arrives but content is empty or truncated | Model output | Read stop_reason: max_tokens means truncation; refusal means a policy decline |
| JSON parse errors on a fraction of responses | Ambiguous — resolve with stop_reason | max_tokens means truncation (raise the ceiling); end_turn with malformed JSON means the model produced bad structure (tighten schema, prompt, or validation) |
| Client timeouts on long generations | Integration — timeout configuration | Stream the response; if streaming succeeds, the generation was fine and the client gave up early |
| The agent calls the wrong tool | Model output, caused by integration input | The tool descriptions and schemas you supplied are the model's only guide — the fix is in the definitions |
| Output quality degraded after a deploy with no code change | Configuration — an unpinned model version or an edited prompt | Diff the prompt and the model identifier, not the application code |
Two localization habits worth internalizing. The HTTP status answers first: a non-200 means no usable generation exists, so stop looking at prompts; a 200 means the model responded, so the fault is in what it produced or in what you did with it. And reproduce outside your stack: if the same prompt and parameters sent directly produce the same bad output, the problem is the prompt or the model configuration; if they produce good output, the problem is in your application's request construction, context assembly, or response handling. That single test resolves most ambiguous cases and is frequently the correct answer choice.
The trap the guide's phrasing sets up: a refusal on HTTP 200 reads like an integration failure. Your code sees a response with no useful content, logs "empty response," and an engineer starts debugging the network layer. It was never a network problem. Guarding on stop_reason turns an hour of misdirected work into a one-line branch.
3. Recovery strategy selection
Recovery is chosen by error class, not by severity.
| Error class | Recovery |
|---|---|
| Transient (429, 5xx, connection failures, timeouts) | Retry with exponential backoff plus jitter, bounded attempts. Official SDKs do this automatically for connection errors, 408, 409, 429, and 5xx, with a configurable retry count |
| Deterministic (400, 401, 403, 404, 413) | Do not retry. Fix the request, the credentials, or the payload size |
| Capacity on a specific model | Fall back to another model. Claude Code exposes this directly as a fallback-model option for when the primary is overloaded or unavailable |
| Refusal | A policy outcome, not an error. Handle it explicitly — surface it, route it, or use a configured fallback path. Retrying the identical request is not a strategy |
| Truncation (max_tokens) | Raise the ceiling, or restructure the task so the output fits. Retrying unchanged truncates again |
| Latency-tolerant bulk work failing under rate limits | Move it to the Batches API rather than fighting the limit synchronously |
Three cross-cutting rules. Jitter is not optional — plain exponential backoff makes every client in your fleet retry at the same moment, reproducing the burst that caused the rate limit. Retries must be safe to repeat — if a request triggers a side effect (a charge, an email, a write), retrying duplicates it, so idempotency keys belong on the request path before the retry loop does. Bound everything — unbounded retries against a persistent outage turn a degraded dependency into an outage of your own, with a bill attached.
4. Trace analysis to identify failure modes
A trace is the full record of a run: system prompt, message history, tool definitions supplied, every tool_use block with its arguments, every tool_result returned, the stop_reason at each step, and per-step token usage and latency.
Read a trace forward to the first divergence, not backward from the crash. In an agentic loop, the visible failure is almost always several steps downstream of the real one. An agent that produced a nonsense final answer usually went wrong at step two — a tool returned an error, the model reasoned around it, and everything after that was confident and wrong. Backward reading finds the symptom; forward reading finds the cause.
The recurring failure modes in tool-using applications localize to three places, and knowing which is which is the whole skill:
| Failure mode | Where it lives | Fix |
|---|---|---|
| Model picks the wrong tool, or none | The tool description — it did not distinguish this tool from its neighbors | Rewrite descriptions to state when to use and when not to use each tool |
| Model supplies malformed or missing arguments | The input schema — too loose, unclear field names, no required list, no descriptions | Tighten the schema; describe each field; mark required fields |
| Model invents a result after a tool failed | The tool_result handling — the error was swallowed or returned as an empty success | Return errors as errors, with a message the model can act on |
| Answers degrade over a long session | Context bloat and drift — accumulated tool output crowding out the instructions | Prune tool output, compact, or isolate work in subagents (Domain 6) |
| Output truncated mid-structure | max_tokens | Raise the ceiling or shrink the requested output |
Instrumentation is what makes any of this possible. Log the request identifier, model identifier, prompt version, stop_reason, and token usage for every call. Without the model and prompt version recorded, "it got worse last Tuesday" is unanswerable.
5. Evals, structured-output validation, and production monitoring
From the guide's Section 2 competency list — lighter weight than the debugging material above, but part of the credential.
Why evals exist. The model is non-deterministic: the same prompt can produce different output across runs, so a single passing manual test proves nothing about reliability. An eval is the response to that — a fixed set of inputs with expected properties, run repeatedly, scored against a threshold. The unit of confidence is the set, not the case. Build that set from real failures: every production bug becomes a permanent case, and the suite grows into a map of your actual failure surface rather than a guess at it.
Run it as a regression gate. In conventional software, tests run when code changes. Here the things that change behavior are the prompt, the model version, the tool definitions, and the schema — none of which are application code. That is why the guide pairs configuration management (prompt versioning, model version pinning) with evaluation: an unpinned model identifier means behavior can change with no commit at all, and without a regression suite you will hear about it from users.
Grading depends on what a case asserts. Deterministic properties — valid JSON, a required field present, a value in an allowed set, a citation supplied — should be checked programmatically, because programmatic checks are cheap, repeatable, and unambiguous. Open-ended qualities need a rubric applied by a model acting as judge or by a human reviewer; both are slower and noisier, so the instinct the exam rewards is to convert as much of the assertion as possible into something machine-checkable before reaching for a judge.
Validating structured output. Three layers, in order: constrain generation by supplying a schema so the model targets a defined shape rather than being asked in prose for "valid JSON"; validate at the boundary by parsing and schema-checking every response before it enters your system, so unvalidated model output never reaches business logic or a database; and recover defensively by retrying with the validation error fed back to the model, rather than crashing or silently accepting a partial object. Underneath all three sits the guide's own phrasing from the Output Handling skill: skepticism toward confident output. Fluency is not accuracy.
Monitoring production quality. Evals catch regressions before release; monitoring catches drift after it. Track stop_reason distribution, error rates by type, latency and token usage, and schema-validation failure rate. A rising share of max_tokens stops or validation failures is a quality regression showing up as operational data — which is why the debugging and eval halves of this domain are one discipline seen at two timescales.
Worked problems
Problem 1. A production service calls Claude and logs "empty response from model" for about 1% of requests. The team has spent a day inspecting the HTTP client, connection pooling, and retry logic. Responses come back with HTTP 200. What should they check first?
A. Increase the client timeout and retry the failing requests B. Check stop_reason on the affected responses — a refusal returns HTTP 200 with no usable content, and max_tokens returns a truncated one C. Switch to a larger model D. Add exponential backoff with jitter
Approach: HTTP 200 is the discriminator. The transport worked and the model responded, so by definition the problem is in the model output or in how the response is handled — the entire integration layer is already exonerated, which is exactly what a day of client debugging failed to notice. The field that distinguishes "finished normally," "was truncated," and "declined" is stop_reason.
Why the others fail: A and D are recovery strategies for transient transport errors, and there are no transport errors here — a 200 arrived every time. C changes a variable with no evidence linking it to the symptom, and a larger model refuses declined content too. The general lesson: never select a recovery before the origin is isolated.
Answer: B
Problem 2. A batch job sends thousands of requests. Roughly 8% fail. The logs show a mix of 429 responses and 400 responses. The current code retries every failure three times with a fixed one-second delay. Throughput is poor and the 400s never succeed. What is the correct fix?
A. Increase the retry count from three to ten B. Separate the classes: retry the 429s with exponential backoff plus jitter, and stop retrying the 400s — fix the malformed requests instead C. Remove retries entirely and let failures surface D. Reduce the delay to 200 ms so retries complete faster
Approach: Two error classes are being handled with one policy, and the policy is wrong for both. 400 is deterministic — the request itself is malformed, so the second and third attempts fail identically and only add load. 429 is transient and retryable, but a fixed delay across a fleet of concurrent workers means every client retries in the same instant, reproducing the burst that caused the limit. Exponential backoff spreads retries over time; jitter spreads them across clients.
Why the others fail: A multiplies the wasted attempts on the deterministic failures and worsens the synchronized-retry problem. C throws away the legitimate recovery for the transient half. D makes the burst tighter, which is the opposite of what rate limiting requires.
Answer: B
Problem 3. An extraction pipeline has run cleanly for weeks. This morning, downstream schema validation starts rejecting about a third of records. No application code has been deployed. What is the most likely cause and the right first move?
A. The parsing library was auto-updated; pin its version B. The model identifier is unpinned or the prompt was edited — diff the prompt and model version, then run the eval set against both configurations to confirm C. Input documents got longer; raise max_tokens D. Add a retry loop around the extraction call
Approach: "No code deployed" narrows the field to the things that change behavior without a commit: the model version, the prompt, the tool or schema definitions. That is precisely why the guide pairs prompt versioning and model version pinning with evaluation. The eval set is the instrument that turns a suspicion into a finding — run it against the old and new configuration and the regression either reproduces or it does not.
Why the others fail: A is possible but does not explain a semantic validation failure — a library update typically breaks parsing outright, not one record in three. C is a real cause of truncation, but it is a guess made before looking at stop_reason, and truncation would show a distinct signature rather than schema-shaped rejections. D retries an operation that is failing systematically, converting a quality regression into a slower, more expensive quality regression.
Answer: B
Exam traps
- Reaching for a recovery before isolating the origin. Retries, backoff, and model swaps are all wrong answers until you know whether the failure is transport or generation. The HTTP status resolves that in one look.
- Retrying deterministic errors. 400, 401, 403, 404, and 413 fail identically every time. An option that applies backoff to a malformed request is testing whether you know the difference between "transient" and "broken."
- Reading content without checking stop_reason. A refusal is HTTP 200 with no usable content and a policy reason attached; max_tokens is HTTP 200 with a truncated one. Both masquerade as integration bugs — and if the object ends mid-string, the generation was cut off, so fix the ceiling rather than the parser.
- Backoff without jitter. Synchronized retries across concurrent clients recreate the burst that triggered the limit. If two options differ only by the word "jitter," that word is the answer.
- Reading traces backward. The step that crashed is downstream of the step that went wrong. Find the first divergence — usually a swallowed tool error the model then reasoned around.
- Fixing a wrong-tool problem in the prompt. Tool descriptions and input schemas are the model's only information about your tools. Wrong tool chosen is a description problem; malformed arguments are a schema problem.
- Treating one passing manual test as verification. Output is non-deterministic. Confidence requires a set of cases, a threshold, and repetition — and a suite built from real failures beats one invented at design time.
- Forgetting that behavior changes without a deploy. An unpinned model identifier or an edited prompt changes system behavior with no commit. Any "it broke and nothing changed" stem points at configuration, not code.