Evaluation, Testing and Optimization

What this domain tests

16% of CCAR-P — roughly 10 of the 63 scored items, the third-heaviest domain after Integration (19%) and Solution Design (17%).

Character matters more here than weight. CCAR-P is the professional-tier architect exam: the guide targets mid- to senior practitioners with 3+ years in systems architecture and 6+ months running Claude or comparable LLM systems in production. Items are written as architectural judgment, not recall. First-hand accounts from candidates who sat all four Claude exams describe CCAR-P as the least memorization-dependent of the set — scenario stems with plausible-looking distractors, where the discriminating skill is knowing which trade-off the scenario is actually forcing. That is the inverse of the Foundations exams, where parameter names and CLI flags decide questions.

So do not study this domain as a list of tool names. Study it as a set of decisions:

• Which metric answers the question the stakeholder asked?
• Which grader is trustworthy for this kind of output?
• What evidence licenses a rollout, and what evidence forces a rollback?
• Given a symptom, which layer of the stack do you open first?
• Which optimization lever pays here, and what does it cost you?

Two format notes from the exam guide: items are multiple-choice and multiple-response, and each item states how many responses to select. Read that line before answering — a "select two" item scored as one answer is a free loss. Candidate reports also mention scenario-matching items. And as of the v1.0 exam guide (July 2026), Anthropic lists no official practice exam for CCAR-P among its preparation resources — only the sample items in the guide itself — which makes disciplined self-testing against the blueprint the main calibration available.

Theory outline

4.1 Define evaluation metrics (accuracy, latency, cost, safety, security)

The guide names five metric families in one breath. That grouping is the point: an architect who reports only accuracy has not evaluated the system, because a system that is accurate, slow, expensive, and unsafe fails in production.

Three principles the exam rewards:

Metrics must be tied to a decision. A useful metric has an owner, a threshold, and a consequence when the threshold is crossed. "We track hallucination rate" is not a control; "hallucination rate above X on the finance slice blocks the release" is. When a stem describes a team drowning in dashboards while a bad change ships anyway, the answer is almost always about gates and thresholds, not about adding another chart.

Percentiles, not averages, for latency. An average latency hides the tail that produces every complaint. SLA conversations are p95/p99 conversations. Distinguish time-to-first-token (what a streaming user perceives) from time-to-completion (what a batch pipeline cares about) — the same system can be excellent on one and terrible on the other, and choosing which to promise is an architectural decision, not a measurement detail.

Cost per outcome beats cost per call. A cheaper model that needs three retries and a human fix-up is not cheaper. Normalize to the business unit: cost per resolved ticket, per reviewed document, per generated report.

Anti-pattern: a single blended "quality score" that averages accuracy, safety, and latency into one number. It makes trade-offs invisible — exactly the thing you are being certified to reason about — and lets a safety regression be offset by a latency win.

4.2 Design evaluation datasets and test frameworks using mixed methodologies

"Mixed methodologies" is the guide's own wording, and it is the load-bearing idea: no single grading method covers an LLM system.

The three graders

The mature pattern is a pyramid: programmatic checks on everything, LLM-judge on a large sample, human review on a small stratified sample that also serves to validate the judge. If the LLM judge disagrees with humans on the calibration sample, the judge is the thing that needs fixing before you trust any of its verdicts.

LLM-judge discipline — the difference between a signal and self-congratulation:

• Do not use the same model, prompt, and context that produced the output. A judge sharing the generator's assumptions confirms them.
• Give the judge a rubric with concrete criteria and a scale, not "rate this 1-10."
• Give the judge the reference material (source documents, expected answer) when faithfulness is what you are measuring — a judge without the sources is guessing about grounding.
• Prefer pairwise comparison when you are choosing between two candidate systems; absolute scores drift, relative preferences are more stable.
• Validate the judge against human labels before you let it gate anything.

Dataset design

A golden set is curated, labeled, versioned, and held out — it never appears in prompts, few-shot blocks, or fine-tuning data. Leakage is the silent killer: an eval example that also lives in the system prompt's examples turns your eval into a memorization check.

Compose it deliberately:

Size it by discriminating power, not by round numbers: the set is big enough when the smallest slice you would block a release over can move by more than run-to-run variance. If a slice has six examples, a single flip swings it 17% and you cannot tell a regression from noise.

Anti-patterns: eval sets built only from cases the system already handles ("our accuracy is 98%" on a set curated from successes); a suite that never grows after launch; production incidents fixed without adding a test; one flat accuracy number with no slices.

4.3 Conduct A/B testing and iterative improvements

This objective covers the whole change-delivery path, and the exam frames it as a rollout-discipline question.

The staged path

1. Offline suite — run the candidate against the golden set. Cheap, fast, no user exposure. Necessary, never sufficient: it only contains distributions you thought to include.
2. Pre-deploy gate — the release is blocked unless the suite clears defined thresholds on defined slices, and the regression subset shows no drop. This is where "we measure things" becomes "we control things."
3. Canary — route a small share of live traffic to the candidate, hold the rest on the incumbent, and watch quality, latency, cost, and error rate side by side. Choose the share by how quickly you accumulate enough events to detect a meaningful move, balanced against how many users a bad change would touch — not by a habit number.
4. A/B test — a controlled comparison with randomized assignment when you need a statistically defensible answer about which variant is better on a metric that matters (resolution rate, escalation rate, task completion).
5. Full rollout with the rollback path still live and the previous version still deployable.

Change hygiene

Change one variable at a time. If you swap the model tier and rewrite the prompt in the same release and quality moves, you have learned nothing about which change did it — and you cannot revert the harmful half. When a stem describes ambiguous results after a release, look for the option that isolates variables.

Version everything that shapes behavior: prompts, model identifier and version, tool definitions, retrieval index build, chunking parameters, temperature and sampling settings. An architecture where a prompt lives as an untracked string in application code has no rollback target — during an incident, "revert to what we had" is not a runnable command. This ties directly to change management in the governance domain.

Statistical honesty, without memorizing thresholds

You are not being tested on formulas. You are being tested on whether you know that a difference needs enough events to be distinguishable from noise, that peeking repeatedly and stopping at the first favorable moment inflates false positives, and that a metric moving in a canary with a handful of samples is not evidence. The exam-safe formulation: define the metric, the minimum effect worth acting on, and the stopping rule before the test runs.

Anti-patterns: offline-green treated as a deploy decision; canary with no defined rollback trigger; comparing a new variant against last month's numbers instead of a concurrent control (traffic mix changes underneath you); running the experiment until it says what you wanted.

4.4 Diagnose system issues (prompt failure, hallucinations, model mismatch)

The diagnostic discipline for LLM systems is the same one that works for distributed systems, applied to a stack with an unusual middle layer.

Step one is always the same question: what changed immediately before the symptom appeared, and does the symptom's signature match that layer? Model version, prompt text, retrieval index, tool schema, upstream data source, traffic mix, dependency version. A symptom that begins the hour a document set was re-indexed points at retrieval, not at sampling parameters.

Symptom-to-layer mapping

Hallucination is not one failure mode. Separate the cases, because the fixes differ:

• Ungrounded generation — the system had no source and invented one. Fix: retrieval, or an explicit instruction and pathway for "I don't know."
• Retrieved-but-ignored — the right chunk was in context and the answer contradicts it. Fix: prompt structure, source-citation requirements, faithfulness grading.
• Retrieved-wrong — the retrieval returned plausible but incorrect material and the model faithfully summarized it. Fix: chunking, indexing, re-ranking, freshness. The model is behaving correctly here; blaming the model wastes a cycle.
• Over-extrapolation — the source supports a weaker claim than the output makes. Fix: rubric-graded faithfulness plus prompt constraints on inference beyond sources.

The published CCAR-P sample items include a Domain 4 diagnostic of exactly this shape, and the reasoning it rewards is worth internalizing: when model version and latency are unchanged and the trigger was a content refresh, the answer is the layer the change touched — not a sampling knob, and not a speculative claim that model weights silently changed.

Anti-patterns: raising temperature or "trying a bigger model" as the first response to any quality complaint; debugging from a user's paraphrase instead of the actual request, retrieved context, and raw response; fixing the individual failing case in the prompt without adding it to the eval set, so it silently returns.

4.5 Optimize token usage, latency, and cost-performance trade-offs

Optimization is a search over levers with a measurement after each one. The exam tests whether you pull them in a sane order and whether you can name what each one costs.

Lever order — cheapest and least risky first:

1. Remove waste. Redundant instructions, entire documents pasted when a section suffices, unbounded conversation history, verbose tool schemas, retrieved chunks that never get used. This costs nothing in quality and is almost always available in a system nobody has profiled.
2. Restructure for reuse. Put stable content — system prompt, policy documents, tool definitions, long reference material — at the front, and keep the variable user content at the end. A stable prefix is what makes caching possible; interleaving variable content into the middle of otherwise static context destroys the reusable prefix. This is a cost and time-to-first-token lever, measurable as a before/after on the same workload. (Mechanics belong to the Models, Prompting and Context domain; here you own measuring whether it paid.)
3. Right-size the model tier. Route by task difficulty rather than using one tier everywhere: a cheaper, faster tier for classification, extraction, and routing; a stronger tier for multi-step reasoning, ambiguous synthesis, and high-stakes output. Escalate on a signal — low confidence, validation failure, or a complexity classifier — rather than downgrading globally and hoping.
4. Constrain output. Output tokens usually cost more than input tokens and dominate completion latency. Asking for a structured, bounded answer instead of an essay is often the single largest latency win available.
5. Restructure the execution. Stream to improve perceived latency; parallelize independent subtasks instead of chaining them; move non-interactive work to asynchronous or batch processing; cache full responses for genuinely repeated queries.

The trade-off table you should be able to reconstruct

Capacity and performance testing. Optimization claims that were measured on a quiet system do not survive contact with production. Load-test at expected peak and beyond: measure p95/p99 under concurrency, not single-request timings; find where throughput limits bind and confirm the system degrades gracefully rather than cascading — queueing and backpressure, timeouts with sensible fallbacks, retries with exponential backoff and jitter so a transient limit does not become a retry storm. Establish the headroom between normal load and the point where latency SLAs break, and know what the fallback path is when you exceed it (smaller model, cached response, queued handling, honest degradation message).

Anti-pattern: optimizing cost without a quality gate. Every cost reduction must be validated against the eval suite, or you have not saved money — you have moved the failure somewhere it is not measured.

4.6 Monitor system performance using logging and observability tools

Evaluation is what you do before shipping; monitoring is evaluation that never stops.

Three planes, and the gap that keeps appearing

Teams instrument the first two because they inherit that from ordinary service monitoring, and skip the third because there is no exception to catch when the model is fluent and wrong. A stem describing "all dashboards green, quality complaints rising" is pointing at the missing quality plane.

What a trace must contain to be debuggable. A single request should be reconstructable end to end: request ID correlating every hop, model and prompt version, the retrieval query and which chunks were returned, tool calls with arguments and results, token counts, latency per stage, stop reason, and the final output. Logging only the input and the final answer leaves you unable to tell which layer failed — the diagnostic table in 4.4 becomes unusable without the middle. Handle sensitive fields at capture time; observability stores frequently have looser access and retention than the primary datastore, which is a governance concern as much as an engineering one.

Drift detection. LLM systems degrade without any code change, which is why "we didn't deploy anything" is not a defence:

Alerting that produces action. Alert on rates and trends against a baseline, not on individual events; route each alert to an owner with a runbook; keep the alert set small enough that people still read it. An alert with no defined response is a notification, and it trains the team to ignore the channel.

Closing the loop. Production is where your best eval data comes from. Sample real traffic, review failures, add them to the golden set as permanent regression cases, and re-run. A system whose eval set is frozen at launch is measuring an increasingly historical version of its own problem.

Worked problems

Problem 1. A team ships a prompt revision and a switch from a smaller model tier to a larger one in the same release. Quality scores rise 6 points, latency rises 40%, and cost nearly doubles. Leadership asks whether the latency and cost increase was worth it. What should the architect do first?

A. Accept the result — quality improved, which was the goal B. Roll back the model change and keep the prompt change, since the prompt is cheaper C. Re-run the comparison as separate experiments, isolating the prompt change and the model change against the same baseline D. Raise the latency SLA to accommodate the new numbers

Approach: The stem hands you two variables and one outcome. Nothing in the data attributes the 6-point gain, so no cost/benefit answer is available yet — and that is what was asked. The architect's move is to make the result attributable: same golden set, same traffic, one variable each.

Why the others fail: A answers a different question and ships an unexplained cost increase permanently. B guesses which change mattered — if the prompt revision produced most of the gain, that is a great outcome, and if it produced none, you have just degraded quality for no reason; guessing is exactly what the experiment exists to avoid. D changes the promise to fit the measurement, which is a decision that can only be made after you know what you are buying.

Answer: C

Problem 2. An internal code-search assistant starts telling engineers to call helper functions that were deleted from the repository months ago, complete with plausible signatures. Model identifier, model version, prompt, and latency are all unchanged. The behavior began the day the repository index was rebuilt under a new chunking configuration. Which is the most productive first investigation?

A. Send more retrieved chunks per query so the assistant has additional material to work from B. Examine what the retriever actually returns for the failing queries, and what the rebuild changed — chunking boundaries, the embedding configuration, and whether the indexed snapshot reflects current repository state C. Add a second pass in which the model reviews and verifies its own answer before returning it D. Pin the model version so provider-side updates cannot change behavior

Approach: Apply the change-correlation rule from 4.4. Exactly one thing moved — the index build — and the symptom signature matches that layer: the generator is fluently and faithfully describing material it was handed, while every generation-side metric sits still. A rebuild can reintroduce stale content or fragment definitions across chunk boundaries so deleted code surfaces as current. You cannot judge the model's behavior until you see the context it received, so retrieving the actual returned chunks for a failing query is step one.

Why the others fail: A increases the volume of a retrieval stream that is returning wrong material — more stale chunks, same wrong answer, higher cost. C asks a model that has no independent knowledge of the repository to check itself against the same bad context; self-verification cannot detect that its source was stale, and it adds latency while manufacturing confidence. D is a sound practice in general and a real control against silent provider-side drift, but the stem already tells you the model version did not move, so it fixes a cause that has been ruled out.

Answer: B

Problem 3 (select two). A team evaluates a customer-facing summarization feature using an LLM judge. The judge is the same model that generates the summaries, prompted with "Rate this summary from 1 to 10." Judge scores average 8.7, while support tickets about inaccurate summaries are climbing. Which two changes most improve the trustworthiness of this evaluation?

A. Increase the number of summaries the judge scores each day B. Give the judge the source document and a rubric with explicit faithfulness criteria, and score against those criteria rather than a bare 1-10 scale C. Use a different model or a differently-constructed judge prompt from the one that produced the summaries, and validate it against a human-labeled sample D. Raise the passing threshold from 7 to 9 E. Replace the LLM judge entirely with exact-match string comparison against reference summaries

Approach: Two independent defects are described. First, the judge cannot assess faithfulness because it never sees the source and has no criteria — a bare numeric scale invites the fluency-equals-quality bias. Second, judge and generator are the same model with shared assumptions, and no human calibration exists to reveal the gap. B fixes the first, C fixes the second. Note the "select two" instruction; the exam guide states every item tells you how many to pick.

Why the others fail: A scales an untrustworthy signal — more of a biased measurement is more bias. D moves a threshold on a number that does not mean what it claims; the scale itself is the problem. E swaps a flawed method for an unusable one: exact match cannot grade open-ended summaries, where many correct outputs share no wording with the reference. This is the mixed-methodology point — programmatic checks belong on structural constraints, not on prose quality.

Answer: B and C

Problem 4. A production agent meets its quality bar but costs more per request than the business case supports. Profiling shows an 8,000-token policy document and tool definitions resent on every request, unbounded conversation history, and verbose free-text responses where the consumer only needs five structured fields. Which sequence does the most good with the least quality risk?

A. Switch to the smallest available model tier immediately, then re-measure quality B. Trim wasted context and history, restructure so the stable policy and tool definitions form a reusable prefix, constrain the output to the five required fields — then evaluate whether a smaller tier still meets the bar C. Cache complete responses keyed on the user's message text D. Truncate the policy document to its first quarter

Approach: The profile names pure waste — content that is resent, history that is never pruned, and output nobody consumes. Those cost nothing in quality to remove, and output tokens typically dominate both cost and completion latency, so constraining the response is the largest single lever here. Tier changes are the one lever that trades quality for money, so it goes last and only against the eval suite.

Why the others fail: A starts with the only risky lever and does it blind. C fails on the traffic shape: agent requests carrying conversational state are rarely repeated verbatim, so hit rates are near zero, and stale cached answers to state-dependent questions are a correctness hazard. D discards policy content the answers require — that is not optimization, it is an undetected accuracy regression.

Answer: B

Problem 5. A RAG assistant's dashboards show stable p95 latency, a 0.2% error rate, and flat cost per request. Meanwhile, escalations to human agents have doubled over six weeks with no deployments in that window. What is the most likely gap, and the right response?

A. The monitoring stack is broken and should be replaced B. Nothing is wrong — no deployment means no regression C. There is no output-quality plane in the monitoring; add continuous sampled quality grading against a fixed rubric, plus retrieval-quality and input-distribution monitoring to identify which drift is responsible D. Add more infrastructure alerts at tighter latency thresholds

Approach: Every green signal is a system-health signal. A fluent, wrong answer returns HTTP 200 at normal latency and normal token count, so this failure is invisible to all three dashboards by construction. The doubled escalation rate is the only quality signal present, and it arrived through the support queue rather than through instrumentation. The absence of a deploy is a clue, not an alibi: corpus growth, input drift, and index staleness all degrade quality with no code change.

Why the others fail: A misdiagnoses — the stack is reporting accurately on what it measures. B is the trap the whole drift section exists to defeat. D adds resolution to the planes that are already healthy and none to the plane that is failing.

Answer: C

Exam traps

• Offline-green treated as a deploy decision. The suite only contains distributions you thought to include. Any option that ships on offline results alone loses to one that adds a pre-deploy gate plus a canary with a defined rollback trigger.
• The self-judging LLM. A judge sharing the generator's model, prompt, and blind spots confirms rather than evaluates. Watch for judges with no rubric, no source material, and no human calibration sample.
• Aggregate accuracy hiding a slice collapse. When a stem mentions one segment, language, region, or document type failing, the answer involves per-slice measurement — not a higher overall threshold.
• Confounded releases. Prompt change plus model change plus index change in one deploy makes any result unattributable and any rollback all-or-nothing. Isolating variables is almost always the correct architectural answer when results are ambiguous.
• Sampling knobs offered as fixes for grounding failures. Temperature, top-p, and "a bigger model" do not repair wrong retrieved context. Match the symptom to the layer that changed.
• Eval-set leakage. Golden-set examples appearing in system prompts or few-shot blocks turn evaluation into a memorization check. Held out means held out.
• Frozen eval sets. A suite that never absorbs production failures measures a launch-day version of the problem. Every incident should become a permanent regression case.
• Optimizing cost without a quality gate. Downsizing the tier first is the seductive wrong answer. Order is: remove waste, restructure for reuse, constrain output, then right-size the model against the eval suite.
• Averages where percentiles were promised. Latency SLAs live at p95/p99; time-to-first-token and time-to-completion are different promises, and streaming improves only the former.
• Monitoring with no quality plane. Green health and cost dashboards alongside rising complaints is a scripted scenario. Fluent-and-wrong returns a 200.
• "We didn't deploy anything." Corpus drift, input drift, index staleness, and upstream model version changes all degrade quality with no code change. Pin versions where you can and re-run the suite when they move.
• Logging only inputs and final outputs. Without retrieved chunks, tool calls and arguments, prompt version, token counts, and per-stage latency, you cannot localize a failure to a layer — the trace has to carry the middle.
• Alerts nobody owns. An alert without a threshold, an owner, and a runbook is a notification, and a noisy channel gets muted before the important one fires.