What this domain tests
19% — the heaviest domain on CCAR-P, roughly 12 of 63 scored items. It is also the domain where the Professional exam's character is clearest, and that character is the opposite of the Foundations exams.
First-hand accounts of CCAR-P describe it as architectural-judgment heavy with comparatively obvious distractors — one experienced architect who sat all four called it the easiest of the set and said the difficulty levels feel "the wrong way around" versus Foundations. That is not an invitation to skip preparation; it is a statement about what kind of preparation pays. CCAR-F rewards recall of CLI flags and parameter names. CCAR-P rewards a working decision framework: given a scenario with a stated constraint, which integration choice serves that constraint, and which options are merely adjacent good practices.
Two format notes that matter for this domain specifically:
- The exam mixes multiple-choice and multiple-response items, and each item states how many responses to select. Expect "Select 2" on integration items, because integration questions naturally have a control and a mechanism (remove the tool and scope the credential). Scenario-matching styles also appear.
- No official practice exam exists for CCAR-P. The exam guide's three sample items are the only published specimens of item style, and one of them is a Domain 3 least-privilege item — a fair signal of how central capability scoping is here.
The recurring cognitive move: the stem names a constraint (a latency SLA, a compliance boundary, an audit finding, a corpus that doesn't fit), and three of the four options are real engineering practices that simply don't address that constraint. You are being tested on whether you can tell "good idea" from "the idea that resolves this."
Theory outline
1. Capability bloat: tool and agent configuration (Task 3.1)
An agent's tool list is its attack surface and its decision space at the same time. Bloat hurts twice, and the second harm is the one architects under-weight.
| Harm | Mechanism | Symptom in the scenario |
|---|---|---|
| Security | Every exposed tool is a capability the model can be argued into invoking | Destructive action taken during an ordinary request; audit finding |
| Accuracy | Overlapping tool descriptions compete for the same intent | Model picks the wrong tool, or thrashes between two similar ones |
| Cost and latency | Tool definitions occupy context on every request | Slower first token, higher per-call cost, no functional benefit |
The exam's core principle: least privilege means removing the capability, not observing it. If a support workflow only ever needs to read tickets and draft replies, then refund and account-deletion tools should not be in that agent's configuration at all. The seductive wrong answers are always the compensating controls:
- Adding logging so misuse can be audited later — detective, not preventive. The damage has occurred.
- Adding a confirmation prompt before the dangerous action — guarding the privilege rather than removing it, and confirmation fatigue is real.
- Using a more capable model that "follows instructions more reliably" — model size is orthogonal to authorization scope. Nothing about a bigger model makes an unnecessary tool necessary.
Removal is the answer when the capability is genuinely not required by the role. Compensating controls are correct only for capabilities that are required but are high-consequence — that is where confirmation gates, thresholds, and human-in-the-loop belong (Domain 5 territory).
Designing against bloat:
- One agent, one bounded role. If a config needs four unrelated tool families, that is usually a signal to split into specialized agents with narrow tool sets rather than one omnibus agent.
- Separate read-scope from write-scope agents. A read-only research agent and a narrowly-scoped action agent is a stronger topology than one agent holding both.
- Distinguish tool descriptions sharply. Two tools whose descriptions both plausibly answer "look up the customer" will be confused; merge them or make the boundary explicit in the descriptions.
- Prune by observed usage. Tools that telemetry shows are never selected are pure cost and pure risk.
2. Authentication and authorization: finding the security gap (Task 3.2)
The exam expects you to keep two words apart under scenario pressure:
- Authentication — establishing who is making the request (API keys, OAuth tokens, workload identity, mTLS).
- Authorization — determining what that established identity is permitted to do (scopes, roles, row-level policy, per-tenant isolation).
Most ARCP security-gap items are authorization failures wearing authentication clothing. The archetype:
An agent integrates with an internal system using a single service account with broad permissions. Individual users interact with the agent. Each user sees only their own data in the legacy UI.
The gap: the agent's credential has become the effective permission set for every user. Authentication succeeded — the service account is genuinely who it says it is — and authorization collapsed, because the downstream system can no longer distinguish user A's request from user B's. Any user who can phrase a request can reach any row the service account can reach. This is the confused-deputy problem, and it is the single most examinable integration security pattern.
The correct shape: propagate end-user identity through the agent to the downstream system, so the downstream system enforces its own per-user authorization. Delegated-identity flows (on-behalf-of token exchange, per-user tokens, or scoped short-lived credentials minted per session) preserve the existing permission model instead of flattening it.
| Gap in the stem | What it actually is | Correct control |
|---|---|---|
| One shared service account for all users | Authorization collapse / confused deputy | Propagate end-user identity; downstream enforces per-user policy |
| Long-lived static key in the agent config or repo | Credential lifetime and custody | Short-lived credentials from a secret manager; rotation; never in source |
| Agent can call an admin endpoint it never needs | Excess privilege | Remove the tool; scope the credential to the minimum |
| Tool arguments assembled from untrusted user text | Injection into a privileged action | Validate and constrain arguments outside the model; parameterize |
| Third-party tool server given the same credential as first-party | Blast radius across trust boundary | Distinct, narrowly-scoped credentials per integration |
Trust boundaries are the framing device. Every hop — user to agent, agent to model provider, agent to internal tool, agent to third-party MCP server — is a boundary at which identity should be re-established and privilege re-narrowed. An architecture where one credential crosses three boundaries is the one the exam wants you to flag.
Multi-tenancy note: in a shared-tenant deployment, isolation must be enforced at the data layer (per-tenant filters applied outside the model, in the retrieval query), never by instructing the model to only discuss the current tenant's data. Prompt-level isolation is a preference; query-level isolation is a guarantee.
3. Accuracy-latency trade-offs, and justifying them (Task 3.3)
The task statement's second half is the exam-relevant half: justify configuration decisions. Items give you a stated business constraint and ask which configuration serves it — the wrong answers are usually optimizations for a constraint the scenario never mentioned.
Work the budget in this order:
- Fix the hard constraint. An interactive assistant with a stated p95 response-time target, a nightly batch with a completion window, a call-centre assist that must land inside a pause in conversation — the SLA is a wall, not a preference.
- Spend the remaining headroom on accuracy using the levers below, cheapest and most reversible first.
- Measure, don't assume. Which lever actually moved quality is a Domain 4 question; this domain expects you to know what each lever costs.
| Lever | Effect on accuracy | Effect on latency | Effect on cost |
|---|---|---|---|
| Larger / more capable model tier | Up on hard reasoning | Up | Up |
| Extended thinking / more reasoning effort | Up on multi-step problems | Up, sometimes sharply | Up (thinking tokens) |
| More retrieved context (higher top-k) | Up until relevance dilutes | Up (bigger prompt) | Up |
| Reranking retrieved candidates | Up (precision) | Up modestly | Up modestly |
| Verification or critique second pass | Up on correctness-critical work | Roughly doubles the path | Roughly doubles |
| Prompt caching on a stable prefix | Neutral | Down (time to first token) | Down on repeated prefix |
| Streaming responses | Neutral | Down on perceived latency only | Neutral |
| Smaller model for a narrow subtask | Down if misapplied | Down | Down |
| Asynchronous batch processing | Neutral | Up (by design) | Down per unit |
Four judgments the exam leans on:
Prompt caching is the free-lunch answer when the stem shows a large repeated prefix. If every request carries the same long system prompt, policy document, or schema followed by a short varying user message, the move is to order the stable content first and the volatile content last, then enable caching. It improves latency and cost simultaneously without discarding required context. The distractors sacrifice something real: truncating the policy loses needed instructions; blindly downsizing the model risks quality; relocating the document into a few-shot block does not create a reusable cacheable prefix. Note the ordering discipline — a cache prefix ends at the first byte that changes, so a single volatile token near the top (a timestamp, a user name) destroys the cacheability of everything after it.
Streaming fixes perception, not throughput. Total tokens and total time are unchanged; time-to-first-visible-token improves. Correct when the complaint is "it feels slow" for a human reader; wrong when a downstream system needs the complete structured output before it can proceed.
Batching trades latency for unit economics. Asynchronous batch processing is discounted relative to synchronous calls and is the right answer for nightly enrichment, backfills, bulk classification, and evaluation runs. It is the wrong answer for anything with a human or a synchronous caller waiting.
Parallelism reduces wall-clock, not work. Independent retrievals or independent tool calls issued concurrently shorten the critical path; steps with real data dependencies cannot be parallelized away, and pretending otherwise is a common distractor.
Route by difficulty rather than downgrading globally. When cost pressure meets a quality floor, the architecturally literate answer is tiering — a smaller, faster model handles the high-volume simple slice, with escalation to a stronger tier on complexity signals or low confidence — rather than moving every request to the cheapest model and eating the quality loss. (Model-tier selection criteria themselves live in Domain 2; here you are choosing the routing architecture.)
4. Observability at scale (Task 3.4)
The task statement says analyze observability challenges — plural, at scale. The challenges are specific to LLM systems and differ from ordinary service monitoring:
- Non-determinism. The same input can produce different outputs, so "reproduce the bug" is not a reliable first step. You need the recorded inputs, not just a repro recipe.
- Multi-stage attribution. A bad answer could originate in retrieval, in tool execution, in the prompt assembly, or in the model's reasoning. Logging only the final response makes these indistinguishable.
- Quality is not a status code. A wrong answer returns HTTP 200. Traditional error-rate dashboards report perfect health while the system is failing its users.
- Volume and sensitivity. Full prompt/response capture at scale is expensive and often contains regulated data, so retention and redaction policy is part of the design, not an afterthought.
What a competent design captures:
| Layer | What to record | What it lets you answer |
|---|---|---|
| Request envelope | Correlation / trace ID, tenant, user identity, model and version, config (temperature, thinking budget) | "Which population and which config regressed?" |
| Retrieval | Query issued, chunk IDs returned, scores, index version | "Did retrieval bring back the right material?" |
| Tools | Tool selected, arguments, latency, success/failure, retries | "Did it pick the right tool and call it correctly?" |
| Model call | Input/output token counts, stop reason, latency, cache hit on the prefix | "Where is cost and time actually going?" |
| Outcome | Final response, user-visible errors, feedback signal, escalation flag | "Did the user get what they needed?" |
The load-bearing principle: one correlation ID stitches every stage of a single logical request together, including nested tool and subagent calls. Without end-to-end tracing, a multi-stage failure requires guesswork; with it, you can localize the failure to a stage in one query. This is the answer whenever a stem describes an intermittent quality problem in a multi-step pipeline that nobody can localize.
Sampling and cardinality at scale. You cannot afford full-fidelity capture of everything forever. The practical design is: aggregate metrics on every request (latency, tokens, cost, error rate, cache hit rate, tool selection distribution), full traces on a sample plus 100% of anomalies — errors, escalations, negative feedback, and outliers on latency or token count. Retain redacted traces longer than raw ones.
Alert on distributions, not single events. Meaningful LLM alerts fire on shifts: retrieval hit-rate dropping, refusal or fallback rate climbing, tool-error rate spiking, token-per-request creeping up, cache hit rate collapsing (usually the sign that someone put a volatile field at the top of the prompt). A single odd response is noise; a moved distribution is a regression.
(How you score quality — LLM-as-judge, golden datasets, regression suites — is Domain 4. Domain 3 owns the plumbing that makes scoring possible.)
5. RAG pipeline design: chunking and indexing (Task 3.5)
Know the pipeline as two paths, because failures live in whichever path the stem just changed:
Ingestion path: source acquisition → parsing and normalization → chunking → enrichment with metadata → embedding → index write → index versioning.
Query path: query understanding and expansion → retrieval (vector, keyword, or hybrid) → optional metadata filtering → optional reranking → context assembly and ordering → generation with citations.
Chunking is the decision the exam probes most. The trade-off is fixed and worth stating as a rule:
| Chunk size | Retrieval precision | Answer completeness | Failure mode |
|---|---|---|---|
| Too small | High — matches are tight | Poor — the answer is split across chunks | Model gets fragments; a procedure loses its steps |
| Too large | Low — one embedding averages many topics | Good if the right chunk is found | Relevant passage diluted by surrounding noise; wasted context |
| Right-sized on semantic boundaries | Good | Good | — |
Design rules that generalize:
- Chunk on structure, not on character counts. Sections, headings, articles, clauses, function definitions, table rows. A splitter that cuts mid-sentence or mid-table produces chunks that are individually meaningless.
- Overlap adjacent chunks so a passage straddling a boundary is fully present in at least one chunk. Overlap costs storage; it buys recall.
- Attach metadata at ingestion: source document, section path, effective date, version, tenant, access-control labels. Metadata is what makes filtering, permission enforcement, freshness policy, and citation possible later — retrofitting it means re-ingesting.
- Carry context into the chunk. A bare paragraph loses its parent heading. Prepending the document title and section path to the chunk text (or storing it as retrievable metadata) is what keeps "Section 4.2: Termination" from being retrieved as an anonymous paragraph about notice periods.
- Version the index and re-embed on model change. Embeddings from two different embedding models are not comparable. A partial re-embed leaves an index where similarity scores are meaningless across the boundary.
- Plan for updates and deletes, not just initial load. Documents change and are withdrawn; an index that only appends will confidently serve retracted policy.
Failure triage, which the exam guide's own sample question models: when a RAG system starts returning confident but incorrect answers after a content or index change, while model version and latency are unchanged, the first place to look is retrieval — a broken or partial re-index, mismatched embeddings, changed chunking, or stale entries. The model is doing its job faithfully on bad material. Options that blame model weights, temperature, or a "shrunken context window" are not things a document refresh can cause. Generalize the reasoning: ask what changed, then follow the change, rather than reaching for the most familiar knob.
Grounding controls belong in the design too: instruct for citation from retrieved context, provide an explicit "the provided material does not contain this" path, and prefer returning nothing over improvising when retrieval comes back empty. A pipeline with no empty-retrieval path is a hallucination generator.
6. Matching retrieval strategy to data shape and query pattern (Task 3.6)
Task 3.5 builds the pipeline; 3.6 asks whether you chose the right retrieval mechanism for the material. Vector search is not the default answer — that is the trap the objective exists to catch.
| Data shape / query pattern | Right strategy | Why vectors alone fail |
|---|---|---|
| Prose, policy, docs; paraphrased conceptual questions | Dense vector (semantic) search | — |
| Exact identifiers: SKUs, error codes, ticket numbers, names | Keyword / lexical (or hybrid) | Embeddings blur rare tokens; near-miss IDs score similarly |
| Mixed corpus, unpredictable phrasing | Hybrid (vector + keyword, fused, then reranked) | Either alone has a systematic blind spot |
| Structured records, aggregations, "how many", "sum by region" | Query the database (text-to-SQL or a parameterized tool) | Retrieval returns rows; it cannot compute an aggregate correctly |
| Highly interlinked entities, multi-hop relationships | Graph or entity-centric retrieval | Similarity does not traverse relationships |
| Time-sensitive material ("current policy") | Metadata filter on effective date, then rank | Nearest neighbours are indifferent to recency |
| Access-controlled corpus | Permission filter applied in the query, pre-ranking | Post-filtering leaks existence and empties result sets |
| Small, stable, entirely relevant corpus | No retrieval — put it in the prompt and cache it | Retrieval adds infrastructure and a failure mode for no gain |
Three refinements worth knowing by name:
- Reranking: retrieve generously (high recall), then use a cross-encoder or model-based reranker to order precisely, then pass only the top few to generation. This is the standard fix for "the right chunk is retrieved but ranked eighth."
- Query transformation: rewriting or decomposing the user's question before retrieval. A multi-part question ("compare the 2024 and 2025 policies on X") retrieves far better as two targeted queries than as one blended embedding.
- Filter before you rank, not after. Permission and tenancy filters must constrain the candidate set inside the query. Post-filtering a ranked list can return an empty page for a user who does have relevant accessible material.
The exam-level judgment: read the stem for the shape of the data and the shape of the question, then pick. Aggregation questions over structured data are a database's job. Exact-identifier lookups are lexical. Conceptual questions over prose are vectors. Unpredictable mixtures are hybrid.
7. Connection protocols and integration mechanisms (Task 3.7)
Four mechanisms, and the exam wants the fit, not the feature list.
| Mechanism | Best fit | Signals in the stem |
|---|---|---|
| MCP server | Reusable tool/data integrations shared across multiple clients and teams; standard capability discovery | "Several teams need the same connector", "we keep rewriting this integration", "expose our internal systems to any assistant" |
| Direct API integration | Embedding Claude inside a product path where you control the request/response contract | "Our application calls Claude for X", latency-sensitive product surface, custom orchestration |
| CLI / local tooling | Developer workstations, CI pipelines, scripted automation over a repo or filesystem | "Automate this in CI", "engineers running it locally", batch scripts |
| Agent-to-agent delegation | One system's agent handing bounded work to another team's or vendor's agent across an organizational or trust boundary | "Another department already owns that capability", cross-org workflow, separate ownership and lifecycle |
Judgment notes:
- Build MCP when reuse is the point; build a direct API integration when the integration is one product's private path. Standing up a server for one consumer that will never have another is over-engineering — and the exam does test recognizing over-engineering.
- Third-party MCP servers are a trust boundary. Treat them as untrusted code with their own credentials, their own scope, and their own audit trail. Tool descriptions and returned content from an external server are inputs to the model, which makes them an injection surface.
- Agent-to-agent is an ownership decision as much as a technical one. It buys independent lifecycles and clean team boundaries; it costs you end-to-end observability and error semantics unless correlation IDs and failure contracts cross the boundary with the request.
- Deployment surface is a related integration decision: Claude is available directly from Anthropic's API and through managed cloud offerings (Amazon Bedrock, Google Cloud Vertex AI). Choose on organizational constraints, not preference — existing cloud commitments and procurement, data-residency and compliance posture, identity integration with the cloud's IAM, and how quickly new models and features need to be available on that surface. The architectural point for the exam: the surface changes auth, networking, and governance, and should not change your application's abstraction over the model.
8. Progressive discovery vs. monolithic context (Task 3.8)
This objective is the context-strategy counterpart to capability bloat, and it generalizes past tools to any large body of material.
Monolithic context: load everything the agent might need into the prompt up front. Progressive discovery: expose a lightweight index or listing, and let the agent fetch details on demand.
| Monolithic | Progressive discovery | |
|---|---|---|
| Best when | Corpus is small, stable, and almost entirely relevant per request | Corpus is large; only a small unpredictable slice matters per request |
| Cost profile | Pays for everything on every request — but a stable prefix is highly cacheable | Pays for a small index plus the fetches actually needed |
| Latency | One round trip | Extra round trips per fetch |
| Accuracy risk | Dilution — the relevant passage competes with everything else | Discovery failure — the agent never fetches the piece it needed |
| Operational risk | Context ceiling as the corpus grows | More moving parts, more to trace |
The decision rule: relevance density and stability. If nearly all of a small, rarely-changing body of material is relevant to nearly every request, monolithic-plus-caching is simpler and often cheaper in practice. As soon as the corpus is large and per-request relevance is a thin unpredictable slice, progressive discovery wins — and its round-trip cost is the price of not drowning the model.
Design guidance for progressive discovery: make the index descriptive enough to choose from (names alone are not enough — the agent needs to know what each item contains), keep the fetch granularity aligned with how the material is actually used, and instrument discovery so you can see when the agent failed to fetch something it needed. That last signal is the one teams forget to log, and it looks identical to a reasoning failure without it.
Worked problems
Problem 1. An internal HR assistant is deployed to all employees. It integrates with the HRIS through a single service account that can read every employee record, and the assistant's system prompt instructs it to only discuss the requesting employee's own data. Compliance flags the design. What is the security gap, and what fixes it? (Select 2)
A. Authentication is broken — the service account credential is not verified by the HRIS B. Authorization is collapsed — every user's request reaches the HRIS with the same broad permissions, so prompt instructions are the only thing separating employees' data C. Propagate the requesting employee's identity to the HRIS so the HRIS enforces its own per-user access policy D. Move the "only discuss your own data" instruction to the top of the system prompt and repeat it in every turn E. Switch to a more capable model tier so the instruction is followed more reliably
Approach: Separate the two words. The credential authenticates fine — the HRIS knows exactly who is calling. What has been lost is per-user authorization: the agent is a confused deputy holding permissions no individual user has. A prompt is a preference, not a boundary. The fix must move enforcement back into the system that owns the data, which means carrying the end user's identity across the hop.
Why the others fail: A misnames the failure; nothing about the service account's authentication is broken, which is precisely why the gap is easy to miss in review. D and E are the two canonical distractors for every prompt-as-a-security-control scenario — strengthening or better-following an instruction reduces the failure rate and never eliminates it, and model capability has nothing to do with authorization scope.
Answer: B and C
Problem 2. A contract-analysis assistant answers questions over 40,000 agreements. Chunks are produced by splitting each document every 4,000 characters. Users report that answers about termination clauses are frequently incomplete — the notice period is right but the cure period is missing. Retrieval scores look healthy. What is the most likely cause?
A. The embedding model is too small for legal text B. Fixed-length splitting cuts clauses across chunk boundaries, so retrieval returns a fragment of the clause C. The top-k is too high and irrelevant chunks are diluting the answer D. The index needs to be rebuilt on a faster vector store
Approach: Read the symptom precisely. Answers are partially correct — a piece of the clause is present and an adjacent piece is missing. That is the fingerprint of boundary damage, not of ranking or of model capability. A character-count splitter has no idea where a clause starts or ends. The fix is chunking on semantic boundaries (clause and section structure) with overlap, plus carrying the section path into the chunk.
Why the others fail: A would degrade relevance broadly, but retrieval scores are healthy and the retrieved material is on-topic. C predicts noisy or off-topic answers, not systematically truncated ones — and lowering top-k would make the missing half less likely to appear, not more. D changes infrastructure performance and touches nothing about correctness.
Answer: B
Problem 3. A logistics team asks their Claude-based assistant questions like "how many shipments to the Nordics were delayed more than 48 hours last quarter, by carrier?" The data lives in a warehouse of structured shipment records. The current design embeds each record as text and retrieves the top 20 by similarity. Counts come back wrong. What should the architect change?
A. Increase top-k from 20 to 200 so more records are included B. Add a reranker so the most relevant shipment records rank first C. Replace similarity retrieval with a parameterized query against the warehouse — generate SQL or call a tool that computes the aggregate — and have the model explain the result D. Chunk each record more finely so individual fields embed better
Approach: Match the strategy to the data shape and query pattern. This is an aggregation over structured records, and no retrieval strategy computes an aggregate — retrieval returns a sample of rows and the model then estimates from a sample, which is exactly why the counts are wrong. Structured data with quantitative questions belongs in a database query; the model's job is translating the question and interpreting the answer.
Why the others fail: A makes the sample bigger and the answer no more correct — the model is still counting a retrieved subset, and now more slowly and expensively. B improves ordering, which is irrelevant when the question needs all matching rows. D is chunking work applied to a problem that is not a chunking problem.
Answer: C
Problem 4. Every request to a customer-facing assistant sends the same 6,000-token system prompt and product-policy document, then a per-request line with the current timestamp and session ID, then the user's short message. Cost per request is high and time-to-first-token is slow. Which two changes most directly address both? (Select 2)
A. Move the timestamp and session ID after the policy document so the stable content forms an unbroken prefix B. Enable prompt caching on the stable prefix C. Truncate the policy document to its first 1,000 tokens D. Stream the response so users see output sooner E. Switch to the cheapest available model tier
Approach: The stem shows a large repeated prefix, which is the caching signature. But caching only pays if the prefix is actually stable — a timestamp sitting between the system prompt and the policy document changes on every request and breaks the cacheable region at that byte. The two moves compose: reorder so stable content is contiguous at the head, then cache it. Both cost and latency improve, and nothing required is discarded.
Why the others fail: C buys savings by deleting policy the assistant needs — that is a correctness regression sold as an optimization. D improves perceived latency only, and does nothing for cost. E trades a quality risk the scenario never authorized for a saving that caching provides without the risk.
Answer: A and B
Problem 5. A support agent has 18 tools, including "issue_refund" and "close_account". Analysis shows tier-1 agents never legitimately need either, and that the model occasionally selects "close_account" when a customer says "I want to close this ticket". The team proposes adding a confirmation dialog before both actions. Evaluate.
A. Correct — a confirmation dialog prevents the unwanted action B. Insufficient — remove both tools from the tier-1 agent's configuration and route the rare legitimate cases to a separate, narrowly-scoped agent or a human C. Correct, provided the dialog text is clear enough D. Better to add logging so misuse can be reviewed weekly
Approach: Two independent problems, one fix. The security problem is excess privilege: a role that never needs a capability is holding it. The accuracy problem is description collision — "close_account" is competing for the intent expressed by "close this ticket". Removing the tools resolves both at once, because a tool absent from the configuration can neither be misused nor mis-selected. Confirmation dialogs address neither: they guard a privilege that should not exist, and confirmation fatigue makes them decay.
Why the others fail: A and C are compensating controls, appropriate only for capabilities genuinely required by the role. D is detective, not preventive — the account is already closed by the time anyone reads the log.
Answer: B
Problem 6. A multi-stage assistant (retrieval, then two tool calls, then generation) intermittently produces answers users report as wrong. Error rates and HTTP status codes are clean. The team logs only the final response text. What should be added first?
A. A larger model tier to reduce reasoning errors B. End-to-end tracing with one correlation ID per request, capturing the retrieval query and returned chunk IDs, each tool call's arguments and result, and model/config metadata C. A daily report counting how many responses contained the word "sorry" D. Full capture of every prompt and response, retained indefinitely
Approach: The team cannot currently distinguish a retrieval failure from a tool failure from a reasoning failure, because only the last stage is recorded — and in an LLM system a wrong answer returns a clean 200, so conventional monitoring reports health. The first move is always to make the pipeline attributable: one trace ID stitching every stage, with the inputs to each stage recorded, not just the output of the last one.
Why the others fail: A is a fix chosen before the diagnosis exists; if retrieval is returning stale chunks, no model tier repairs it. C is a crude proxy metric that measures tone, not correctness. D is directionally right about capture but wrong about design at scale — unbounded full-fidelity retention is expensive and a regulated-data liability; the practical pattern is aggregate metrics on everything, sampled traces plus 100% of anomalies, with redaction and retention policy.
Answer: B
Exam traps
- Compensating controls offered instead of removal. Logging, confirmation dialogs, and "a better model will follow instructions" are all wrong when the capability simply is not needed by the role. Least privilege removes; it does not observe.
- Authentication named when authorization is the failure. A single broad service account authenticates perfectly and destroys per-user access control. Look for one credential serving many users — that is a confused deputy, and the fix is propagating end-user identity.
- Prompt-level isolation for tenancy or permissions. "Instruct the model to only discuss the current tenant" is never the answer. Filters belong in the retrieval query, applied before ranking.
- Vector search as the reflex. Aggregations belong in a database, exact identifiers belong in lexical search, multi-hop relationships want graph traversal, and a small stable corpus may want no retrieval at all.
- Blaming the model when the change was in the data. Confident-but-wrong answers after a document refresh or re-index, with model and latency unchanged, point at retrieval. Follow what changed.
- Volatile content above stable content. A timestamp or user ID placed before a long system prompt destroys the cacheable prefix. Stable first, dynamic last — and reordering is often half of the correct multiple-response answer.
- Streaming sold as a latency fix. It improves perceived time-to-first-token only. It does not reduce total latency, total tokens, or cost, and it does not help a downstream system that needs the complete output.
- Batch processing for interactive paths. Asynchronous batching lowers unit cost by raising latency. Correct for backfills and nightly jobs; wrong whenever someone is waiting.
- Raising top-k as a quality fix. Past the point of relevance, more context dilutes rather than informs, and it costs latency and money. Reranking, query rewriting, or better chunking are usually the intended answers.
- Monitoring that only captures outputs. Without the retrieval query, chunk IDs, and tool arguments, you cannot attribute a failure to a stage. Also watch for options that propose unbounded full capture — at scale the answer is aggregate metrics plus sampling plus all anomalies, with redaction.
- Standing up an MCP server for a single private consumer. MCP earns its keep on reuse across clients and teams. One product path calling Claude directly is not under-engineered.
- Treating a third-party MCP server as trusted. It is a separate trust boundary with its own credentials and its own injection surface, including the content it returns.
- Monolithic context by default because it is simpler. Large corpus with thin per-request relevance means progressive discovery. Small, stable, wholly-relevant material means monolithic and cached. Relevance density decides.
- Ignoring the stated constraint. Multiple options will be genuinely good practice. Only one addresses the SLA, the audit finding, or the corpus size the stem actually gave you.