SAP-C03 practice question: Pipeline security scanning, signing and AI evaluation (2 of 2)
Operational Excellence and Automation A free question from our original SAP-C03 practice exam. Try it, then open the answer to see why each option is right or wrong.
A fashion retailer's customer-service assistant is built on Amazon Bedrock. Every week, prompt and configuration changes are released by the content team, and a recent change made the assistant reveal internal discount rules to customers.
The release team wants the pipeline to test each prompt change against a fixed set of golden questions, measure answer quality and safety, and block the release if the scores fall below agreed thresholds. The test must run automatically in the pipeline, with no servers to manage.
Which solution meets these requirements?
Choose one answer.
AA. Have a reviewer read a random sample of the assistant's answers after each release, and roll back the change if the reviewer finds a problem.
BB. Add a pipeline stage that runs a Bedrock evaluation job on the golden questions for each prompt change and fails the stage when a quality or safety score falls below its threshold.
CC. Track the assistant's response latency, token counts and error rates on a CloudWatch dashboard for each release, and alert the team when latency rises above its usual level for that hour.
DD. Run the unit tests for the Lambda functions that call the model, and skip the prompt changes because they are configuration rather than code.
Show the answer and explanation
Answer: B
A. Wrong: A random sample after release catches problems only after customers see them, and it does not block a release. The option is not an automated gate.
B. Right: Correct. The evaluation job runs automatically in the pipeline and compares the scores to the agreed thresholds, and the failed stage stops the release. Bedrock runs the evaluation, so the team manages no servers.
C. Wrong: Latency and token counts show performance and cost, not whether answers reveal internal rules. The option does not evaluate answer quality or safety.
D. Wrong: The unit tests cover the code that calls the model and not the answers the prompt produces. Prompt changes are the risk that caused the incident.
Explanation
A Bedrock evaluation stage in the pipeline tests each prompt change against golden questions and fails the release when quality or safety falls below threshold. Sampling after release, latency dashboards and code-only tests each leave the bad prompt reaching customers.
Want more on Pipeline security scanning, signing and AI evaluation? The 5 full-length practice sets have 7 more Pipeline security scanning, signing and AI evaluation questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →