Pipeline security scanning, signing and AI evaluation: SAP-C03 study notes and practice questions
What to know about Pipeline security scanning, signing and AI evaluation for the SAP-C03 exam, from the Operational Excellence and Automation domain (20% of the exam guide), then 2 free questions to test it. Part of the SAP-C03 study guide.
SAP-C03 asks how to stop vulnerable or untrusted artifacts from reaching production with the least manual effort, and how to check that a generative AI application still answers well before it ships.
Image and package scanning
ECR basic scanning
ECR enhanced scanning
Engine
AWS native, CVE database
Amazon Inspector
Finds
Operating system package vulnerabilities
OS and programming language packages
When
Scan on push or manual
Scan on push and continuous rescans as new CVEs appear
Amazon Inspector also scans EC2 instances and Lambda functions continuously, and can scan images inside a CodeBuild or other CI job before they’re pushed, so the build fails early.
Inspector findings flow to AWS Security Hub CSPM and EventBridge, where a rule can open a ticket or block promotion.
Use Inspector SBOM export when the scenario asks for a software bill of materials.
Signing and verifying artifacts
AWS Signer with the Notation client signs container images stored in ECR; a verifier (for example a Kubernetes admission controller on EKS) rejects unsigned images at deploy time.
Lambda code signing: a code signing configuration names the allowed Signer profiles; set to Enforce, it rejects unsigned or tampered packages (Warn only logs).
ECR immutable tags stop a signed tag being overwritten with different content.
Evaluating AI applications
Amazon Bedrock Evaluations runs evaluation jobs: automatic metrics, human reviewers, or LLM-as-a-judge scoring such as correctness, completeness, faithfulness and harmfulness.
RAG evaluation scores a Knowledge Base on retrieval alone or on retrieve-and-generate.
You can evaluate responses you bring yourself, so models hosted outside Bedrock can be compared too.
Put the evaluation in the pipeline: run it against a fixed prompt dataset on every prompt, model or retrieval change, and fail the stage below a score threshold. Bedrock Guardrails then filter content at run time (see responsible AI).
Where to put each gate
Build: dependency and image scan in CodeBuild; fail on critical findings.
Push: ECR enhanced scanning on push; sign the image.
Pre-deploy: verify signature; evaluate AI quality; manual approval for production.
Runtime: continuous Inspector rescans; EventBridge alerts for new critical CVEs in running images.
How to choose
Scenario wording
Choose
“Find vulnerabilities in Python and Java libraries inside images”
ECR enhanced scanning (Inspector)
“Alert when a new CVE affects an image already deployed”
Enhanced continuous scanning plus EventBridge rule
“Only code from our pipeline may run in Lambda”
Lambda code signing in Enforce mode with Signer
“EKS must reject images that weren’t signed”
Signer and Notation signatures checked by an admission controller
“Check a prompt change didn’t reduce answer quality”
Bedrock Evaluations (LLM-as-a-judge) as a pipeline gate
Common traps
Basic scanning when language packages or continuous rescanning are required.
Scanning only at push time and never again, so new CVEs in old images go unseen.
Lambda code signing in Warn mode when the requirement is to block.
CodeGuru, which is out of scope for SAP-C03.
Manual spot checks of AI answers instead of a repeatable evaluation dataset and threshold.
A news publisher builds its container images with AWS CodeBuild and stores them in Amazon ECR. Its security team found that images are scanned only once, when pushed, and a critical package vulnerability published a week later went unnoticed in a deployed image.
The security team wants images scanned continuously after they are pushed, with a deployment blocked when an image has a critical finding in its operating system or application packages. The team wants no new scanning servers to manage.
Which solution meets these requirements?
Choose one answer.
AA. Enable Amazon ECR enhanced scanning, which uses Amazon Inspector to rescan images continuously, and add a pipeline step that blocks the deployment when an image has a critical finding.
BB. Keep basic scanning on the repositories, and have an engineer review the scan report for each image once a month before the next release.
CC. Run an open-source scanner on a fleet of EC2 instances that pulls every image from the registry each night, and have the scanner's results written to a shared report for the security team.
DD. Add a scan step to the build that fails only when the build itself errors, and let the deploy stage run whatever image the build produced.
Show the answer and explanation
Answer: A
A. Right: Correct. Enhanced scanning through Inspector rescans images as new vulnerabilities are published, and the pipeline check blocks any image with a critical finding. AWS runs the scanning, so no servers need managing.
B. Wrong: Basic scanning runs at push time and does not rescan for newly published vulnerabilities. A monthly manual review is too slow to catch a week-old critical finding.
C. Wrong: The fleet has to be built, patched and scaled by the team, which the security team wants to avoid. A shared file also does not block a deployment on its own.
D. Wrong: A build error does not reflect the vulnerabilities in the packages, so critical findings still reach production. The option does not block anything on the scan result.
Explanation
Enhanced scanning through Inspector rescans images after they are pushed, and a pipeline check on the findings blocks deployments with critical issues without any scanning servers. Basic push-time scans, monthly reviews, self-run scanners and build-only checks each miss the continuous or blocking requirement.
A fashion retailer's customer-service assistant is built on Amazon Bedrock. Every week, prompt and configuration changes are released by the content team, and a recent change made the assistant reveal internal discount rules to customers.
The release team wants the pipeline to test each prompt change against a fixed set of golden questions, measure answer quality and safety, and block the release if the scores fall below agreed thresholds. The test must run automatically in the pipeline, with no servers to manage.
Which solution meets these requirements?
Choose one answer.
AA. Have a reviewer read a random sample of the assistant's answers after each release, and roll back the change if the reviewer finds a problem.
BB. Add a pipeline stage that runs a Bedrock evaluation job on the golden questions for each prompt change and fails the stage when a quality or safety score falls below its threshold.
CC. Track the assistant's response latency, token counts and error rates on a CloudWatch dashboard for each release, and alert the team when latency rises above its usual level for that hour.
DD. Run the unit tests for the Lambda functions that call the model, and skip the prompt changes because they are configuration rather than code.
Show the answer and explanation
Answer: B
A. Wrong: A random sample after release catches problems only after customers see them, and it does not block a release. The option is not an automated gate.
B. Right: Correct. The evaluation job runs automatically in the pipeline and compares the scores to the agreed thresholds, and the failed stage stops the release. Bedrock runs the evaluation, so the team manages no servers.
C. Wrong: Latency and token counts show performance and cost, not whether answers reveal internal rules. The option does not evaluate answer quality or safety.
D. Wrong: The unit tests cover the code that calls the model and not the answers the prompt produces. Prompt changes are the risk that caused the incident.
Explanation
A Bedrock evaluation stage in the pipeline tests each prompt change against golden questions and fails the release when quality or safety falls below threshold. Sampling after release, latency dashboards and code-only tests each leave the bad prompt reaching customers.
Want more on Pipeline security scanning, signing and AI evaluation? The 5 full-length practice sets have 7 more Pipeline security scanning, signing and AI evaluation questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Pipeline security scanning, signing and AI evaluation