Automated tests and test reports in pipelines: DOP-C02 study notes and practice questions
What to know about Automated tests and test reports in pipelines for the DOP-C02 exam, from the SDLC Automation domain (22% of the exam guide), then 2 free questions to test it. Part of the DOP-C02 study guide.
Testing questions ask where a test belongs in a pipeline and how its result should stop a bad release.
Test types and where they run
Test
Stage
Unit tests, linting, static analysis (SAST)
Build stage, on every commit
Dependency and secret scanning
Build stage
Integration and API tests
After deploying to a test environment
Image scanning
After pushing to a staging repository, before promotion
Dynamic security tests (DAST), load tests
Staging environment
Synthetic canaries, smoke tests
After production deployment, with alarms
CodeBuild reports
Declare reports in the buildspec: JUnit XML, NUnit, TestNG, Cucumber and Visual Studio TRX for tests; JaCoCo, SimpleCov, Clover and Cobertura for code coverage.
The console shows per-test results and trends with no custom parsing.
The test command’s exit code fails the build; reports don’t fail it on their own.
Gates
Amazon Inspector scans ECR images (enhanced scanning) and Lambda functions; a step waits for the scan and fails on critical findings.
Amazon CodeGuru Security and third-party tools for SAST.
Pipeline stage conditions can block a stage when an alarm is in ALARM.
Manual approval for exploratory checks.
After deployment
CloudWatch Synthetics canaries test user journeys and APIs on a schedule.
CodeDeploy hooks (such as AfterAllowTestTraffic or ValidateService) run checks and fail the deployment to trigger rollback.
AWS Fault Injection Service experiments test resilience.
Common traps
Scanning images only in production.
Parsing test output with custom scripts when CodeBuild reports do it.
A test step that always exits 0.
Running the scan before the push to the staging repository, so the scanned image may not be the one promoted.
A game studio's matchmaking service has unit tests, integration tests against a staging database and a load test. All of them run as stages of an AWS CodePipeline pipeline that builds the service with CodeBuild.
Developers read the test output in the build logs. A failing test has sometimes still left the build green, because one script step ignores the exit code of the test command. The release lead wants a failed test to stop the pipeline, and wants each run to show per-test results and code coverage in the CodeBuild console.
The team does not want to write a log parser or run a separate reporting service.
Which change meets these requirements with the LEAST operational overhead?
Choose one answer.
AA. Add a reports section to the buildspec that publishes the JUnit XML test output and a coverage report, and make the test command exit non-zero when a test fails.
BB. Pipe the test output to a Lambda function that searches for the word FAILED, writes a summary to Amazon S3, and sends an SNS message to the release lead when it finds a match.
CC. Run the tests in a separate CodeBuild project that sits outside the pipeline, so that failures do not block deployments, and review its logs by hand before each release.
DD. Set every build command to continue after errors, and export the test counts as pipeline variables that a manual approval action displays to the release lead.
Show the answer and explanation
Answer: A
A. Right: Correct. CodeBuild reads JUnit XML and coverage files from the reports section and shows them in the console. A test command that exits non-zero fails the build, and the pipeline stops before the next stage. Learn more: Test reports in AWS CodeBuild (opens in a new tab)
C. Wrong: Tests outside the pipeline cannot stop a release, which is the outcome the release lead asked to prevent. Reviewing logs by hand is slow and easy to skip.
When a buildspec declares reports, CodeBuild turns JUnit XML and coverage output into per-test results in the console without custom parsing. A test command that exits non-zero fails the build, and the pipeline stops before later stages run. Log parsers, separate projects and ignored errors each add work or let failures through.
A video streaming service runs an AWS CodePipeline pipeline for its playback API. The pipeline runs unit tests, scans the container image for vulnerabilities, runs a dynamic application security test against a staging endpoint and runs a load test.
The security team wants image vulnerabilities found before any image reaches production, so that a critical finding stops the release. The dynamic security test needs the staging endpoint to be running, so it must run after the staging deployment.
The load test is slow, so it should run only for pushes to the release branch and never hold up pull request feedback. The engineers want managed AWS services wherever they can do the job.
Which THREE actions meet these requirements? (Choose three.)
Choose three answers.
AA. Push the image to a staging repository with ECR scanning on push, add a pipeline step that waits for the scan to complete and fails on any critical finding, and promote the image only after that step passes.
BB. Run the load test on every pull request in a CodeBuild project with a 30-minute timeout, so that performance regressions are caught in review before the code is merged.
CC. Add a CodeBuild action after the staging deployment that runs the dynamic security scanner against the staging endpoint, and fails the stage when the scanner reports a high-severity issue.
DD. Run the security test before the build stage, against a development endpoint that is not yet deployed, so that findings appear on pull requests before the code is merged.
EE. Create a separate CodeBuild project for the load test with a webhook filter group that matches only pushes to the release branch, so that pull requests never start the test.
FF. Run the load test as the last stage of the pipeline on every commit, and skip the production deployment whenever the load test runs longer than the release window.
Show the answer and explanation
Answers: A, C, E
A. Right: Correct. ECR starts a scan when the image is pushed, but the scan runs after the push returns, so the gate step waits for the scan to complete before reading the findings. Failing the step on a critical finding stops the image from reaching production. Learn more: Image scanning in Amazon ECR (opens in a new tab)
D. Wrong: The endpoint does not exist until staging is deployed, so the test cannot run against it. The team would also need to stand up a second environment just to host a scan.
E. Right: Correct. A webhook filter on the head reference limits the load test to release-branch pushes, which keeps pull request feedback fast. CodeBuild runs the test with no extra service to manage. Learn more: GitHub webhook events (opens in a new tab)
F. Wrong: Running the test on every commit still holds up each change, and skipping deployments on a slow test would block releases for reasons unrelated to the code. The release team asked for a test limited to release branches.
Explanation
Image scanning belongs after the image is pushed to a staging repository and before promotion, with a gate that waits for the scan to finish so a critical finding blocks the release. A dynamic security test needs a running endpoint, so it runs after the staging deployment and fails its stage on high-severity findings. The slow load test runs only for release-branch pushes through a CodeBuild webhook filter, which keeps pull request feedback fast.
Want more on Automated tests and test reports in pipelines? The 5 full-length practice sets have 8 more Automated tests and test reports in pipelines questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Automated tests and test reports in pipelines