Training with SageMaker AI: MLA-C02 study notes and practice questions
What to know about Training with SageMaker AI for the MLA-C02 exam, from the ML Model and Foundation Model (FM) Development domain (24% of the exam guide), then 3 free questions to test it. Part of the MLA-C02 study guide.
You have your own training script; packages in requirements.txt install at start-up
Extend a prebuilt container
You need extra system libraries on top of a framework image
Bring your own container (BYOC)
Custom runtimes or compiled libraries; follow the contract: read from /opt/ml/input, write the model to /opt/ml/model
SageMaker HyperPod
Long-running, large-scale foundation model training on resilient clusters
How a job runs
You choose the instance type and count, the image, input channels (training, validation) from S3, EFS or FSx for Lustre, and hyperparameters.
SageMaker AI starts the instances, runs the code, uploads model.tar.gz from /opt/ml/model to S3 and stops the instances, so you pay only while it runs.
Logs go to CloudWatch Logs; metrics you print can be captured with metric definitions (regex).
Warm pools keep instances for a short time to start the next job faster.
Local mode runs the container on your machine for quick debugging.
Managed Spot training
Uses spare capacity at a large discount.
Set MaxWaitTimeInSeconds and a checkpoint location in S3; after an interruption SageMaker AI resumes from the latest checkpoint.
Best for jobs that can be restarted and aren’t urgent.
Debugging and profiling
SageMaker Debugger captures tensors and system metrics, and built-in rules flag vanishing gradients, overfitting or idle GPUs.
Common traps
Building a custom image when script mode plus requirements.txt is enough.
Spot training without checkpoints, which restarts from zero.
Running training on a notebook instance that stays on for days.
A logistics broker has a PyTorch training script that predicts freight delay risk. The data scientist wants to run it as a SageMaker AI training job, and the script needs two extra Python packages that are not in the default framework image.
The team wants to keep its own script, avoid building and maintaining a Docker image, and still use the managed PyTorch container that SageMaker AI provides.
Which approach meets these requirements?
Choose one answer.
AA. Build a custom Docker image that installs PyTorch and the two packages, push it to Amazon ECR, and point the training job at that image.
BB. Use the SageMaker AI PyTorch estimator in script mode, and list the two packages in a requirements.txt file in the source directory.
CC. Copy the script into an Amazon EC2 instance with PyTorch installed and run it there, then upload the trained model to Amazon S3 manually.
DD. Convert the PyTorch script into a built-in SageMaker AI algorithm, since built-in algorithms can run any framework code.
Show the answer and explanation
Answer: B
A. Wrong: A custom image works, but it adds the image build and maintenance the team wants to avoid. The stem asks to keep the managed container.
C. Wrong: This runs the training outside SageMaker AI, so the managed job, its logs and its model artifacts are lost. The stem asks for a SageMaker AI training job.
D. Wrong: Built-in algorithms do not run arbitrary framework scripts. The team's script would have to be rewritten to fit a built-in algorithm.
Explanation
Script mode lets a SageMaker AI framework estimator run the team's own training script in the managed container, and packages listed in requirements.txt are installed at start-up. A custom image adds build work, and running on EC2 or using a built-in algorithm loses the managed job.
An esports league trains a video highlight model every night on a GPU instance in SageMaker AI. The run takes about five hours, and the team wants to cut its training bill without losing more than a few minutes of progress when capacity is reclaimed.
The training script already saves a checkpoint every few minutes and can resume from the latest one. The model output must still land in the team's S3 bucket.
Which configuration meets these requirements?
Choose one answer.
AA. Use On-Demand instances for the job, and purchase a Compute Savings Plan that covers the account's Amazon EC2 usage.
BB. Enable managed Spot training without a checkpoint location, so that an interrupted job restarts from the first epoch.
CC. Enable managed Spot training, and configure a checkpoint location in S3 so that the job restores from its last checkpoint after an interruption.
DD. Halve the number of epochs to shorten each run, and keep the job on On-Demand capacity at the current instance size.
Show the answer and explanation
Answer: C
A. Wrong: Compute Savings Plans apply to Amazon EC2, AWS Fargate and AWS Lambda usage, not to SageMaker AI training instances, so the training bill would not fall. SageMaker AI has its own Savings Plans, and the option still pays On-Demand rates for the job.
B. Wrong: Without a checkpoint location, an interrupted job loses all its progress and must start over, which can cost more than Spot saves. The stem allows only a few minutes of lost progress.
D. Wrong: Fewer epochs may hurt model quality and do not reduce the instance cost. The stem asks to lower cost while keeping the training progress.
Explanation
Managed Spot training uses discounted spare capacity, and with a checkpoint location in S3 SageMaker AI restores the job from its latest checkpoint after an interruption. Compute Savings Plans do not cover SageMaker AI usage, and skipping checkpoints or cutting epochs does not meet the requirement.
A ski resort technology group has a proprietary C++ library that its snow-forecast model depends on. The library is compiled for a specific Linux distribution and cannot be installed through a framework container or a requirements file.
The team needs a managed SageMaker AI training job that runs this library, reads its data from Amazon S3 channels, and writes the model to the output location that SageMaker AI expects.
Which approach meets these requirements?
Choose one answer.
AA. Use a built-in algorithm for training, and call the C++ library from a preprocessing step that runs in a notebook instance before each training job starts.
BB. Install the library from source in a notebook instance, then start a framework estimator that runs in the default framework container.
CC. Run the training on a plain Amazon EC2 instance outside SageMaker AI, and write the model to Amazon S3 with a script.
DD. Build a Docker image that holds the library and training code, follows the SageMaker AI container contract, and push it to Amazon ECR for the job.
Show the answer and explanation
Answer: D
A. Wrong: The built-in algorithm would not run the library in training, and a notebook step outside the managed job breaks the single managed run the team wants. The stem asks for the library in the training job itself.
B. Wrong: The default container does not contain the library, and installing a compiled library in a notebook does not give the training job access to it. The stem depends on the job running the library itself.
C. Wrong: This runs the training outside SageMaker AI and loses the managed job, logs and artifact handling. The stem asks for a managed training job.
D. Right: Correct. A bring-your-own-container image can run any code, reading channel data from /opt/ml/input/data and writing the final model to /opt/ml/model, which SageMaker AI uploads to S3. The image is stored in Amazon ECR and referenced by the training job. Learn more: Containers with custom training algorithms (opens in a new tab)
Explanation
A bring-your-own-container image can include a compiled library and follow the SageMaker AI container contract, reading inputs from /opt/ml/input and writing the model to /opt/ml/model. Built-in algorithms, notebook steps and default framework containers cannot run the library inside the managed job.
Want more on Training with SageMaker AI? The 5 full-length practice sets have 8 more Training with SageMaker AI questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →