Hyperparameter tuning: MLA-C02 study notes and practice questions
What to know about Hyperparameter tuning for the MLA-C02 exam, from the ML Model and Foundation Model (FM) Development domain (24% of the exam guide), then 2 free questions to test it. Part of the MLA-C02 study guide.
Tuning questions ask how to find good hyperparameters faster or cheaper, or why a tuning job isn’t improving.
Search strategies
Strategy
How it works
Use it when
Bayesian (default)
Learns from finished jobs to choose the next values
Limited budget; jobs run mostly in sequence
Random
Picks values independently
Many jobs in parallel; it loses nothing by not waiting for results
Grid
Tries every combination of categorical values
Few categorical options; you want every combination
Hyperband
Stops weak jobs early and gives more resources to promising ones
Iterative training (such as deep learning) where early results predict final ones
High parallelism helps random search but reduces the benefit of Bayesian search, because fewer finished results guide each choice.
Setting up a job
Choose an objective metric (such as validation:auc) and whether to maximize or minimize it.
Define ranges: continuous, integer or categorical. Use logarithmic scaling for values that span orders of magnitude, such as a learning rate from 0.0001 to 0.1.
Set max jobs and max parallel jobs.
Early stopping ends jobs that are unlikely to beat the best so far.
Warm start
IDENTICAL_DATA_AND_ALGORITHM: continue from a parent tuning job with the same data and image.
TRANSFER_LEARNING: reuse a parent job’s knowledge with new data or a changed algorithm.
Other levers
Tune the few hyperparameters that matter most first (learning rate, depth, regularization).
Autopilot (in Canvas) tunes automatically as part of AutoML.
Track tuning runs in MLflow or SageMaker Experiments.
Common traps
Linear scale for a learning rate range spanning several orders of magnitude.
Bayesian search with very high parallelism expecting it to learn between jobs.
IDENTICAL_DATA_AND_ALGORITHM after the dataset changed.
A regional bank tuned an XGBoost credit model last quarter with an automatic model tuning job. It has since received three months of new loan applications, and the team wants to re-tune the same algorithm, starting from the best settings it already found.
The team wants the new tuning job to reuse what the earlier job learned, even though the training data is now different. Its parent job used the same XGBoost algorithm and image version.
Which warm start configuration should the team use?
Choose one answer.
AA. Use TRANSFER_LEARNING, because it allows the input data to change while the new job still uses the parent job's results as a starting point.
BB. Use IDENTICAL_DATA_AND_ALGORITHM, because the new job reuses the parent job's results.
CC. Start a new tuning job without warm start, because warm start is allowed only when the input data stays exactly the same as in the parent tuning job.
DD. Skip tuning, and run one training job on the new data with the parent job's best hyperparameters set as fixed values.
Show the answer and explanation
Answer: A
A. Right: Correct. TRANSFER_LEARNING lets the new tuning job use different input data, and even a different version of the training image, while starting from the parent job's results. The total number of static and tunable hyperparameters must still match the parent. Learn more: Run a warm start hyperparameter tuning job (opens in a new tab)
B. Wrong: This type requires the same input data and training image as the parent. The bank now has new data, so the warm start would not be allowed.
C. Wrong: This discards what the earlier job learned, which the team wants to reuse. Warm start with transfer learning allows the data to change, so the option misreads the rule.
D. Wrong: This reuses the earlier best values but performs no search, so it cannot adjust them to the new data. The team wants to re-tune the model, starting from what the parent job learned.
Explanation
Warm start with TRANSFER_LEARNING lets a new tuning job use new input data while starting from the parent job's results. IDENTICAL_DATA_AND_ALGORITHM requires the same input data and training image, which the bank no longer has. Running a single job with fixed values performs no tuning at all.
A logistics broker is tuning a gradient-boosting freight model with a budget of 60 training jobs. Its account has capacity to run 20 jobs at the same time, and the team wants the search to finish as quickly as possible.
The team wants to explore the hyperparameter space widely rather than refine one area. The algorithm reports its objective metric only when each training job finishes, so no job can be judged or stopped partway through.
Which search strategy should the team choose?
Choose one answer.
AA. Bayesian search, because it learns from every completed job before choosing the next settings.
BB. Random search, because with many jobs running in parallel it explores the space broadly and finishes sooner than a model-guided search.
CC. Grid search over every combination of values, because it tests each candidate setting systematically and guarantees the best settings within the 60-job budget.
DD. Hyperband, because it stops weak training jobs early and shifts their budget to the most promising configurations.
Show the answer and explanation
Answer: B
A. Wrong: Bayesian search gains most from results that arrive one or a few at a time. With 20 jobs running together, each round has little finished information to learn from.
C. Wrong: Grid search supports only categorical hyperparameters and runs one job for every combination, which quickly exceeds a 60-job budget. It does not guarantee the best settings for continuous ranges.
D. Wrong: Hyperband decides which jobs to stop from intermediate results reported during training. This algorithm reports its metric only at the end of each job, so Hyperband cannot do its early stopping.
Explanation
Random search chooses values independently of earlier results, so it can use full concurrency without hurting the search and explores the space broadly. Bayesian search learns from completed jobs and gains less when 20 run at once, grid search handles only categorical values, and Hyperband needs intermediate metrics to stop jobs early.
Want more on Hyperparameter tuning? The 5 full-length practice sets have 9 more Hyperparameter tuning questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →