ML lifecycle, MLOps and metrics: AIF-C01 study notes and practice questions
What to know about ML lifecycle, MLOps and metrics for the AIF-C01 exam, from the Fundamentals of AI and ML domain (20% of the exam guide), then 3 free questions to test it. Part of the AIF-C01 study guide.
Lifecycle questions often come as ordering questions: put the steps of an ML project in the right order. Others ask which metric fits a goal.
The pipeline
Define the business problem and how success is measured.
Collect data.
Prepare data: clean it, handle missing values, and engineer features (exploratory data analysis comes here).
Train the model on a training set.
Tune hyperparameters (settings chosen before training, such as learning rate).
Evaluate on data the model hasn’t seen (a validation or test set).
Deploy for real-time or batch inference.
Monitor for drift and degraded accuracy, then retrain as needed.
MLOps
MLOps applies DevOps practices to ML: versioned data and models, reproducible experiments, automated pipelines (such as SageMaker Pipelines), a model registry, and monitoring in production (SageMaker Model Monitor). It makes models repeatable, scalable and easier to keep accurate.
Model metrics
Metric
Measures
Prioritize when
Accuracy
Share of all predictions that are right
Classes are balanced
Precision
Of the cases flagged positive, how many really are
False alarms are costly
Recall
Of the real positive cases, how many were found
Missing a case is costly (fraud, disease)
F1 score
Balance of precision and recall
You need both
AUC
How well the model separates classes across thresholds
Comparing classifiers
Mean squared error / RMSE
Average size of errors for numbers
Regression
Business metrics
Cost per user, return on investment, customer satisfaction, conversion rate and time saved show whether a model is worth it, and they matter as much as technical metrics.
Common traps
Evaluating on training data. Always test on held-back data.
Accuracy on imbalanced data. If 1% of cases are fraud, a model that says “never fraud” is 99% accurate and useless; use recall or precision.
Deploying and forgetting. Data changes, so models need monitoring.
A bank is building its first ML model to flag suspicious card transactions.
Put these pipeline stages in the order they normally happen, first stage first.
Put these 5 steps in order.
AA. Train the model on the prepared training data.
BB. Collect historical transactions and their confirmed fraud outcomes.
CC. Deploy the model and monitor its predictions in production.
DD. Clean the data and engineer useful features.
EE. Evaluate the model against held-back test data.
Show the answer and explanation
Correct order: B, D, A, E, C
B. Collect historical transactions and their confirmed fraud outcomes. First: the team needs past transactions with known results, because a model can only learn from examples that already have answers.
D. Clean the data and engineer useful features. Second: raw transaction records contain gaps and errors, so they are cleaned and turned into useful inputs such as spending patterns.
A. Train the model on the prepared training data. Third: training uses the cleaned, feature-rich data prepared in the previous stage to learn the patterns that separate fraud from genuine payments.
E. Evaluate the model against held-back test data. Fourth: the model is scored on data it never trained on, which shows whether it generalizes before it is released.
C. Deploy the model and monitor its predictions in production. Fifth: only after testing does the model go live, and it is monitored so that drops in accuracy are caught early.
Explanation
An ML pipeline starts with collecting labeled historical data, then prepares the data, trains the model, evaluates it on held-back data, and finally deploys and monitors it. Evaluating on data the model never saw is what prevents a model that only memorized its training set from going live. This question tests the normal order of the lifecycle.
A hospital is testing a model that screens scans for a serious but treatable condition. In a validation set of 1,000 scans, 40 show the condition.
The model flags 36 of those 40 and 60 healthy scans for follow-up. Missing a patient is far worse than a follow-up check.
Which metric should the team focus on MOST?
Choose one answer.
AA. Precision, because it shows how many flagged scans truly have the condition.
BB. Accuracy, because it counts every correct prediction equally.
CC. Recall, because it shows what share of patients with the condition the model finds.
DD. Root mean squared error, because it penalises large mistakes.
Show the answer and explanation
Answer: C
A. Wrong: Precision here is 36 out of 96 flagged scans, or 37.5 per cent. It matters most when false alarms are costly, which the hospital says is not the main risk.
B. Wrong: Accuracy here is 87.6 per cent, but most scans are healthy, so the total can look good while missing real cases. It hides the four missed patients.
C. Right: Correct. Recall is 36 out of 40, or 90 per cent. The team can lower the decision threshold to find more of the 40 patients, at the cost of more follow-up checks, which the hospital accepts. Learn more: What is machine learning? (opens in a new tab)
D. Wrong: RMSE measures the size of errors in numeric predictions, such as prices. A yes-or-no screen is a classification problem, so RMSE does not fit.
Explanation
Recall measures the share of real cases that the model finds, so it is the metric to prioritize when a missed case is the costly error. The figures show precision at 37.5 per cent and accuracy at 87.6 per cent, which both hide the four missed patients differently. This question tests choosing the metric from the business cost of each type of error.
A logistics firm's data scientists retrain pricing models by hand on their laptops and copy the files to production when they remember. The CTO wants to adopt MLOps.
Which TWO practices are part of MLOps? (Choose two.)
Choose two answers.
AA. Track which data, code and parameters produced each model, so results can be reproduced.
BB. Run training, evaluation and deployment through one automated, repeatable pipeline.
CC. Freeze each model at release and stop monitoring its predictions in production.
DD. Let each scientist release models by hand, using a personal copy of the process.
EE. Skip automated tests so that new models reach production faster.
Show the answer and explanation
Answers: A, B
A. Right: Correct. Versioning the data, code and model artifacts lets the team reproduce any model and audit how it was made. This is one of the core MLOps practices. Learn more: What is MLOps? (opens in a new tab)
B. Right: Correct. Automating each stage in a pipeline makes training and release repeatable and consistent. It replaces manual copying and remembered steps.
C. Wrong: MLOps includes continuous monitoring of data and model performance after release. Stopping monitoring removes a key safeguard.
D. Wrong: Personal manual processes are the problem the CTO wants to fix. They make releases inconsistent and hard to audit.
E. Wrong: MLOps relies on automated testing before release to catch errors. Skipping tests trades a short delay for production failures.
Explanation
MLOps applies DevOps discipline to machine learning, so models are versioned, reproducible and released through automated pipelines. Monitoring and testing also belong to the practice, which is why options that stop them are wrong. This question tests recognizing the core MLOps practices.
Want more on ML lifecycle, MLOps and metrics? The 5 full-length practice sets have 18 more ML lifecycle, MLOps and metrics questions, 325 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →