Tuning inference parameters (temperature, top-p, top-k, stop sequences): AIP-C01 study notes and practice questions
What to know about Tuning inference parameters (temperature, top-p, top-k, stop sequences) for the AIP-C01 exam, from the Operational Efficiency and Optimization for GenAI Applications domain (12% of the exam guide), then 1 free question to test it. Part of the AIP-C01 study guide.
Inference parameters are set per request. The exam tests whether you change the right setting for the output you need, and whether you check the change with data instead of a single example.
Core parameters
Temperature scales randomness. Near zero, the model picks the most likely tokens and output is more repeatable. Higher values give more varied wording. Top-p limits sampling to the smallest set of tokens whose probabilities add up to p. Top-k limits sampling to the k most likely tokens, on models that support it. Max tokens caps the length of the response. A stop sequence ends generation when the model emits a chosen string, such as a closing tag or a delimiter.
Tune one sampling control at a time
Change temperature or top-p, not both, so you can attribute any effect to one setting. When you tune temperature, leave top-p at its default. Sampling settings do not make a model follow instructions better. Clear prompts, schemas and examples do that.
Reasoning budgets
Some models support extended reasoning, where a thinking budget limits how many tokens the model spends reasoning before it answers. A larger budget can help hard, multi-step tasks, and it raises cost and latency. Set the budget per use case rather than globally, and confirm that it leaves room for the final answer.
A/B test configurations
Treat parameters as configuration you measure. Build a golden dataset, run two parameter sets over it, and compare task accuracy, format compliance, latency and cost. Run each set several times, because generation varies. Store the chosen settings with the prompt version so you can roll back.
How to choose
Scenario wording
Choose
Extract invoice fields exactly the same way each time
Low temperature with a JSON schema
Generate five varied marketing taglines
Higher temperature, then check variety in evaluation
The answer must end at a fixed delimiter
Stop sequence
Cap long answers for a mobile screen
Max tokens plus a prompt that asks for brevity
Hard multi-step reasoning is inaccurate
Larger reasoning budget, measured for cost
Two settings look close
A/B test on the golden dataset
Common traps
High temperature for extraction, which produces inconsistent fields.
Changing temperature and top-p together, so the result cannot be explained.
Using max tokens to force a format. The output is cut off, not reshaped.
Treating a stop sequence as a safety control. It only ends the output.
Tuning on one example, which says nothing about the whole set.
A dental group uses a model on Amazon Bedrock to read patient referral emails and extract the referral number, the appointment date and the insurance amount into a JSON record. A downstream scheduling system rejects any record that does not match the expected format.
The same email often produces slightly different values or wording across runs, so the extracted fields change between two runs of the job. Auditors want the same email to produce the same record each time it is processed.
The extraction must stay on the same model and keep the JSON format, and the output should be as repeatable as the model allows.
Which change meets these requirements?
Choose one answer.
AA. Set the temperature to 0 and leave top-p at its default, so the model picks the most likely tokens and the same email gives nearly the same record each run.
BB. Raise the temperature to 0.9, run each email three times, and keep the most common value for each field, so that the random variation is removed by voting across the runs.
CC. Set top-k to 500 and top-p to 1.0 so that the model considers a wider range of word choices, which gives the extraction more room to find the correct referral number.
DD. Remove the JSON format instruction, and add a stop sequence of a closing brace so that the model ends each response early and the record is shorter.
Show the answer and explanation
Answer: A
A. Right: A temperature of 0 makes sampling nearly greedy, which makes outputs highly repeatable for the same input. Leaving top-p at its default adds no extra randomness, so the model and the JSON format stay unchanged. Learn more: Inference request parameters (opens in a new tab)
B. Wrong: A high temperature increases the variation between runs, and voting across several runs triples the cost without guaranteeing a single repeatable result. It works against the goal of the same record each time. Learn more: Inference request parameters (opens in a new tab)
C. Wrong: A wider range of choices increases variation between runs, and simple fields such as dates do not need more candidate words. The option moves the output away from repeatable results rather than toward them. Learn more: Inference request parameters (opens in a new tab)
D. Wrong: Removing the format instruction makes the output less likely to match the scheduling system's schema, and an early stop sequence can cut the JSON before it is complete. The option breaks the format requirement. Learn more: Inference request parameters (opens in a new tab)
Explanation
A temperature of 0 makes the model's sampling nearly deterministic, so the same email tends to yield the same extracted record, and the JSON instruction is kept. Higher temperature, wider sampling and format-breaking stop sequences all increase variation or break the schema.
Want more on Tuning inference parameters (temperature, top-p, top-k, stop sequences)? The 5 full-length practice sets have 5 more Tuning inference parameters (temperature, top-p, top-k, stop sequences) questions, 375 in all, none of them in the free exam. One payment of $14.99, no subscription. See the practice sets →
Study Tuning inference parameters (temperature, top-p, top-k, stop sequences)