Build a determinism test that can actually fail

Write the failure condition first. Decide whether a changed token, decision or external action breaks the claim. Then build a test that can expose that difference instead of merely displaying a saved answer.

What if repeated runs reuse cached answers?

First establish what executed, then compare its output. Three matching rows cannot establish three fresh computations if they were copied from previous results. Keep the evaluation ID separate from the underlying model-call identity.

Promptfoo documents separate cache namespaces for repeat indices. That permits distinct results for repeats 0, 1 and 2, while a later evaluation can reuse each index’s saved result. Its documentation recommends disabling caching alongside repetition when fresh LLM calls are required. This is documentation review; we did not run Promptfoo or a model endpoint.

For that runner, the documented pattern is:

promptfoo eval --repeat 3 --no-cache

The command states the requested behavior at the evaluation layer. Retain execution evidence before claiming that every component obeyed it. Provider-side reuse, tool-response reuse, retrieval caches and recorded workflow steps may have different boundaries. If a boundary is unobservable, state that limit.

Use a fictional three-repeat check to define the acceptance rule. In pass A, request fresh computation and retain the call evidence for every repeat. In pass B, deliberately enable reuse and verify that your report labels the reused results. Pass B tests your ability to detect reuse; its matching answers do not add fresh trials to pass A. If pass C changes the model, rule or dataset, give it a separate manifest and comparison result.

Record a missing call trace as unresolved. A new local run ID, a later timestamp or a repeated success message alone does not establish fresh model computation. PyTorch’s guidance adds a separate condition: reproducibility is not guaranteed across releases, commits or platforms even with identical seeds. A fresh run still needs a defined environment.

The core Checklist version 1.3 includes a blank per-attempt CSV and JSON alongside the ten checks. Use it to retain requested mode, observed mode, evidence and output identity. All external service tests in these instructions remain untested. The pack’s completed example concerns only our local source-group calculation.

Do matching tool calls prove the same action occurred?

Matching tool names establishes only the comparison you ran. Compare the arguments and relevant state before claiming two actions match, then inspect what happened at the destination. LangChain’s evaluation guidance distinguishes final-response, single-step and trajectory evaluation. It explicitly notes that a tool-name trajectory comparison does not evaluate the tools’ inputs.

Consider a fictional notification workflow. Its approved action sends document version 3 to test recipient A. Two traces can both contain lookup_document followed by send_notice, while one sends version 2 or addresses recipient B. A tool-name check passes both traces; a comparison of the approved request would not. This is a teaching example, not an observed customer incident.

Decide whether your requirement is an exact path or a permitted outcome with explicit constraints before testing. A trace with an extra permitted validation step can differ from the reference sequence while still satisfying the workflow requirement. LangChain documents this limitation of exact-path scoring. Do not silently change the comparison after observing a failure; retain the original result and document any revised test separately.

For one declared test boundary, include an unchanged valid request, a changed recipient, a changed document version and an omitted-action case. Record expected outcomes first. Then run fresh authorized tests, retain every attempt and leave missing destination evidence unresolved. A repeated final message or reused receipt should not substitute for those observations. These are proposed procedures; no notification service was executed here.

Use Checklist version 1.2 to keep the equality rule, correctness and action confirmation separate. The ten criteria remain stable. PyTorch’s reproducibility guidance adds a different limit: matching input and seed do not guarantee matching results across releases, commits or platforms. Specify those conditions separately from the action fields.

What if the reviewed input changes before execution?

Test the changed request as a separate case and retain the original evidence. A repeatability result describes the fixed conditions actually exercised. A payment approval describes the request and scope actually authorized. Neither record should be silently rewritten when an input changes.

Consider a fictional workflow: a reviewer approves USD 100 to test account A; before execution, a revised extraction proposes USD 150 to account B. The engineering question is whether the system recognizes the material change, applies the current authorization policy and prevents an uncovered request from executing. The determinism question is whether the declared decision repeats under the same complete conditions. The correctness question is whether the amount and destination match the source. Record each answer separately.

Our five-case changed-evidence worksheet includes the changed-request case and an unchanged-request positive control. Start with blank observed results. Execute only against disposable targets within your authorized test boundary, preserve the exact approved and pending requests, and reconcile actual destination state. Expected behavior in a worksheet is not observed behavior in your application.

LangGraph’s interrupt documentation explains why this also matters on resume: the node restarts and code before the pause can run again. Post-approval side effects can also repeat if the node fails after committing them. A renewed permission check therefore complements idempotency and destination verification; it does not replace them. PyTorch gives a separate boundary: results are not guaranteed across releases, commits and platforms. Keep runtime-change tests out of an unchanged-configuration denominator.

Report what was compared, what changed, what remained untested and what happened at the destination. If you cannot recover the approved request, retain an unresolved authorization finding instead of inferring it from the final success message.

What claim are you testing?

Write a statement before the test: “For these inputs, state and versions, this step produces the same specified result.” Choose the result: text, structured fields, calculations, decisions or external actions. Numeric tolerance is a different criterion from exact equality; state which one you use.

This protocol is our practical recommendation. Its emphasis on environment boundaries follows PyTorch’s documented limits across releases, commits, platforms and CPU/GPU. Read the source.

What should the test record contain?

• Test case. What to retain: Identifier, complete input and intended task.
• Starting context. What to retain: Conversation, retrieved records, tool responses and relevant state.
• Configuration. What to retain: Model, prompt, process and rule versions; environment and settings.
• Run evidence. What to retain: Raw result, action arguments, destination response and exception path.
• Comparison. What to retain: Exact matches, differences, distinct results and any declared tolerance.
• Correctness check. What to retain: Reference result and explanation of the acceptance criteria.
• Limits. What to retain: Hidden conditions, excluded cases and untested environments.

How should you run the comparison?

1. Freeze the declared inputs and configuration. Reset the relevant state between runs.
2. Use fresh executions. Record whether a cache is enabled so returned stored output is not mistaken for repeated computation.
3. Compare each run with the chosen baseline. Keep the raw records, including failures and unresolved results.
4. Test exception paths deliberately. Check both the pause and the authorized resolution.
5. Run environment-change tests separately: a new model, rule version or platform asks a different question from fixed-condition repetition.

Use a controlled test environment for actions that would otherwise send messages, move money or modify live records. Confirm the real integration behavior in an appropriately authorized validation step.

How should you report the result?

Illustrative example: first capture a reference baseline, then perform 20 additional fresh runs of one fixed case. If 18 outputs match the baseline and two differ, baseline agreement for the 20 comparison runs is 18 ÷ 20 = 90%. If all 20 match, report 100% observed agreement for that case and configuration, with the number of runs; do not claim proof for every future input.

For workflow actions, compare intended actions and resulting state rather than automatically treating unique request IDs or timestamps as decision changes. Document exactly what is excluded from equality and why. Keep those metadata fields in the audit record.

Does replay test the same thing as a fresh run?

Establish which steps execute again. Temporal documents that a completed Activity is not executed again during workflow replay; its recorded result is reused. LangGraph documents reuse of earlier results and reexecution of downstream nodes, including LLM and API calls, after a chosen checkpoint. Those calls may produce different results. Temporal Activity documentation; LangGraph time-travel documentation.

Our recommendation is to label each test as saved-result replay, partial reexecution or a fresh run, then record exactly which steps were exercised. A reused result provides no new observation of that step’s computation. Keep retry and repeated-action tests separate, with destination confirmation and duplicate-effect controls.

How do you test whether cancellation stopped an action?

Compare the destination with the cancellation record. A request to stop, a worker receiving that request, and a completed rollback are different observations. Temporal documents that Activities must Heartbeat to receive Cancellation, and a Workflow can continue without waiting for acceptance. Read the primary documentation.

Our recommended test plan uses a fictional action in a sandbox and four separately recorded cases:

For example, if a sandbox reservation is created before cancellation arrives, retain the creation receipt. If an approved cancellation later removes the reservation, keep that second receipt too. Report both state changes. Neither a stopped worker nor a repeated “cancelled” message proves the first effect never happened. This is a teaching scenario, not an observed incident.

Download the eleven-procedure action worksheet and complete DA-A08–DA-A11. All observations start blank and outcomes untested. Record the documented cancellation mechanism, timeout and retry policy for the actual version under test. Keep the results separate from fresh-run equality and correctness findings; PyTorch’s environment limits also remain relevant to any broader reproducibility claim.

How do you test a retry after an uncertain action?

Keep this separate from fixed-condition repeatability. Temporal’s Activity documentation distinguishes reuse of a completed Activity result during replay from its default automatic retry behavior.

Our recommended test starts with a sandbox destination that makes a change but withholds the response. Record the logical operation, every request attempt and the destination’s before-and-after state. Exercise a retry under the destination’s documented duplicate-request contract. Check whether the intended effect occurred once; keep the result unresolved if destination evidence is unavailable. The workflow saying “done” does not supply that evidence.

Use the Checklist’s eleven action, retry and cancellation procedures for lost acknowledgements, concurrent duplicates, changed approval arguments and saved-result replay. These are manual test designs, not observed production results. Report their findings separately from equality across fresh computations.

Can you trust a passing score alone?

Inspect the artifact and the execution behind it. METR documents evaluation behavior in which models raised scores by modifying tests or using reference answers instead of solving the intended problem. These are laboratory/evaluation observations, not a general production incident rate. Read METR’s report.

Our recommendation is to keep the evaluator and reference answers outside the tested agent’s write permissions and independently review discrepancies. A passing score, repeated output and correct business outcome should remain separate fields.

What if only completed runs match?

Keep failures and unavailable results in the run ledger even when they cannot enter an output-equality comparison. Report how many fresh attempts started, how many produced the required evidence, how many comparisons were possible and how many matched. A completed-only table can conceal the very boundary the test should expose.

For a hypothetical six-run plan, suppose four fresh runs produce the same answer, one stops with a recorded error and one times out without a result. With the first completed run as baseline, the remaining three completed outputs all match: 3/3 output comparisons. The plan still has 4/6 runs with usable output, one recorded error and one unresolved attempt. It is misleading to describe this as six successful deterministic runs. These are illustrative counts, not results from a platform test.

Write the acceptance rule in advance. Your workflow might require all six runs to complete, or permit a named retry policy, but it must retain the original failure, the new attempt and any changed input. Replacing missing input files before retrying changes the conditions; that successful retry cannot demonstrate repetition of the original attempt. The provider’s grading report gives a concrete reason to preserve attempts that never produced an accepted result.

Also distinguish zero observed destination changes from destination evidence unavailable. For an action timeout, inspect a trusted destination record before retrying. If you cannot determine whether the first attempt took effect, keep its outcome unresolved; a later success message alone cannot establish that the action happened exactly once.

Framework replay is a separate dimension. Temporal reuses a completed Activity’s recorded result during Workflow replay; LangGraph reexecutes nodes after the checkpoint, including external calls. Record which mode and steps actually ran before counting a new comparison. These documentation statements were rechecked October 30; no framework was executed for this guide.

Use action/retry worksheet v1.2 for the proposed failed-attempt test. Start every observation as untested and attach attempt-level evidence before assigning a result. For factual claims inside the outputs, use the companion measurement example, which separates claim, attempt and request counts.

What comes next?

Use the Deterministic AI Checklist to collect evidence and unresolved questions. Use the deterministic AI definition to explain the scope to reviewers. Read LLM reproducibility for inference-specific limits. For factual support, use the companion hallucination measurement guide.

Reading scope

All five sources were rechecked on 2026-10-11 through indexed passages after direct opens returned no body. Full live-page text remained unavailable. LangGraph’s replay/fork opening and code and Temporal’s Activity/history/replay guidance support the new replay section. The other accessible openings continue to support the scoped claims below.

• PyTorch: Introduction and initial reproducibility guidance read; main documentation can change. Accessed 2026-10-11.
• METR: Opening research account and example read; these are evaluation observations, not customer incident rates.
• Thinking Machines Lab: Introduction read; cited for the distinction between temperature-zero sampling and reproducible inference.

October 14 maintenance: Temporal’s completed-result replay and automatic retry guidance was rechecked through exact-page indexed text. PyTorch’s direct recheck returned no readable body, so its previous access scope remains unchanged; this update does not claim a new full-source audit.

LangGraph’s indexed Overview and Replay passages were also rechecked October 14 and continue to support the stated replay distinction.

October 16 maintenance: PyTorch’s environment limits, Temporal’s completed-Activity replay/default retry distinction and LangGraph’s checkpoint reexecution were read in the page bodies and remain supported. The filled Checklist review now links actual local toy outputs to review fields, keeping production actions and approvals untested. METR and Thinking Machines access scopes above remain historical; this is not a claim that those sources were re-read today.

What if the input changes but its meaning does not?

Declare the comparison rule before running the test. The Checklist dataset export review uses exact bytes: appending a newline leaves parsed JSON values unchanged but invalidates the approved input hash. The observed rejection is correct under that rule. If your application promises semantic equivalence, define and test its normalization separately. Never silently drop fields or change the rule after seeing a mismatch.

The downloadable run record includes two fresh exports and an altered-input control. It verifies the local export’s consistency, not inference determinism, source truth or authorization. The October 16 follow-up live PyTorch recheck returned no readable body and subsequent retrieval attempts failed; the earlier same-day body read remains the source evidence for the environment limits above.

Can you reproduce the selection without rechecking the evidence?

Yes. Our review-date example computes a ten-record queue from two frozen files. Repeating that calculation tests the selection rule; it does not reopen the sources, determine whether a court order changed, or validate an AI system. Those require a separate source-review receipt.

Declare whether you are recomputing a result, replaying recorded state or testing a live dependency. Temporal reuses a completed Activity’s recorded result during workflow replay; LangGraph reexecutes nodes after the selected checkpoint, including external calls. A reader should be able to tell which operation supplied every claimed result. These examples do not establish identical behavior across framework versions.

Can date handling make a repeatable result misleading?

Yes. Repeatedly converting a month-only value into its first day passes an equality test while adding precision that the input lacks. Our filled date-precision review instead preserves the date, precision and milestone explanation. Its successful fresh runs establish repeatable file generation; the evidence for the underlying dates remains a separate question.

October 21 source recheck: PyTorch still limits reproducibility across releases and platforms; Temporal documents reuse of completed Activity results during replay; LangGraph reexecutes nodes after the checkpoint, including interrupts. METR’s evaluation examples and Thinking Machines’ inference discussion were also reopened. Those source claims remain supported within their stated scope; this offline date example does not test any of those frameworks or a model.

What if a correct calculation supports the wrong conclusion?

A test should challenge the interpretation as well as the arithmetic. The principal evidence inventory can correctly count distinct hosts without establishing independent investigators. Its synthetic example gives four exact URLs, three document URLs after fragment removal and two hosts. All three totals are correct under different declared rules.

Before running your test, write down the unit being counted, the normalization rule and the claim the result is allowed to support. Include a case where fragments differ, one where a query parameter differs and one where a subdomain differs. Reject missing evidence fields rather than silently counting them as zero. Our executed review records these checks separately from source truth and workflow approval.

These are fresh local file calculations. They do not exercise Temporal, LangGraph or an AI model. October 23 source rechecks confirm that Temporal reuses completed Activity results during replay, while LangGraph reexecutes downstream nodes. PyTorch retains its environment and release limits. Name what actually ran before interpreting matching outputs.

What invalidates an earlier repeatability claim?

An earlier result establishes what happened under its recorded conditions. A material change requires a new comparison before you extend that claim. PyTorch explicitly limits reproducibility across releases, commits and platforms, including CPU/GPU differences with identical seeds. Keep old results, but identify the configuration they tested.

For example, if policy version A and policy version B produce the same answer on one saved invoice, that is one cross-version observation. It does not show that either version repeats on fresh executions or that both policies are correct. First repeat each version under fixed conditions; then compare versions against separately approved expected decisions, including changed boundaries and exceptions. Record failed and untested cases alongside passes.

Our proposed review record contains the original model/runtime/rule/data versions, the changed item, the expected effect, the affected cases, fresh-run results and an owner for unresolved differences. A rule change can invalidate the expected answer even when the engine remains deterministic. A permission change can invalidate an authorization conclusion even when output bytes still match.

Protect the acceptance tests from the component being evaluated. METR documented evaluation manipulation; a matching score after an agent edits its grader does not preserve the original test. Independently retain the baseline and inspect the evaluator as well as the output.

The evidence-freshness supplement offers a blank worksheet and a worked editorial example. Its repeatable queue calculation tests local metadata processing; production controls and model behavior remain untested. Use it to record the boundary of a claim, not to approve a deployed system by analogy.

Which deliberate changes should a comparison test catch?

Start with a legitimate input, then change one property and state the expected result before running the test. The following local file-comparison checks were executed for the November 1 Register analysis; they do not test a deployed AI model.

Two fresh subprocesses also matched all six published outputs. The nine-check receipt records the actual results and runtime. A test that only repeats the same input can reproduce the same bug; deliberate changes show whether the comparison notices differences it is supposed to notice. These checks still do not prove that incident summaries are true or that an evaluator is protected from tampering.

For an AI workflow, choose analogous cases using its actual business contract. Keep environment changes separate from fixed-environment repetition: PyTorch documents limits across releases and platforms. Record the expected behavior and unresolved evidence in the version 1.1 release worksheet, then test the real workflow before extending the claim.