The Deterministic AI Checklist: ask for the record

A repeatability claim needs a record a reviewer can inspect. Start with the boundary and the test conditions. Then request the decision, exception and action evidence; a repeated answer does not fill those gaps.

How should you use this checklist?

Use version 1 to review one named workflow and configuration at a time. Ask every vendor or internal team for the same evidence. This is our proposed evaluation method, not a certification, external standard or ranking.

For every row, record evidence status (provided, partial or missing) and test outcome (met, not met or untested). Add an owner and follow-up. Do not add the rows into a single accuracy score: the questions assess different properties.

Download the CSV worksheet or JSON checklist. Both contain the ten criteria below and blank review fields; neither executes tests.

Jump to the ten checks · Prepare a release review

How to cite Deterministic AI. The Deterministic AI Checklist, worksheet version 1, citation and review-record update October 13, 2026. https://www.deterministicai.org/checklist/ (include your access date). The checklist text, CSV/JSON worksheet and review-record template are available under CC BY 4.0. Credit Deterministic AI, link this page and the license, and identify changes. Linked source documents retain their own terms. This license does not cover contributor portraits or other site artwork.

Can you run a small example first?

Download the Python example, version 1. It uses Python 3.9 or later and only the standard library. Save the file locally and run it from that directory:

python3 deterministic-ai-example-v1.py > result.json
python3 deterministic-ai-example-v1.py --fault variation > variation.json
python3 deterministic-ai-example-v1.py --fault wrong > wrong.json

This is our illustrative test artifact, not a vendor benchmark. It makes no network calls and executes no payment or other external action. The first command should exit with status 0. The two deliberate failure controls should exit with status 1; those failures show that the checks can reject bad results.

The toy rule approves a verified supplier’s invoice at or below 10,000 cents, routes a larger invoice for review, and leaves an unverified supplier or missing amount unresolved. Four fixtures spell out the inputs and expected decisions. These are invented teaching rules, not accounting advice or evidence about an AI product.

Each case gets one reference baseline followed by five fresh function calls with copied inputs. The comparison checks the exact decision and reason values; the run identifier is retained as evidence and excluded from equality. There is no result cache or external state in the toy function. These are fresh computations in one process, not separate processes or fresh model calls.

What should the example report?

The default case set produces four baselines and 20 comparison runs. Reference correctness is checked on all 24 executions, including the baselines. Keep those denominators separate. With the supplied code, the expected observations are:

• Default: 20/20 comparisons match their case baseline, and 24/24 executions match the fixture reference.
• Injected variation: 8/20 comparisons match; 12/24 reference checks pass. This deliberately changes output on alternating runs to test the comparator. It is not evidence of spontaneous nondeterminism under unchanged conditions.
• Injected wrong result: 20/20 comparisons match, while 0/24 reference checks pass. Consistent output can still fail the approved rule.

The JSON retains every output, case and run identifier, the reference result, a runtime/source/input manifest and the untested areas. You can adjust the comparison count with --comparisons 10; the default’s numerical expectations above then change.

Which checklist evidence does this supply?

The example illustrates a boundary, inputs, versions, fresh calls, equality and correctness—the subjects of checks 1–6 below. Two fixtures illustrate exception outcomes, but do not test human authorization. External action confirmation, actual replay and production evaluation integrity remain untested. The example records its own source hash for comparison; a hash alone does not prevent tampering or establish trusted provenance.

Before adapting it to a real workflow, define independent expected results, reset all relevant state, expose the actual computation and retain failures. Keep production reference answers and evaluator controls outside the tested agent’s write access. That recommendation follows the evaluation-tampering observations in METR’s report; the toy script itself provides no such isolation.

What are the ten checks?

1. What exactly is claimed to be deterministic?

Request a written boundary: interpretation, calculation, policy decision, action arguments, generated explanation, or the whole workflow. Name what is compared and what is excluded. Use the definition of deterministic AI to state the claim precisely.

2. Can you reconstruct the inputs and starting state?

Request the input record, retrieved document versions, relevant conversation, tool responses and initial business state. Mark conditions that cannot be recovered. For a hypothetical invoice approval, this includes the invoice, applicable policy and approval state.

3. Which versions and environment are covered?

Request a manifest for the model, prompt, rules, workflow and relevant runtime configuration. Test changes to that manifest separately. PyTorch explicitly limits reproducibility across releases, commits, platforms and CPU/GPU execution, even with identical seeds. That supports recording the environment instead of relying on a seed alone. Source: PyTorch.

4. Were the comparisons fresh executions?

Request run identifiers, cache settings, state-reset procedure and raw results. Identify any reused model or tool outputs. Apply the testing protocol to fresh computations under the declared conditions; record stored-result reuse as a separate test mode.

5. What counts as an equal result?

Request the comparison rule before execution: exact text, structured values, decision, action or resulting state. If numeric tolerance or excluded timestamps are appropriate, document them and preserve the original records. Report the number of cases, runs and differences, including exceptions.

6. Was correctness checked independently?

Request an approved reference result or business rule and compare the actual outcome with it. In a hypothetical tax calculation, matching repeated amounts answers the repeatability question; checking the applicable rule, inputs and arithmetic answers the correctness question. Report both findings.

7. What happens when evidence is missing or conflicting?

Request a demonstration of missing input, conflicting records and an unauthorized request. Record whether the workflow stops, asks for clarification or routes an exception. Review who may resolve it, what they can change and whether reuse of the resolution requires approval.

8. What proves the intended action occurred?

Request the authorized action arguments, destination response and resulting state. In a test environment, exercise failed and ambiguous responses and inspect handling of repeated requests. Define success using destination evidence and keep unresolved outcomes visible.

9. What does replay reuse or execute again?

Request a step-by-step account of the replay behavior. Temporal documents that workflow replay uses the recorded result of a completed Activity without executing it again. LangGraph documents that nodes after a selected checkpoint reexecute, including LLM and API calls, and may produce different results. These are different documented behaviors, not a comparison of overall product quality. Temporal; LangGraph.

Review replay, retries and fresh runs separately. Before rerunning an external action, establish how duplicate effects are prevented and how its status will be confirmed.

10. Can a reviewer trust the evaluation record?

Request raw evidence and control over the evaluator, reference answers and acceptance criteria. METR reports models altering tests or scoring code and accessing reference answers in software-development and AI R&D evaluations. These observations support our recommendation to restrict the tested agent’s ability to alter its evaluator and to inspect the actual artifacts. They do not establish a production incident rate. Source: METR.

What should the review conclude?

Use a bounded conclusion, for example: “For workflow W, configuration C and the listed cases, we observed the recorded agreement across fresh runs. Correctness, exception handling and action confirmation have separate results. These gaps remain unresolved.” This is a report template, not a claim about an evaluated product.

Identify critical missing evidence explicitly. Assign a follow-up owner and a decision about whether the workflow may proceed within the approved scope. Keep the raw records alongside the review.

Read LLM reproducibility for model-level limits. For source support, pair this checklist with the companion hallucination measurement guide.

How do you turn the checks into a release decision?

Review one named workflow release at a time. Save the filled worksheet with a review record template. The template does not run tests; it keeps the scope, evidence and decision together. Our recommended sequence is:

1. Agree on the claim. Name the component, fixed inputs and state, version manifest, compared fields and exclusions. Set acceptance criteria before seeing results.
2. Collect the observations. Retain case IDs, baselines, comparison-run counts, raw outputs and independent expected results. Link to the records rather than replacing them with a score.
3. Exercise a failure. Show a differing result and a consistently wrong result. The example above demonstrates why those are separate checks. Test missing evidence and unauthorized actions where they fall inside the workflow boundary.
4. Resolve each row. Keep evidence status separate from test outcome. A supplied report can still show a failed test. Missing evidence leaves the criterion untested, even if a vendor says it passes.
5. Record the disposition. Give repeatability, correctness and auditability their own findings. Name unresolved conditions, the responsible reviewer and what must change before release. Schedule a new review when the declared configuration changes.

This sequence is an editorial review method, not a universal release threshold. The accountable owner must set requirements for the intended task. A successful toy test or a completed form does not authorize production deployment.

What should a reviewer be able to reconstruct?

The record should connect claim → configuration → cases → raw results → finding → decision. Include where each artifact is stored and who approved the reference results. Record reused results explicitly: Temporal documents that replay uses a completed Activity’s recorded result. That observation cannot serve as another fresh execution of the Activity.

When the environment changes, keep the old decision tied to its original manifest. PyTorch’s reproducibility guidance does not guarantee matching results across releases or platforms. A repeat test after an upgrade evaluates the new boundary; it does not retroactively enlarge the old claim.

For examples of the factual failures a repeatability test does not detect, consult the companion Hallucination Tracker. Use incident records to design tests; do not treat them as comparative vendor scores.

Release history

• 2026-10-13, citation update: stable /checklist/ address, citation guidance, explicit worksheet reuse license and downloadable review record. The ten version-1 criteria are unchanged.
• 2026-10-13, example update: runnable toy example with deliberate variation and wrong-answer controls.
• 2026-10-11: ten-check worksheet first published.

What source material was available?

All four sources were rechecked October 13, 2026 through substantive indexed text after direct opens returned no body. PyTorch’s introduction and initial randomness guidance, METR’s opening and first example, LangGraph’s replay/fork explanation and code, and Temporal’s Activity guidance support the scoped claims above. Full live-page completeness was not verified; later METR search snippets were not used.

The downloadable example was executed locally in its three modes. Those runs validate the illustrated toy behavior only; no independent model, vendor runtime or production workflow evaluation was performed.