Engineering note · Reproducibility contracts

What “deterministic” means in an AI tool

Deterministic should name a relationship between supported inputs and output bytes. If a tool cannot state that relationship precisely, the word is still only an adjective.

The useful definition is a contract

A useful deterministic-output claim answers a narrow question: under which tool version, input scope, ordering rules, serializer, and environment should a rerun produce the same bytes? It does not mean that every machine, dependency set, model provider, or execution environment will agree forever.

The public implementations of yy-corpus-evidence and YY Workload Receipt make two versions of that contract inspectable. They deliberately use different output recipes, which is the first important lesson: canonical means canonical for a named format, not canonical for all JSON everywhere.

1. Fix the byte recipe

Corpus Evidence writes UTF-8 JSON with sorted object keys, compact separators, Unicode characters left unescaped, non-finite numbers refused, and one trailing newline. Those choices live in a small canonical encoder. A change to indentation, escaping, separators, numeric handling, or the final newline would change the bytes even if a parser produced the same in-memory object.

Workload Receipt instead writes sorted-key JSON with two-space indentation and a trailing newline. That is also deterministic within its format. It is not byte-identical to the Corpus Evidence recipe, and neither tool claims that it should be.

2. Make ordering explicit

Filesystem enumeration and collection iteration should not decide the artifact order by accident. Corpus Evidence sorts directory entries, then sorts emitted assets by relative path and scan issues by their subject and reason. Workload Receipt sorts canonical event encodings before calculating its input fingerprint, orders aggregate buckets, and writes manifest entries by filename.

These are implementation choices, not properties supplied by JSON or a filesystem. Remove one of the sorts and two equivalent observations can become noisy diffs.

3. Frame and version the identity

Corpus Evidence computes its manifest identity from the canonical manifest bytes. Its stable-identity helper includes a domain prefix, an identity version, and an eight-byte length before every text part. Length framing matters: without it, the parts ab and c could become indistinguishable from a and bc after naive concatenation.

The schema and identity versions also make change visible. A future format may legitimately produce different bytes; that should be an explicit version boundary rather than a silent break in the claim.

4. Do not turn identity into authentication

The manifest identifier is an unkeyed digest. It detects corruption and accidental divergence, but it is not a signature, an authentication mechanism, or tamper evidence. Anyone who changes a manifest can recompute a valid identifier. The public README and test suite state that limitation directly.

This is the same boundary a SHA-256 file manifest has: the digest can support an integrity comparison, while trust in the reference value must come from somewhere else.

5. State where repeatability stops

Corpus Evidence v0.1.0 supports CPython 3.11 on Windows. Other Python versions and platforms are outside its released support matrix. It also reads a live filesystem rather than an atomic snapshot; common concurrent changes are surfaced, but a hostile filesystem is outside the threat model.

None of this is a claim that seeded model training, GPU kernels, or changing external APIs will reproduce. Those workflows have their own state, versions, concurrency, and provider boundaries. The honest contract should name only the variables it actually controls.

A checklist for any deterministic claim

  • Which exact input bytes and paths are inside the boundary?
  • What input is excluded or reported as an evidence gap?
  • Which collections are ordered, and by which key?
  • What canonical serializer produces the output bytes?
  • Which schema, identity, and tool versions apply?
  • Which runtimes and operating systems are supported?
  • Are external services, clocks, randomness, or concurrency involved?
  • Is the resulting identifier merely unkeyed integrity evidence?

A README does not need to reproduce the implementation, but it should answer these questions well enough that tests can pin the contract and reviewers can recognize a legitimate version change.

Inspect three concrete artifacts

Start with the open-source Corpus Evidence implementation and its tests. Then compare the different, equally explicit Workload Receipt format using its synthetic public sample. YY Dataset Preflight is separate tooling built with the same design preference for stable, inspectable outputs; its public sample demonstrates its own packet contract, not proof of the claims made about either open-source tool.