Doc system test

Client-side state and determinism

Everything about a project lives in its own folder, and the last run is rebuilt by replaying templates, so every run must be repeatable.

Meta block front matter of this doc
namespaces
  • durian.cloud/cyanprint/project-lifecycle
  • durian.cloud/cyanprint/prompting
  • durian.cloud/cyanprint/composition
  • durian.cloud/cyanprint/registry
  • durian.cloud/cyanprint/authoring
type
design
related
  • ../contexts/project-lifecycle/domain/state-file.mdx
  • ../contexts/project-lifecycle/architecture/template-stacking.mdx
  • ../contexts/project-lifecycle/implementation/three-way-merge.mdx
  • ../contexts/prompting/implementation/stateless-replay.mdx
  • ../contexts/prompting/domain/deterministic-state.mdx
  • ../contexts/composition/implementation/node-cache.mdx
  • ../contexts/registry/implementation/reference-pinning.mdx
  • The shop example
issues
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+deterministic-state-not-saved
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+update-drops-history
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+stack-order-unsorted-on-rerun
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+cache-key-before-questions
  • https://github.com/AtomiCloud/sulfone.boron/issues?q=TBD+plugin-order-race

The problem. A template writes files into a user's folder, the user edits them, and months later the template has a new version. To bring the new version in without losing the user's edits, CyanPrint needs to know what the template wrote last time. Most tools store that copy. CyanPrint stores nothing on a server and no copy of the old files.

The idea, in two lines. Everything about a project lives in the project's own folder: one small file, .cyan_state.yaml, names each template, its version and the answers given. What the templates wrote last time is rebuilt on demand by running the same versions again with the same answers, so that rebuild must give the same files every time.

The picture below shows where each piece of state lives. Only the left box belongs to the project; the servers keep nothing about it.

flowchart LR
  subgraph Folder["target folder ./shop (the only project state)"]
    SF[".cyan_state.yaml<br/>templates, versions, answers,<br/>history, managed files, clashes"]
    FILES["the files<br/>(template output + user edits)"]
  end
  subgraph Machine["user's machine, not the project"]
    CACHE["node cache<br/>~/.cache/cyanprint"]
  end
  subgraph Servers["shared servers, no project state"]
    REG["registry<br/>immutable versions,<br/>pinned dependencies"]
    COORD["coordinator<br/>containers per version id,<br/>sessions per run"]
  end
  SF -- "which versions, which answers" --> REPLAY((replay))
  REG -- "the exact version" --> REPLAY
  COORD -- "runs it" --> REPLAY
  CACHE -. "may short-cut a node" .-> REPLAY
  REPLAY -- "rebuilt last run (merge base)" --> MERGE((three-way merge))
  FILES -- "local" --> MERGE
Where state lives

What state exists, and where it lives

A run of cyanprint create or cyanprint update reads and writes the state in this table. Read it as "who owns what".

StateLives inLifetime
which templates the folder has, their versions and answers.cyan_state.yaml in the target folderas long as the folder
the files, with the user's editsthe target folderas long as the folder
a template version, its images and its pinned dependenciesthe registryforever (a version never changes)
running containers and work volumesthe coordinatorone run (a session) or until cleaned
one node's output from an earlier runthe node cache on this machineuntil evicted or cleared
questions answered so far in this runthe CLI's memoryone run

Newcomers get this wrong: the coordinator does not "remember" a project between runs, and the registry has no record of which folders use a template. Delete .cyan_state.yaml and CyanPrint no longer knows the folder was made by a template.

The state file holds one template record per stacked template, keyed <user>/<template>, plus two lists the CLI recomputes on every run. Here is ./shop/.cyan_state.yaml after step 3 of the shop example (two stacked templates):

yamlyaml
atomi/web:
active: true
history:
- version: 5
time: 2026-03-01T10:00:00Z
answers:
node_version: { type: String, value: "22" }
module_type: { type: String, value: esm }
name: { type: String, value: shop }
port: { type: String, value: "3000" }
features: { type: StringArray, value: [docker] }
deterministic_states: {}
files: [.dockerignore, .editorconfig, .eslintrc.json, .gitignore, Dockerfile, README.md, app.config.json, jest.config.js, package.json, src/index.ts, src/server.ts, tsconfig.json]
atomi/ci:
active: true
history:
- version: 2
time: 2026-03-09T16:20:00Z
answers:
ci_provider: { type: String, value: github }
deterministic_states: {}
files: [.editorconfig, .github/workflows/ci.yml, package.json]
managed_files: [.dockerignore, .editorconfig, .eslintrc.json, .github/workflows/ci.yml, .gitignore, Dockerfile, README.md, app.config.json, jest.config.js, package.json, src/index.ts, src/server.ts, tsconfig.json]
file_conflicts:
- path: .editorconfig
resolution: lww_all_no_resolver
winner_template: 42c8d1e0-9a8b-4c7d-8e6f-5a4b3c2d1e0f
variations:
- template_id: e5d07b3a-2f1e-4d3c-9b8a-7f6e5d4c3b2a
- template_id: 42c8d1e0-9a8b-4c7d-8e6f-5a4b3c2d1e0f
- path: package.json
resolution: resolver
resolver_used: { id: 9d3e5a40-8b7c-4d6e-9f0a-1b2c3d4e5f6a, docker_ref: atomi/json-merger, docker_tag: "1" }
variations:
- template_id: e5d07b3a-2f1e-4d3c-9b8a-7f6e5d4c3b2a
- template_id: 42c8d1e0-9a8b-4c7d-8e6f-5a4b3c2d1e0f
  • Template records are top-level keys, not under a templates: key (CyanState flattens them).
  • history only grows. The last history entry is what the next run replays; its time orders the stack.
  • deterministic_states is always {} today. See where determinism breaks.
  • file_conflicts records the clashes of the last layering pass; the field rules are in the state file format.

Why the client holds the state

The same choice runs through the whole product, not only prompting:

  • Prompting: the CLI sends every answer so far on every call (stateless replay).
  • Project: the folder's state file is the only record of a project.
  • The base of the merge: rebuilt from the state file, never stored.

The contract: what must be the same on every replay

A replay gives the same files only if every input to a template run is the same. These are the inputs, and what fixes each one.

Input to one template runFixed byHolds today?
the template versionversion id from the state file's version number; a version is never changedyes
its dependency versionsreference pinning at pushyes
its processors, plugins and resolverspinned in the template version at pushyes
its Docker imagesa reference and a tag stored at pushonly if the author never reuses a tag
the answersthe last history entryyes, for the template's own questions
values the script makes (random, time)deterministic stateno: never saved
the order of dependenciesdependency flattening sorts siblings by version idyes
the order of stacked templatessort by history entry timeno on rerun and on upgrade-by-create
the order of pluginsthe order in the template's Cyanno: order of registry lookups finishing

The two ways this contract is used:

Two jobs the same replay does
Asking questionsRebuilding the merge base
Whenduring one run, one call per questionat the start of every create and update
What is replayedthe template script, from the topevery stacked template, whole dependency tree
What must repeatearlier values made with d.getevery file the templates wrote last time
If it does not repeata value changes between questionsevery line holding the value is a merge conflict

Where the code keeps it

These are the places that make a replay repeat. Each is one line of what and one of why.

  • Pinned versions. cyanprint push turns atomi/docker (no version) into a fixed version id, the latest at that moment, and stores it in the template version. A run never picks a version. See reference pinning.
  • Sorted dependency order. Siblings are sorted by version id before the walk (flatten_impl, "Sort dependencies by ID to ensure deterministic execution order"), children before parents, root last.
  • Sorted stack order (create of a new template, and update). sort_specs orders template specs by history entry time, "for consistent LWW ordering".
  • Canonical cache key. The cache key hashes answers and states through sorted maps, so map order never changes it.
  • Sorted managed files. managed_files is sorted and de-duplicated, so the state file does not churn.
  • The determinism helper within one run. The CLI passes the state each question reply returns into the next call, so d.get("instance-id", uuid) gives one UUID for the whole run.

Here is the real code for the last point: how one run threads deterministic state from call to call, and where it stops.

sulfone.heliumsdks/node/src/domain/service/stateless_determinism.ts
  1. lines 6-17d.get returns the stored value for the key, or makes one and stores it in the map that goes back to the CLI.
    sdks/node/src/domain/service/stateless_determinism.ts:6-17typescript
    get(key: string, origin: () => string): string {
    if (!this.states) throw new Error('NullReferenceException: States dictionary is null');
    const state = this.states[key];
    if (state != null) {
    return state;
    }
    const val = origin();
    this.states[key] = val;
    return val;
    }
  2. sulfone.helium/sdks/node/src/domain/template/service.ts:16-35On a question the reply carries deterministicState; on the final reply it carries only the Cyan, so the values made in the last call go nowhere.
    sdks/node/src/domain/template/service.ts:16-35typescript
    async template(answer: TemplateInput): Promise<TemplateOutput> {
    const i = new StatelessInquirer(answer.answers);
    const d = new StatelessDeterminism(answer.deterministicState);
    try {
    const r = await this._template.template(i, d);
    return {
    data: r,
    } satisfies TemplateFinalOutput;
    } catch (e) {
    if (e instanceof OutOfAnswerException) {
    const q: Question = e.question;
    return {
    deterministicState: d.states,
    question: q,
    } satisfies TemplateQnAOutput;
    }
    throw e;
    }
    }
  3. sulfone.iridium/cyanprompt/src/domain/services/template/engine.rs:35-36Interactive run: start from the states the caller passes in.
    cyanprompt/src/domain/services/template/engine.rs:35-36typescript
    let mut answers: HashMap<String, Answer> = initial_answers.unwrap_or_default();
    let mut state_data: HashMap<String, String> = initial_states.clone().unwrap_or_default();
  4. sulfone.iridium/cyanprompt/src/domain/services/template/engine.rs:74-77After each answer, take the template's returned state for the next call.
    cyanprompt/src/domain/services/template/engine.rs:74-77typescript
    Some(val) => {
    answers.insert(question_id.clone(), val);
    state_data = q.deterministic_state;
    }
  5. sulfone.iridium/cyanprompt/src/domain/services/template/engine.rs:84-86Complete carries the Cyan and the answers only: no deterministic state.
    cyanprompt/src/domain/services/template/engine.rs:84-86typescript
    TemplateOutput::Final(c) => {
    Ok(TemplateState::Complete(c.cyan, answers.clone()))
    }
  6. sulfone.iridium/cyanprompt/src/domain/services/template/engine.rs:145Headless run: always starts from an empty state, whatever the caller has.
    cyanprompt/src/domain/services/template/engine.rs:145typescript
    let mut state_data: HashMap<String, String> = HashMap::new();
  7. sulfone.iridium/cyancoordinator/src/state/services.rs:62-71The state writer saves a history entry with a new, empty map.
    cyancoordinator/src/state/services.rs:62-71typescript
    let template_key = format!("{}/{}", username, template.template.name);
    let deterministic_states = HashMap::new();
    let history_entry = TemplateHistoryEntry {
    version: template.principal.version,
    time: Utc::now(),
    answers: answers.clone(),
    deterministic_states,
    };

Where the code breaks it

Each break below is what the code does today. The consequence is shown on the shop example.

1. Deterministic state is never saved

  • The SDK's final reply has no deterministic state; the CLI's Complete outcome has none; the state writer writes {}.
  • A headless run ignores any passed-in state.
  • Consequence: every replay makes new values. On an upgrade, the merge base (old version, new UUID B), the folder (first UUID A) and the incoming set (new version, new UUID D) all differ on one line, so app.config.json gets conflict markers on every upgrade.

The tree below shows ./shop after the upgrade in step 4; app.config.json holds conflict markers only because the UUID was not saved.

./shop after update web 5 → 6 (step 4)
Before
  • app.config.jsonkeptshow file
    {
      "name": "shop",
      "instanceId": "4f9c2e71-aaaa"
    }
    
  • jest.config.jskept
  • notes.mdkeptuser file
  • README.mdkept
After
  • app.config.jsonconflictmarker labels are libgit2's defaults; shown without themshow file
    {
      "name": "shop",
    <<<<<<<
      "instanceId": "4f9c2e71-aaaa"
    =======
      "instanceId": "c07d5b19-dddd"
    >>>>>>>
    }
    
  • jest.config.jsremovedweb:6 dropped it; user never edited it
  • notes.mdkeptuser file
  • README.mdconflictuser and web:6 changed the same line

2. The node cache hides the break, on one machine only

  • The node cache stores a node's output under a key of its version, images, answers and deterministic state. A hit returns the old bytes, old UUID included.
  • So on the machine that made the folder, replays of an unchanged template often hit the cache and repeat exactly. On a teammate's machine, in CI, or with --no-output-cache, they do not.
  • Consequence: the same command on the same folder gives a clean merge on one machine and conflict markers on another (step 3 of the example).

3. Stack order is random on a rerun or an upgrade done with create

  • cyanprint create of a template the folder already has builds its template specs by iterating the state file's map, and does not sort them. Rust's HashMap order changes from process to process.
  • Consequence: when two stacked templates write one file with no shared resolver, which one wins can change between two identical runs (.editorconfig in the example: indent 2 or 4).

4. cyanprint update drops its own history

  • The batch run appends each upgraded template's new history entry to the state file. The update then saves the copy of the state it read before the run over it.
  • Consequence: the state still names the old version and lacks new answers, so the next update rebuilds the base from the old version again and asks the new question again (steps 4 and 5).

5. The cache key is made before the questions

  • A node's key uses the answers known before it asks its questions; the answered output is stored under it.
  • Consequence: a later run that starts with the same, emptier answers is served the stored output and answers without asking. A rerun asks nothing (step 6); a second create atomi/web:5 ./blog on the same machine gets shop's answers (step 7).

6. Plugin order is a race

  • The coordinator looks up every plugin of a Cyan in parallel and runs them in the order the lookups finish.
  • Consequence: two plugins that edit one file can apply in a different order on two runs.

7. Image tags can move

  • Push stores each image as a reference plus the tag the author gave. If an author pushes a new image under the same tag, the same version id runs different code.

Step by step: one value through the shop's life

Step through what happens to the d.get("instance-id") value in app.config.json across the example. "Cache" is on (the default) unless the step says otherwise.

instanceId across the shop example
  1. Step 1: create web:5

    The script makes A during the first question call; the CLI passes A back on every later call, so the files get A. The history entry saves deterministic_states: {}.

    fileA
    saved state{}
    cache entry (answers {})A
  2. Step 3, base: rebuild web:5

    Saved answers, empty state. The key now holds the full answers, so the cache misses; the live run makes B and stores it.

    fileA
    baseB
    cache entry (full answers)B
  3. Step 3, incoming: web:5 again

    Same key as the base run, so the cache hits and returns B. Base equals incoming, so the merge keeps the file's A.

    fileA
    baseB
    incomingB
    resultA (clean)
  4. Step 3 without the cache

    Base and incoming both run live and make two new values. The line differs in all three.

    fileA
    baseB
    incomingC
    resultconflict markers
  5. Step 4: update web 5 → 6

    The base hits the cache (B). web:6 is a new version, so it runs live and makes D.

    fileA
    baseB
    incomingD
    resultconflict markers
  6. With deterministic state saved

    What the design intends: the saved A would be passed into every replay, so base, local and incoming all hold A.

    fileA
    baseA
    incomingA
    resultA (clean)

Template stacking is one part of this

Template stacking (the owner's "client-side stacking") is client-side state applied to more than one template. Each cyanprint create into a folder that already has a state file adds a template record; nothing in the registry links the stacked templates. On every run the CLI replays every stacked template twice (old versions for the base, new for incoming) and lays their outputs on top of each other, oldest history time first. So stacking needs the same contract as above, plus one more input: the stack order. Details: template stacking.

Questions for the designer

These are open because no design note, PR or comment states the reason. Each is in REPORT-2 with what the code shows.

  1. Was rebuilding the base (instead of storing it) chosen for size, for trust, or for portability between machines?
  2. Should deterministic state be saved per history entry, and should the final template reply return it?
  3. Should push store image digests instead of tags, so a version id fixes the code that runs?
  4. Is the stack order meant to be install time, or last-upgrade time (today it is the last history entry's time)?