Doc system test

Template stacking

How several templates installed into one folder are each replayed and laid on top of each other on the user's machine, oldest first.

Meta block front matter of this doc
namespaces
  • durian.cloud/cyanprint/project-lifecycle
  • durian.cloud/cyanprint/composition
type
flow
code
  • sulfone.iridium/cyanprint/src/run.rs
  • sulfone.iridium/cyanprint/src/update/spec.rs
  • sulfone.iridium/cyanprint/src/update/orchestrator.rs
  • sulfone.iridium/cyancoordinator/src/operations/composition/operator.rs
  • sulfone.iridium/cyancoordinator/src/operations/composition/layerer.rs
related
issues
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+stack-order-unsorted-on-rerun

The problem. A project often needs more than one template: a web app template, then a CI template, then a docs template. They are made by different people, know nothing of each other, and may write the same file.

The idea. Each cyanprint create into a folder that already has a state file adds one more stacked template. On every run, every stacked template is replayed on its own, and their outputs are laid on top of each other, oldest first, using the same per-file rules as dependency layering.

This is client-side state applied to many templates: the stack exists only in the folder's state file. Nothing in the registry links two stacked templates.

The picture shows one run with two stacked templates.

flowchart TB
  SF[".cyan_state.yaml<br/>atomi/web (1 Mar), atomi/ci (9 Mar)"] --> PS["previous specs<br/>web:5, ci:2 (saved)"]
  SF --> CS["current specs<br/>web:6, ci:2 (update)"]
  PS --> W5["replay web:5<br/>(its own tree, layered)"] & C2a["replay ci:2"]
  CS --> W6["run web:6<br/>(its own tree, layered)"] & C2b["run ci:2"]
  W5 --> SB["stack: web under ci"] --> B[merge base]
  C2a --> SB
  W6 --> SI["stack: web under ci"] --> I[incoming set]
  C2b --> SI
  B --> M((three-way merge))
  I --> M
  L[the folder] --> M
Stacking web and ci in ./shop

How it works

  1. Read the stack. Every template record with active: true becomes a template spec: its last history entry's version, answers, deterministic state and time.
  2. Make two lists. Previous specs (what was run last time) and current specs (what runs now). Which versions and answers go in each depends on the command; see the table below.
  3. Order them. Each list is sorted by the time of the template's last history entry, oldest first, on cyanprint update and when create adds a new template. On a rerun or an upgrade done with create, the lists are not sorted.
  4. Run each spec. Each spec runs its own whole dependency tree and comes back as one file set. Each spec opens its own coordinator sessions.
  5. Stack each list. If a list has more than one file set, they are layered per path:
    • one writer: copied;
    • two or more writers: each stacked template votes the conflict resolver it declares for that path, or none;
    • all vote the same resolver with the same config: the resolver is called with every version;
    • otherwise: the template laid last wins.
  6. Record. Every clash goes into file_conflicts (only the last layering pass is kept), and each template's own output paths go into its files list and the managed_files union.

The lists per command are these.

CommandPrevious specsCurrent specs
create of a new templateevery stacked template, savedthe same, plus the new one with no answers
create of a stacked template at a new version (upgrade)every stacked template, savedthe same, that one at the new version, saved answers
create of a stacked template at its version (rerun)every stacked template, savedthe same with every template's answers cleared
updateevery stacked template, savedevery one at its newest (or picked) version, saved answers

Here is the real code for the order and the vote.

sulfone.iridiumcyanprint/src/update/spec.rs
  1. lines 74-93One spec per active record, from its last history entry; the entry's time becomes installed_at.
    cyanprint/src/update/spec.rs:74-93rust
    /// Read specs from .cyan_state.yaml (pure function)
    pub fn get(&self, state: &CyanState) -> Vec<TemplateSpec> {
    state
    .templates
    .iter()
    .filter(|(_, s)| s.active)
    .filter_map(|(key, s)| {
    let (username, template_name) = parse_template_key(key)?;
    let entry = s.history.last()?;
    Some(TemplateSpec::new(
    username,
    template_name,
    entry.version,
    entry.answers.clone(),
    entry.deterministic_states.clone(),
    entry.time,
    ))
    })
    .collect()
    }
  2. lines 164-167The only ordering rule: oldest installed_at first.
    cyanprint/src/update/spec.rs:164-167rust
    /// Sort specs by installation time for consistent LWW ordering
    pub fn sort_specs(specs: &mut [TemplateSpec]) {
    specs.sort_by(|a, b| a.installed_at.cmp(&b.installed_at));
    }
  3. sulfone.iridium/cyanprint/src/run.rs:618-620create sorts the specs it read from the state file.
    cyanprint/src/run.rs:618-620rust
    // Build specs using composable primitives
    let mut prev_specs = manager.get(&state);
    sort_specs(&mut prev_specs);
  4. sulfone.iridium/cyanprint/src/run.rs:644-681Upgrade-by-create rebuilds both lists from state.templates.iter(): a HashMap, never sorted.
    cyanprint/src/run.rs:644-681rust
    TemplateUpdateType::UpgradeTemplate {
    previous_version,
    previous_answers,
    previous_states,
    } => {
    // Upgrade: prev uses old version, curr uses new version
    let target_key = format!("{}/{}", username, template.template.name);
    // Build prev_specs with the OLD version for the target template
    let prev: Vec<_> = state
    .templates
    .iter()
    .filter(|(_, s)| s.active)
    .filter_map(|(key, s)| {
    let (u, t) = parse_template_key(key)?;
    let entry = s.history.last()?;
    if key == &target_key {
    Some(TemplateSpec::new(
    u.to_string(),
    t.to_string(),
    previous_version,
    previous_answers.clone(),
    previous_states.clone(),
    entry.time,
    ))
    } else {
    Some(TemplateSpec::new(
    u.to_string(),
    t.to_string(),
    entry.version,
    entry.answers.clone(),
    entry.deterministic_states.clone(),
    entry.time,
    ))
    }
    })
    .collect();
  5. sulfone.iridium/cyanprint/src/run.rs:762-763Rerun: reset clears answers and deterministic state of every spec, not only the named one.
    cyanprint/src/run.rs:762-763rust
    // Use manager.reset() to clear answers for curr_specs
    let curr = manager.reset(prev.clone());
  6. sulfone.iridium/cyanprint/src/update/orchestrator.rs:191-193update sorts both lists.
    cyanprint/src/update/orchestrator.rs:191-193rust
    // Sort both lists by installation time for consistent LWW ordering
    sort_specs(&mut prev_specs);
    sort_specs(&mut curr_specs);
  7. sulfone.iridium/cyancoordinator/src/operations/composition/operator.rs:632-651Stacking takes resolver votes from the stacked templates only, never from their dependencies.
    cyancoordinator/src/operations/composition/operator.rs:632-651rust
    // Convert root templates to ResolvedDependency for helper functions
    // (preset_answers are not applicable for horizontal layering)
    let root_dependencies: Vec<ResolvedDependency> = root_templates
    .iter()
    .map(|t| ResolvedDependency {
    template: t.clone(),
    preset_answers: HashMap::new(),
    })
    .collect();
    // Build resolver registry from ONLY root templates (horizontal layering scope)
    let registry = Self::build_resolver_registry(&root_dependencies);
    let template_infos = Self::build_template_infos(&root_dependencies);
    // Use resolver-aware layerer
    let layerer = ResolverAwareLayerer::new(registry, template_infos, client.clone(), headless);
    let result = layerer.layer_merge(vfs_list)?;
    // Track conflicts for state writing
    self.file_conflicts = layerer.get_conflicts();

Worked example: ./shop, step 3

Step through stacking ci:2 onto web:5 (step 3 of the shop example). The state shown is the incoming list.

Stacking web:5 and ci:2
  1. Read the stack

    One record, atomi/web. ci is new, so it joins the current list with no answers and the time now.

    previousweb:5 (1 Mar)
    currentweb:5 (1 Mar), ci:2 (9 Mar)
  2. Run each spec

    web:5 runs its tree (docker:7, lint:2, node-base:3, web:5) and comes back as one set; ci:2 asks ci_provider and comes back as another.

    setsweb:5 set, ci:2 set
  3. Path .editorconfig

    Both write it. web:5 declares json-merger only for package.json; ci:2 the same. Both vote none.

    votesnone, none
    ruleall none
    resultci:2's file (laid last)
  4. Path package.json

    Both write it and both declare json-merger:1 with config arrays=concat for it.

    votesjson-merger, json-merger
    ruleagreed
    resultresolver merges both
  5. Path .github/workflows/ci.yml

    Only ci:2 writes it.

    resultcopied
  6. Record

    file_conflicts gets .editorconfig and package.json. The clashes inside web:5's own tree from step 1 drop out of the record.

    file_conflicts.editorconfig, package.json

Why it is built this way

Hard cases

  • Every stacked template runs twice per command. Adding ci to a folder with five templates runs eleven template trees, each with its own sessions.
  • Stack order is the last history time, not the install time. An upgrade adds a new history entry with the time of that run, so an upgraded template moves to the top of the stack on the next run. cyanprint update keeps the old time for the run itself.
  • The stack is unsorted on rerun and upgrade-by-create. Two identical runs can lay web and ci in a different order, so a clash with no agreed resolver can flip.
  • A rerun clears every template's answers, not only the named one's. With the node cache on, the empty-answer keys from the first runs hit and nothing is asked; with it off, every question of every stacked template is asked.
  • Headless answers go only to the template being changed. A flat answer file is never used to answer another stacked template's same-named question.
  • One writer that votes none breaks agreement. If ci:2 did not declare json-merger for package.json, ci:2's whole package.json would win and web's scripts would be gone.
  • update drops the new history entries (see update), so the stack order and answers it should have saved are lost.

Questions for the designer

  1. Should the stack be ordered by first install time (the field is named installed_at) or by last upgrade time (what the code reads)?
  2. Should a rerun clear only the named template's answers?
  3. Should create's upgrade and rerun paths sort the stack like update does?