Doc system test

Composition run

How one template version's whole dependency tree runs - flatten, then per node namespacing, presets, cache, live run and answer carry-over, then layering.

Meta block front matter of this doc
namespaces
  • durian.cloud/cyanprint/composition
  • durian.cloud/cyanprint/coordinator
  • durian.cloud/cyanprint/prompting
  • durian.cloud/cyanprint/project-lifecycle
  • durian.cloud/cyanprint/authoring
type
flow
code
  • sulfone.iridium/cyancoordinator/src/operations/composition/operator.rs
  • sulfone.iridium/cyancoordinator/src/template/executor.rs
  • sulfone.iridium/cyanprint/src/run.rs

The problem. A template version is not one program but a dependency tree of them. Each node asks its own questions in its own container, and the user must get one folder's worth of files and one set of answers.

The idea. A composition run turns the tree into a run order, runs the nodes one after another while one composition state carries answers forward, then lays all outputs into one file set. The batch run calls it once per template spec. A stacked template already in the state file is run twice (its base, then its incoming spec); a template that is new to the project runs once.

The picture shows one composition run.

flowchart TD
  T["template version + caller answers + deterministic map"] --> F["dependency flattening: run order"]
  F --> N{next node}
  N -->|"group (no images)"| G["add to order, skip"] --> N
  N -->|"has images"| A["build its answers:<br/>namespacing, then presets"]
  A --> K{"cache hit?"}
  K -->|yes| H["cached files + answers"]
  K -->|no| R["run live: warm, Q&A, build<br/>(opens a session)"]
  R --> Q{"need input?"}
  Q -->|yes| STOP["stop: empty file set,<br/>the question, the sessions"]
  Q -->|no| S["store in cache"]
  H --> C["carry answers into the state"]
  S --> C --> N
  N -->|"none left"| L["dependency layering"] --> OUT["one file set, final state,<br/>sessions, commands"]
One composition run

How it works

  1. Flatten. Dependency flattening fetches every node from the registry and returns the run order: children first, siblings by version id, root last, each with its preset answers.
  2. Start the state. A copy of the caller's answers and deterministic map. The clash list from any earlier run is cleared.
  3. Namespacing on or off. On when more than one node has images (answer namespacing).
  4. For each node, in order:
    1. Group: added to the order and skipped. It makes no files and opens no session.
    2. Answers: globals and this node's scoped answers from the state, then presets for ids still missing.
    3. Cache: if the node cache is on and the node is published, compute the key and look up. A hit gives files and answers with no container.
    4. Live: otherwise run the node on the coordinator: template warm and session warm in parallel, bootstrap in parallel with the question walk, then the build. The session id is kept for release (session release).
    5. Need input (headless only): stop here. No later node runs, no layering, an empty file set; the question goes back with every session so far.
    6. Store a completed live node in the cache, then carry its answers into the state (answer carry-over).
  5. Layer every output in run order (dependency layering), recording each file clash.
  6. Collect commands from every node in run order, groups included.
  7. Return the file set, the final state, the session ids and the commands.

Nodes run strictly one at a time; the only parallel work is inside one node. Each node's container work goes through the coordinator; the CLI holds all state.

Step through one live node as calls between the CLI and the coordinator (the coordinator side is in its own docs).

One live node: docker:7 in step 1
CLI
composition run
Coordinator
:9000
docker:7 template
shared container
template warm + session warm

1/5In parallel. Session warm returns the session id.

Here is the real code of the loop.

sulfone.iridiumcyancoordinator/src/operations/composition/operator.rs
  1. lines 591-604The entry: flatten, start the state, run the composition, collect commands.
    cyancoordinator/src/operations/composition/operator.rs:591-604rust
    let dependencies = self.dependency_resolver.resolve_dependencies(template)?;
    let shared_state = CompositionState {
    shared_answers: answers.clone(),
    shared_deterministic_states: deterministic_states.clone(),
    execution_order: Vec::new(),
    need_input: None,
    };
    let (vfs, final_state, session_ids) =
    self.execute_composition(&dependencies, &shared_state, headless)?;
    let commands = Self::collect_commands(&dependencies);
    Ok((vfs, final_state, session_ids, commands))
    }
  2. lines 195-211A copy of the state, an empty output list, the session guard, and a cleared clash list.
    cyancoordinator/src/operations/composition/operator.rs:195-211rust
    let mut shared_state = initial_shared_state.clone();
    let mut vfs_outputs = Vec::new();
    // Coordinator sessions are allocated during each dependency's warm/bootstrap
    // (recorded as `actual_session_id`) and are normally returned to the caller for
    // release. But several fallible steps run AFTER a session is recorded and BEFORE
    // this vector reaches the caller — `update_from_template_state` (the need_input
    // branch and the completed branch), `unpack_archive`, and resolver/default VFS
    // layering. A `?` from any of them discards the locally accumulated sessions,
    // leaking the Boron sessions until server-side expiry. The guard releases every
    // accumulated session on ANY early return while armed; the success paths
    // `disarm()` it and move the ids out. This is the RAII fix for the recurring
    // composition-internal session-leak class — one mechanism covers every current
    // and future `?` between acquisition and return, instead of per-path patching.
    let mut sessions = CompositionSessionGuard::new(self.client.clone());
    // Clear previous conflicts
    self.file_conflicts.clear();
  3. lines 240-257Groups are only added to the order.
    cyancoordinator/src/operations/composition/operator.rs:240-257rust
    for dep in dependencies {
    let template = &dep.template;
    // Check if template has execution artifacts (properties field)
    if template.principal.properties.is_none() {
    cprogress!(
    headless,
    "⏭️ Skipping template: {}/{} (v{}) - no execution artifacts (group template)",
    template.template.name,
    template.template.name, // TODO: Need username
    template.principal.version
    );
    // Update execution order tracking even for skipped templates
    shared_state
    .execution_order
    .push(template.principal.id.clone());
    continue;
    }
  4. lines 426-447A miss runs the node live: warm, questions, build, with this node's answers and the shared deterministic map.
    cyancoordinator/src/operations/composition/operator.rs:426-447rust
    if !served_from_cache {
    // CACHE MISS: execute as today. (FR3) The "Executing" line lives
    // here (not before the cache check) so a cache HIT is never
    // reported as an execution; a non-cached run still emits it for
    // every node. Routed through `cprogress!` so under headless it is
    // suppressed (the single-JSON-on-stdout contract); interactive is
    // unchanged.
    cprogress!(
    headless,
    "🚀 Executing template: {}/{} (v{})",
    template.template.name,
    template.template.name, // TODO: Need username
    template.principal.version
    );
    let (archive_data, template_state, actual_session_id) =
    self.template_operator.template_executor.execute_template(
    template,
    &session_id,
    Some(&template_answers),
    Some(&shared_state.shared_deterministic_states),
    )?;
  5. lines 466-484Need input: record the question and return at once with an empty file set and the sessions.
    cyancoordinator/src/operations/composition/operator.rs:466-484rust
    let need_input = matches!(
    template_state,
    cyanprompt::domain::services::template::states::TemplateState::NeedInput(_, _)
    );
    if need_input {
    let namespaced = namespace_template_state(
    &template_state,
    namespace,
    &initial_shared_state.shared_answers,
    );
    shared_state
    .update_from_template_state(&namespaced, template.principal.id.clone())?;
    // need_input is a legitimate terminal state, not an error: the caller
    // owns the session release from here, so disarm the guard and hand the
    // accumulated ids back. (`update_from_template_state` above can still
    // `?` — in that case the guard stays armed and releases the sessions.)
    let all_session_ids = sessions.disarm();
    return Ok((VirtualFileSystem::new(), shared_state, all_session_ids));
    }
  6. lines 511-522A completed node's answers join the state.
    cyancoordinator/src/operations/composition/operator.rs:511-522rust
    // Update shared state with results (caller-targeted answers namespaced
    // in a multi-template composition so they do not collide with another
    // dependency's; global/derived answers stay raw and propagate). The
    // "was this caller-targeted?" decision uses the immutable initial caller
    // input, not the evolving accumulator — see `namespace_template_state`.
    let namespaced = namespace_template_state(
    &template_state,
    namespace,
    &initial_shared_state.shared_answers,
    );
    shared_state
    .update_from_template_state(&namespaced, template.principal.id.clone())?;
  7. lines 534-553After the loop: layer with resolver votes and keep the clashes.
    cyancoordinator/src/operations/composition/operator.rs:534-553rust
    } else if let Some(ref client) = self.client {
    // Use resolver-aware layering
    // Vertical layering: collect resolvers from ALL templates in dependency tree
    let registry = Self::build_resolver_registry(dependencies);
    let template_infos = Self::build_template_infos(dependencies);
    let layerer =
    ResolverAwareLayerer::new(registry, template_infos, client.clone(), headless);
    let result = layerer.layer_merge(&vfs_outputs)?;
    // Track conflicts for state writing
    self.file_conflicts = layerer.get_conflicts();
    result
    } else {
    // Use default layering (LWW)
    self.vfs_layerer.layer_merge(&vfs_outputs)?
    };
    Ok((layered_vfs, shared_state, sessions.disarm()))
  8. lines 699-711Commands from every node in run order.
    cyancoordinator/src/operations/composition/operator.rs:699-711rust
    /// Collect commands from resolved dependencies in post-order.
    /// Iterates over dependencies (already in post-order from resolve_dependencies),
    /// collects non-empty commands from each template, and flattens into a single vec.
    pub fn collect_commands(dependencies: &[ResolvedDependency]) -> Vec<String> {
    let mut commands = Vec::new();
    for dep in dependencies {
    let template_commands = &dep.template.commands;
    if !template_commands.is_empty() {
    commands.extend(template_commands.iter().cloned());
    }
    }
    commands
    }

Worked example: step 1 of the shop

Step through cyanprint create atomi/web:5 ./shop (interactive, cache on, empty cache), from the shop example.

Composition run of web:5
  1. Flatten

    Four nodes, all with images: namespacing is on.

    orderdocker:7, lint:2, node-base:3, web:5
    state(empty)
  2. docker:7

    Key with no answers: miss. Runs live as session s1; node_version is asked (22). Stored as K1.

    statenode_version
    sessionss1
    cache0/1
  3. lint:2

    Miss. Runs live (s2); asks nothing.

    statenode_version
    sessionss1, s2
    cache0/2
  4. node-base:3

    Preset module_type esm fills the gap. Miss; runs live (s3); asks nothing.

    statenode_version, module_type
    sessionss1 to s3
    cache0/3
  5. web:5

    Miss; runs live (s4). name, port and features are asked; d.get makes instance id A.

    statenode_version, module_type, name, port, features
    sessionss1 to s4
    cache0/4
  6. Layer

    package.json: no consensus, web:5's wins. .gitignore: all none, node-base:3's wins.

    files12 paths
    clashespackage.json, .gitignore
  7. Return

    The batch run merges the set into ./shop and saves the five answers; sessions are released at the end.

    commandsnpm install (from node-base:3)

Step through the same run in headless mode with an empty answers file and an empty cache (with step-1 entries on the machine, docker:7's key hits and the whole tree is served from the cache instead).

Need input stops the run
  1. docker:7

    Runs live (s1). node_version has no answer: need input.

    need_input7a90…/node_version
    sessionss1
  2. Stop

    lint:2, node-base:3 and web:5 never run; nothing is layered or cached.

    file setempty
    returnedquestion, s1

Why it is built this way

Hard cases

  • A template already in the state file runs its whole tree twice per command (base and incoming); a new one (step 1 web:5, step 3 ci:2) runs once. With the cache off, a create into a folder with web and ci runs nine nodes live.
  • Need input wastes work. Every node before the question has warmed, been asked, built and possibly been stored in the cache; the next round repeats it from the start.
  • A tree of only groups makes an empty file set and a "No execution artifacts produced" line.
  • The progress line names the template twice: Executing template: web/web (v5), since the code has no user name at hand.
  • Commands come from every node, groups included, but only from the current specs: the base run's commands are dropped because they "would be duplicates or stale" (run.rs, lines 337-338).
  • Clashes from this run are kept only until the next layering pass of the same command (template stacking replaces them).

Questions for the designer

  1. Should a headless need input keep the work of earlier nodes, instead of redoing it next round?
  2. Should independent nodes (no answer flows between them) run in parallel?