Doc system test

Session release

The nested owners that make sure every coordinator session a create or update opens is released, on success, need input and failure.

Meta block front matter of this doc
namespaces
  • durian.cloud/cyanprint/composition
  • durian.cloud/cyanprint/coordinator
  • durian.cloud/cyanprint/project-lifecycle
  • durian.cloud/cyanprint/prompting
type
guard
code
  • sulfone.iridium/cyancoordinator/src/template/executor.rs
  • sulfone.iridium/cyancoordinator/src/operations/composition/operator.rs
  • sulfone.iridium/cyanprint/src/run.rs
  • sulfone.iridium/cyanprint/src/headless.rs
  • sulfone.iridium/cyanprint/src/main.rs
  • sulfone.iridium/cyanprompt/src/domain/services/template/engine.rs
issues
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+session-leak-on-user-abort
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+double-release-on-build-send-error

The problem. Every node that runs live opens a coordinator session: processor, plugin and merger containers plus a work volume. A create of web:5 opens four; a stack of two templates opens up to ten. Any of a dozen later steps can fail, and a lost session id means containers left running until the coordinator's own cleanup, hours later.

The idea. At every moment exactly one owner holds each session id, and each owner releases what it holds if it is dropped while still armed. On success or need input, an owner disarms and hands the ids up to the next owner; the last one releases them at the command's end.

The picture shows who owns a session id, and when.

stateDiagram-v2
  [*] --> Executor: warm creates the session
  Executor --> Released: executor error (warm status, bootstrap, Q&A, build)
  Executor --> Composition: node completes or needs input
  Composition --> Released: later node fails, answer type clash, unpack or layering error
  Composition --> Batch: composition returns (disarm)
  Batch --> Released: stacking, merge, write or history save fails
  Batch --> Run: batch returns (take)
  Run --> Released: state save or post-template command fails
  Run --> Boundary: done or need_input (take)
  Boundary --> Released: headless or interactive cleanup
  Released --> [*]
Who owns a session id

How it works

Each owner is a guard: it holds a list of session ids and releases them all on drop unless disarmed. A release is a best-effort DELETE /executor/{session} on the coordinator; errors are ignored.

  1. The executor (one node). Warm and executor warm run in parallel; the executor warm's result is read first, so its real session id is known even when template warm fails. Every error after that (template warm, bootstrap, Q&A error, build send or status) cleans that session before returning Err. On Complete or need input it returns the id.
  2. The composition guard (one tree). Each non-empty id the executor returns is pushed. A ? anywhere later in the tree (another node's executor, an answer type clash, an archive that will not unpack, a cached-answer clash, a conflict resolver call during dependency layering) drops the guard and releases every id it holds. On need input or success it disarms and returns the ids.
  3. The batch guard (one create or update). Each composition's ids are added as they come back, for every previous and current template spec. Stacking, the three-way merge, the deletes, the write and the history save all run under it. It hands the ids up on need input or success.
  4. The run guard (after the batch). A fresh guard takes the ids for the state file save and the post-template commands, and hands them up on done.
  5. The boundary. Headless: finish_headless releases every id, then prints the envelope (done or need_input). Interactive: main prints Cleaning up session: … for each id and releases it. On Err the boundary has no ids; the guards below have already released them.

A node cache hit opens no session at all.

Here is the real code, owner by owner.

sulfone.iridiumcyancoordinator/src/template/executor.rs
  1. lines 116-135Executor: read the executor warm first, so cleanup knows the real session id.
    cyancoordinator/src/template/executor.rs:116-135rust
    // Read the executor warm result FIRST. `warm_template` and `warn_executor` run
    // concurrently, so one can succeed (creating an executor session) while the other
    // fails. Reading the executor result before the template result means that when
    // `template_warm` errors below, the executor session id is already available to
    // `clean_warmed_session` — otherwise the `?` on the template result would
    // propagate before cleanup runs, leaking the already-created executor session.
    let executor_warm = rx12.blocking_recv().unwrap()?;
    // The warmed executor session was created during Phase 1 (warm/bootstrap). The
    // success and NeedInput paths return `actual_session_id` so the CLI can clean it
    // fire-and-forget. Any later error path (concurrent warm failure, warm-status,
    // bootstrap, Q&A, or build failure) returns `Err` WITHOUT surfacing the session
    // id, so the warmed session would be left dangling (only the 12-hour coordinator
    // auto-cleanup would reclaim it). Clean it best-effort right here on those error
    // paths so a failed run does not leak a coordinator session/volume. Cleanup
    // failures are ignored — they must not mask the real error.
    let clean_warmed_session = || {
    let client = CyanCoordinatorClient::new(self.coordinator_endpoint.clone());
    let _ = client.clean(executor_warm.session_id.clone());
    };
  2. lines 331-345A build failure cleans the session; success returns the id to the caller.
    cyancoordinator/src/template/executor.rs:331-345rust
    // A build-phase failure (non-success build response or body-read error) returns
    // `Err` without surfacing the session id — clean the warmed session before
    // propagating so the run does not leak a coordinator session/volume.
    let response = match response {
    Ok(bytes) => bytes,
    Err(e) => {
    clean_warmed_session();
    return Err(e);
    }
    };
    // Return the actual session ID used for this execution
    let actual_session_id = executor_warm.session_id.clone();
    Ok((response, prompter_state, actual_session_id))
  3. sulfone.iridium/cyancoordinator/src/operations/composition/operator.rs:45-68Composition guard: holds ids, disarm hands them over.
    cyancoordinator/src/operations/composition/operator.rs:45-68rust
    struct CompositionSessionGuard {
    client: Option<CyanCoordinatorClient>,
    sessions: Vec<String>,
    }
    impl CompositionSessionGuard {
    fn new(client: Option<CyanCoordinatorClient>) -> Self {
    Self {
    client,
    sessions: Vec::new(),
    }
    }
    /// Record a coordinator session id allocated during a dependency's warm/bootstrap.
    fn push(&mut self, session_id: String) {
    self.sessions.push(session_id);
    }
    /// Hand the accumulated session ids to the caller on a legitimate terminal state
    /// (success or need_input). The guard no longer releases them on drop.
    fn disarm(mut self) -> Vec<String> {
    std::mem::take(&mut self.sessions)
    }
    }
  4. sulfone.iridium/cyancoordinator/src/operations/composition/operator.rs:70-81Drop while armed: release every id, ignore errors.
    cyancoordinator/src/operations/composition/operator.rs:70-81rust
    impl Drop for CompositionSessionGuard {
    fn drop(&mut self) {
    if let Some(client) = &self.client {
    for sid in &self.sessions {
    // Best-effort: a release hiccup must not mask the original error that
    // triggered the early return. Matches the closure-based teardown in
    // cyanprint (e.g. `cleanup_group_sessions`).
    let _ = client.try_cleanup(sid);
    }
    }
    }
    }
  5. sulfone.iridium/cyancoordinator/src/operations/composition/operator.rs:449-453Each live node's id is pushed right after the executor returns.
    cyancoordinator/src/operations/composition/operator.rs:449-453rust
    // Track session for cleanup regardless of outcome (the session was
    // created during warm/bootstrap even when Q&A stops early).
    if !actual_session_id.is_empty() {
    sessions.push(actual_session_id);
    }
  6. sulfone.iridium/cyanprint/src/run.rs:152-170Batch guard: add each composition's ids; on need input, take them and return.
    cyanprint/src/run.rs:152-170rust
    let (vfs, final_state, session_ids, _commands) = operator.execute_template(
    &template_res,
    &spec.answers,
    &spec.deterministic_states,
    headless,
    )?;
    session_guard.extend(session_ids);
    // Headless: a (re-created) prev template stopped on an unanswered question.
    // Surface it immediately; no files are written. Hand the sessions to the caller
    // (which cleans them at the headless boundary) by disarming via `take`.
    if let Some(question) = final_state.need_input {
    return Ok((
    session_guard.take(),
    Vec::new(),
    Vec::new(),
    HashMap::new(),
    Some(question),
    ));
    }
  7. sulfone.iridium/cyanprint/src/run.rs:845-851Run guard: re-armed for the state save and commands.
    cyanprint/src/run.rs:845-851rust
    // The run is committed, but several fallible steps remain (state save, post-template
    // commands) BEFORE the session ids reach `finish_headless`. A `?`/early `return Err`
    // from any of them would drop the ids and leak the coordinator sessions. Hand them to a
    // cleanup guard so every error path below releases them; `take()` disarms it on the
    // happy `done` return (where `finish_headless` then cleans normally).
    let mut session_guard =
    SessionCleanupGuard::new(|sid: &str| release_session(&coord_client, sid), session_ids);
  8. sulfone.iridium/cyanprint/src/headless.rs:211-234Headless boundary: clean the ids, then print one envelope.
    cyanprint/src/headless.rs:211-234rust
    pub fn finish_headless<W, F>(
    r: Result<CyanRunResult, Box<dyn Error + Send>>,
    writer: &mut W,
    clean_sessions: F,
    ) -> Result<(), Box<dyn Error + Send>>
    where
    W: std::io::Write,
    F: FnOnce(&[String]),
    {
    use cyanprompt::domain::models::headless::{HeadlessEnvelope, QuestionWire};
    let env = match r {
    Ok(result) => {
    clean_sessions(&result.session_ids);
    match result.need_input {
    Some(question) => HeadlessEnvelope::NeedInput {
    question: QuestionWire::from(&question),
    },
    None => HeadlessEnvelope::Done,
    }
    }
    Err(e) => HeadlessEnvelope::error(e.to_string()),
    };
    emit_to(writer, &env).map_err(|e| Box::new(e) as Box<dyn Error + Send>)
    }
  9. sulfone.iridium/cyanprint/src/main.rs:447-462Interactive boundary: clean each id; on error there are none to clean.
    cyanprint/src/main.rs:447-462rust
    match r {
    Ok(result) => {
    println!("Completed successfully");
    let coord_client = CyanCoordinatorClient::new(coordinator_endpoint.clone());
    println!("Cleaning up all sessions...");
    for sid in result.session_ids {
    println!("Cleaning up session: {sid}");
    let _ = coord_client.clean(sid);
    }
    println!("Cleaned up all sessions");
    }
    Err(e) => {
    eprintln!("Error: {e:#?}");
    println!("No sessions to clean up");
    }
    }

Worked example: step 3 of the shop

Step through cyanprint create atomi/ci:2 ./shop (step 3 of the shop example), with the cache on. Session ids are written s1 to s5.

Sessions of one stacking create
  1. Base web:5

    Four cache misses: each node runs live and its session goes to the composition guard, then to the batch guard.

    composition(returned)
    batchs1, s2, s3, s4
  2. Incoming web:5

    Four cache hits: no session.

    batchs1, s2, s3, s4
  3. ci:2

    One live node.

    batchs1, s2, s3, s4, s5
  4. Stacking package.json

    Both vote json-merger; the call goes through the coordinator proxy.

    batchs1 to s5 (armed)
  5. If the conflict resolver times out

    The proxy returns 502 after 30 seconds; the error returns through the batch run; its guard drops and releases all five. Interactive main prints the error.

    releaseds1 to s5
    files writtennone
  6. If it works

    Merge and write, then the run guard holds the ids through the state save. The boundary releases them.

    run guards1 to s5
    endCleaning up session: s1 … s5

Step through a failure inside one tree: step 1, if web:5's build call fails.

A build failure in web:5's tree
  1. docker:7, lint:2, node-base:3

    Three live nodes complete.

    composition guards1, s2, s3
  2. web:5 build fails

    The executor cleans its own session s4 and returns Err.

    releaseds4
  3. Composition returns Err

    Its guard drops while armed.

    releaseds1, s2, s3, s4

Why it works this way

Hard cases

  • User abort exits the process. In interactive Q&A, pressing escape at a node's first question with no answers yet calls exit(0). No guard runs, so every session opened so far, and the current node's, stay open until the coordinator's auto-cleanup (12 hours per the executor comment).
  • Panics. An unwrap or panic inside the executor fires before its id reaches any guard, so that node's session leaks. Guards above still release theirs as the panic unwinds.
  • Executor warm itself fails. No id comes back; if the coordinator made a session anyway, nothing can release it.
  • A build send error releases the session twice. The failed send cleans the session in its error mapping (executor.rs:308-311), the error then reaches the match on the response, and its Err arm cleans the same session again (executor.rs:334-339): two DELETE calls for one id. Release is best-effort, so it is harmless; the guard text above says "cleans that session" once. Found while reading; not run.
  • Release is synchronous. Each release is a blocking HTTP call (600 second client) despite "fire-and-forget" comments; a slow coordinator slows every exit path.
  • Need input releases too. A headless run that stops for a question still opened, warmed and releases sessions for every node it reached, on every round.

Questions for the designer

  1. Should each node's session be released as soon as its archive is in hand, instead of at the end?
  2. Should user abort return an error instead of exiting, so the guards release sessions?
  3. Should releases run in the background, so a slow coordinator does not delay exits?