Node cache
The on-by-default local cache of one node's output and answers - its key, who may use it, the file format, and the cases where it serves the wrong thing.
The problem. Every create and update runs each stacked template already in the state file twice (its base, then its incoming spec; a new template runs once), and each run starts containers for every node. Most nodes give the same output as last time.
The idea. Before a node runs, hash everything that should decide its output into a cache key. If an entry with that key is on disk, use its files and answers and skip the node. The node cache is on by default, lives on the user's machine, and is shared by every folder that points at the same directory. A directory the cache creates is mode 0700, so another account can use it only if it already existed with wider modes (store.rs:62-69).
The picture shows one node's path through the cache.
flowchart TD
I["node's answer map + deterministic map<br/>(before any question)"] --> E{"cache on and node published?"}
E -->|no| L["run live"]
E -->|yes| K["key = SHA-256 of canonical JSON"] --> G{"entry decodes?"}
G -->|no| L
G -->|yes| U{"archive unpacks?"}
U -->|no| X["evict entry"] --> L
U -->|yes| H["use cached files, replay cached answers<br/>(no questions, no containers)"]
L --> N{"need input?"}
N -->|yes| S["stop; nothing stored"]
N -->|no| W["store files + answers under the SAME key"]
H --> Y["layering always runs after"]
W --> YHow it works
- Turn on or off.
--no-output-cacheorCYANPRINT_NO_CACHEset to1,true,yesoron(any case, trimmed) turns it off. Otherwise it is on for create, update andtryof a group. - Pick the folder.
--cache-dir, elseCYANPRINT_CACHE, else the platform cache folder pluscyanprint(~/.cache/cyanprinton Linux,~/Library/Caches/cyanprinton macOS), else.cache/cyanprint. - Who may be cached. A node with images, a version number above 0, and a version id not starting with
10ca1orlocal-. Groups,trytemplates and dev templates never are;cyanprint testnever uses the cache. - Make the key right after the node's answer map is built, before any question is asked. The key is SHA-256, as 64 lowercase hex characters, over this JSON, in this order:
format_version(1),template_id(version id),template_version(version number);images: blob and template Docker reference and tag;answers: the node's whole answer map (inherited, scoped-stripped, presets), sorted by id;deterministic_states: the shared map, sorted;pluginsandprocessors: each(id, version, docker_reference, docker_tag), sorted.
- Look up. A file named by the key. Any read, length, checksum or JSON fault is a miss.
- Hit. Unpack the archive (on failure: evict and run live), then replay the stored answers into the composition state through the same namespacing path as a live run. No session, no container, no question.
- Miss. Run live. If the node completes, check its archive unpacks, then store the archive and its completed answers under the key from step 4.
- Summary. After the run, one line on stdout:
♻️ Cache: 4/9 nodes served from cache.--debugadds a HIT or MISS line per node, also on stdout.
The entry file is: 32 bytes SHA-256 of the payload, 8 bytes payload length, then the payload: 8 bytes archive length, the tar.gz archive, and the answers as JSON. Writes go to a temp file in the same folder (mode 0600), are synced, then renamed. The folder gets mode 0700 when created and a CACHEDIR.TAG file.
Here is the real code.
- lines 341-351The key is computed from template_answers: the map BEFORE the node's questions.
cyancoordinator/src/operations/composition/operator.rs:341-351rust // Compute the content-addressed key only when the node can be cached,// so a disabled / non-cacheable run pays nothing for keying. (FR5, L5)let cache_key = if self.cache.enabled() && crate::cache::is_cacheable(template) {Some(crate::cache::compute_key(template,&template_answers,&shared_state.shared_deterministic_states,))} else {None}; - lines 383-405Hit: replay the cached answers through namespacing into the shared state; count the hit.
cyancoordinator/src/operations/composition/operator.rs:383-405rust let replay_state = TemplateState::Complete(Cyan {processors: Vec::new(),plugins: Vec::new(),},entry.state,);let namespaced = namespace_template_state(&replay_state,namespace,&initial_shared_state.shared_answers,);shared_state.update_from_template_state(&namespaced,template.principal.id.clone(),)?;served_from_cache = true;// The entry was genuinely served: count it now, AFTER// the archive unpacked and state replayed, so a// poisoned entry that fell back to execution is not// reported as a served hit. (FR15, H2)self.cache.record_hit();} - lines 406-416An entry that decodes but will not unpack is evicted; the node runs live.
cyancoordinator/src/operations/composition/operator.rs:406-416rust Err(e) => {tracing::debug!("cache hit for {} unpacked invalid, evicting and re-executing: {e}",template.principal.id);self.cache.evict(key);// Do NOT record a hit: `lookup` already counted this// node toward the *attempt* total, but it was NOT// served from cache. Fall through to live execution.}} - lines 486-509Miss: unpack first, then store the archive and the COMPLETED answers under the pre-question key.
cyancoordinator/src/operations/composition/operator.rs:486-509rust // Validate the archive can actually unpack BEFORE caching it, so a// bad (Ok((invalid_bytes, Complete(..)))) result is never persisted.// (FR13, C1) The to-store copy is taken only when this node will be// cached, so a disabled / non-cacheable run never clones the full// archive bytes. (L16) `unpack_archive` consumes the bytes by value,// so the clone must precede the move.let to_store = cache_key.as_ref().map(|_| archive_data.clone());let vfs = self.template_operator.vfs.unpack_archive(archive_data)?;vfs_outputs.push(vfs);// Store ONLY a non-interactive Complete result, and only after the// archive has unpacked cleanly. The raw (un-namespaced) answers are// stored so a later replay can re-namespace per consuming dependency.// QnA panics upstream, Err short-circuits via `?` above, and a NeedInput// already returned, so failures/partials are never cached. (FR13)if let (Some(key), TemplateState::Complete(_, ref answers)) =(cache_key.as_ref(), &template_state){let entry = CacheEntry {archive: to_store.expect("to_store is Some iff cache_key is Some"),state: answers.clone(),};self.cache.store(template, key, &entry);} - sulfone.iridium/cyancoordinator/src/cache/key.rs:61-71The key material, in serialization order.
cyancoordinator/src/cache/key.rs:61-71rust #[derive(Serialize)]struct CacheKeyMaterial<'a> {format_version: u32,template_id: &'a str,template_version: i64,images: Option<TemplateImages<'a>>,answers: BTreeMap<&'a str, &'a Answer>,deterministic_states: BTreeMap<&'a str, &'a str>,plugins: Vec<PinnedArtifact<'a>>,processors: Vec<PinnedArtifact<'a>>,} - sulfone.iridium/cyancoordinator/src/cache/mod.rs:101-119Eligibility: images, version above 0, not a synthetic id.
cyancoordinator/src/cache/mod.rs:101-119rust pub fn is_cacheable(template: &TemplateVersionRes) -> bool {if template.principal.properties.is_none() {return false; // dev mode / group template — no execution artifacts}if template.principal.version <= 0 {return false; // unpublished / synthetic}let id = &template.principal.id;if id.starts_with("10ca1") || id.starts_with("local-") {// Synthetic local template ids produced by `build_synthetic_template`// (try_cmd.rs). `10ca1` is the synthetic-id prefix; `local-` the dev// prefix. A real published template whose registry id happened to start// with `10ca1` would be excluded here too — that is a fail-safe (it would// re-execute, never serve stale), and the only `10ca1` producer is// synthetic ids, so the false-negative risk is vanishing. (FR14, L19)return false;}true} - sulfone.iridium/cyancoordinator/src/cache/store.rs:201-223Write: key check, folder and marker, temp file 0600, fsync, rename, parent fsync. Errors are swallowed.
cyancoordinator/src/cache/store.rs:201-223rust fn put_inner(&self, key: &str, entry: &CacheEntry) -> std::io::Result<()> {// Validate the key BEFORE any filesystem work so an invalid key never// creates the directory or a temp file. (security)let dest = self.entry_path(key)?;self.ensure_dir()?;let encoded = Self::encode(entry);// Temp file in the SAME dir so the rename is atomic (same filesystem). (FR8)let mut tmp = tempfile::NamedTempFile::new_in(&self.dir)?;set_file_perms_0600(tmp.path())?;tmp.write_all(&encoded)?;tmp.flush()?;tmp.as_file().sync_all()?;tmp.persist(&dest).map_err(|e| std::io::Error::other(format!("persist failed: {e}")))?;// fsync the parent dir so the rename is durable.if let Ok(dir) = fs::File::open(&self.dir) {let _ = dir.sync_all();}Ok(())}
Worked example: the shop, with the cache on
Step through what the cache holds and serves across the shop example. Keys are named K1, K2, …; real ones are 64 hex characters.
- Step 1: create web:5
Four misses. Each key holds the answers before that node's questions; each entry holds the answers after.
K1 docker:7 key {} → stores node_versionK2 lint:2 key node_versionK3 node-base:3 key node_version, module_typeK4 web:5 key node_version, module_type → stores all 5, file Asummary 0/4 - Step 3: base web:5
Replayed with all five saved answers: new keys, so four misses. web:5 runs live and makes instance id B.
K5-K8 key has all 5 answers; web:5 file B - Step 3: incoming web:5 and ci:2
Same keys as the base: four hits, B again. ci:2 misses with key {} (K9). Base equals incoming, so the folder's A is kept - by luck.
summary 4/9app.config.json A kept - Step 3, cache off
Base makes B and incoming makes C live. Both differ from A on one line: conflict markers.
app.config.json conflict - Step 5: update again
Step 4 stored web:6 under a key without runtime. The lost history replays the same key: a hit, so runtime bun comes back unasked.
web:6 hit, runtime not asked - Step 6: rerun ci:2
Every answer is cleared, so web:5's nodes are looked up with key {} again: K1 to K4 hit. ci:2 hits K9. Nothing is asked.
questions noneweb:5 file A from step 1 - Step 7: create ./blog
Another folder, same keys K1 to K4: blog gets name shop, port 3000 and instance A, unasked.
summary 4/4./blog name shop
The tree below is the cache folder after step 1, on Linux.
- cyanprint/
CACHEDIR.TAGaddedownership markershow file
Signature: 8a477f597d28d172789f06886806bc55 # This file marks this directory as a cyanprint execution-output cache. # For information about cache directory tags see https://bford.info/cachedir/- K1addeddocker:7, key answers {}
- K2addedlint:2
- K3addednode-base:3
- K4addedweb:5: files with instance A, answers incl. name shop
Why it works this way
Hard cases
- The cache hides lost determinism. Deterministic state is not saved, so a live replay makes new values; a hit returns the old ones. Step 3 is clean only because base and incoming hit the same entry. Anyone with the cache off (CI, a new machine) gets conflict markers there.
- No lock. Two runs that miss the same key both run the node; the last rename wins. Readers never see a half-written file.
- No eviction. No size limit, age limit or LRU.
cyanprint cache clearremoves only digest-named files, and only if the folder'sCACHEDIR.TAGmatches CyanPrint's text exactly. A corrupt entry stays until a store replaces it. - Stored before success. A node is stored before layering; if a conflict resolver call later fails, the entry stays.
- Mutable tags. The key holds image reference and tag, not digest. A re-pushed tag is not noticed.
- A cache hit warms no conflict resolver. Conflict resolver containers start only in the coordinator's template warm (sulfone.boron/docker_executor/template_executor.go:387-420), and the CLI runs template warm only for a miss (sulfone.iridium/cyancoordinator/src/template/executor.rs:93-99, reached from operator.rs:426-447). In a run where every node hits (shop step 6), the agreed stacking call still goes through the proxy to
cyan-resolver-<id>; if no earlier run left that container up, the proxy answers 502 (server.go:758-768) and the run fails. Inferred from the code; found by reading; not run. - Hits bring unpruned answers into headless runs. An entry made interactively stores every seeded and typed answer (engine.rs:84-86; operator.rs:501-507). A headless hit replays them all into the state (operator.rs:383-398) and so into the state file: a second path for the pruning bug.
- The key includes a map headless runs ignore. The key hashes the shared deterministic map (operator.rs:343-348), but
start_headlesstakes answers only (executor.rs:208-214). Harmless today only because saved maps are always empty. - Counts are attempts. The summary counts cache lookups of eligible nodes, not all nodes; a hit is counted only after it unpacked and replayed.
- A bad cached answer fails the run. If the replayed answers clash in type with the shared state, the run fails; the entry is not evicted.
Questions for the designer
- Should the key include the node's answers after its questions, or should a hit still walk the questions? (The successor sulfone.lite keys a processor cache on inputs only: artifact integrity, config and input files; lite docs/requirements/simplification-spec.md, lines 171-177.)
- Should headless runs (and reruns) bypass entries made interactively?
- Should the cache have a size or age limit? (sulfone.lite plans age and size eviction, line 177.)
- Should the key use image digests instead of tags?
- Should the cache store each node's deterministic values, so a cache-off replay matches a cache-on one? (sulfone.lite persists deterministic state in the state file: simplification-spec.md, lines 78-79.)