Doc system test

Three-way merge

How create and update fold new template output into a folder the user has edited, in one git merge of whole file sets, with a rebuilt base.

Meta block front matter of this doc
namespaces
  • durian.cloud/cyanprint/project-lifecycle
  • durian.cloud/cyanprint/composition
type
algorithm
code
  • sulfone.iridium/cyanprint/src/run.rs
  • sulfone.iridium/cyancoordinator/src/fs/merger.rs
  • sulfone.iridium/cyancoordinator/src/fs/loader.rs
  • sulfone.iridium/cyancoordinator/src/fs/mod.rs

The problem. The user has edited files a template wrote. A new template version changes some of the same files. CyanPrint must bring in the template's changes, keep the user's, and show the user where both touched the same line.

The idea. Treat it like a git merge of two branches that started from one commit: the merge base is what the templates wrote last time, one branch is the folder now (local set), the other is what the templates write now (incoming set). The merge base is never stored: it is rebuilt by replaying the templates, which is why runs must be deterministic.

The picture shows the three sets and the one merge.

flowchart LR
  P["previous template specs<br/>(state file: old versions, saved answers)"] --> RB["replay + stack"] --> B[merge base]
  C["current template specs<br/>(new versions, new templates)"] --> RI["run + stack"] --> I[incoming set]
  D["target folder on disk<br/>(gitignore respected)"] --> L[local set]
  B --> G["git merge in a temp repo<br/>branches current and incoming<br/>rename detection 50%"]
  L --> G
  I --> G
  G --> M[merged set]
  M --> DEL["delete: in local, not in merged"]
  M --> W["write merged set to disk"]
Three sets, one merge

How it works

The batch run builds the three sets, then calls the merge once for the whole folder.

  1. Merge base. Every previous template spec is run with its saved version and answers; each gives one file set (its whole dependency tree, already layered). The sets are stacked into one.
  2. Incoming set. Every current template spec is run the same way and stacked.
  3. Local set. The target folder is walked with gitignore rules on (.gitignore, .git/info/exclude, the global ignore file). .git/ and .cyan_state.yaml are skipped.
  4. Merge. In a new temporary git repo:
    1. write the merge base, commit;
    2. create branches current and incoming from that commit;
    3. on current, remove the committed files, write the local set, commit;
    4. on incoming, the same with the incoming set;
    5. merge incoming into current with rename detection at 50 percent likeness;
    6. if lines clash, leave git's conflict markers in the files; otherwise commit;
    7. read every file in the temp folder back as the merged set.
  5. Delete. Every path in the local set that is not in the merged set is deleted from the folder.
  6. Write. Every file of the merged set is written to the folder.
  7. Save. New history entries, managed_files and file_conflicts go into the state file.

Here is the real code for steps 4 to 6. The walk follows batch_process into the merger and back.

sulfone.iridiumcyanprint/src/run.rs
  1. lines 275-288Load the folder, merge the three sets, delete what the merge removed, write the rest.
    cyanprint/src/run.rs:275-288rust
    let local_vfs = operator.load_local_files(target_dir)?;
    let merged_vfs = operator.merge(&prev_vfs, &local_vfs, &curr_vfs)?;
    // Clean up files that were deleted during merge
    let deleted = operator.cleanup_deleted_files(target_dir, &local_vfs, &merged_vfs)?;
    if !deleted.is_empty() {
    crate::hprogress!(
    headless,
    "🗑️ Removed {} file(s) no longer in template",
    deleted.len()
    );
    }
    operator.write_to_disk(target_dir, &merged_vfs)?;
  2. sulfone.iridium/cyancoordinator/src/fs/loader.rs:32-39The local set is a gitignore-aware walk: ignored files never enter the merge.
    cyancoordinator/src/fs/loader.rs:32-39rust
    } else {
    // Otherwise, walk the directory respecting .gitignore
    let walker = WalkBuilder::new(dir)
    .hidden(false) // Process hidden files (except .git which is excluded by default)
    .git_ignore(true) // Respect .gitignore files
    .git_exclude(true) // Respect .git/info/exclude
    .git_global(true) // Respect global gitignore
    .build();
  3. sulfone.iridium/cyancoordinator/src/fs/merger.rs:141-152A temp repo starts from the merge base; two branches fork from it.
    cyancoordinator/src/fs/merger.rs:141-152rust
    println!("🔄 Starting Git folder-level 3-way merge");
    }
    // Create temporary repository with base VFS
    let (repo, temp_dir) = self.create_temp_repo(base)?;
    // Commit the base state
    let base_commit = self.commit_all(&repo, "Base state")?;
    // Create branches for current and incoming
    repo.branch("current", &repo.find_commit(base_commit)?, false)?;
    repo.branch("incoming", &repo.find_commit(base_commit)?, false)?;
  4. sulfone.iridium/cyancoordinator/src/fs/merger.rs:158-186On current: remove the files git tracks, write the local set, commit.
    cyancoordinator/src/fs/merger.rs:158-186rust
    // Checkout current branch and apply current VFS
    let current_branch = repo.find_branch("current", git2::BranchType::Local)?;
    repo.set_head(current_branch.get().name().unwrap())?;
    repo.checkout_head(Some(git2::build::CheckoutBuilder::new().force()))?;
    // Clear the working directory and apply current VFS
    for entry in repo.index()?.iter() {
    let path = std::str::from_utf8(&entry.path).unwrap();
    let full_path = temp_dir.path().join(path);
    if full_path.exists() {
    std::fs::remove_file(full_path)?;
    }
    }
    // Write current VFS to the working directory
    for (path, content) in &current.files {
    let full_path = temp_dir.path().join(path);
    // Create parent directories if needed
    if let Some(parent) = full_path.parent() {
    std::fs::create_dir_all(parent)?;
    }
    // Write file content
    std::fs::write(&full_path, content)?;
    }
    // Commit current state
    let current_commit = self.commit_all(&repo, "Current state")?;
  5. sulfone.iridium/cyancoordinator/src/fs/merger.rs:230-237Rename detection on, threshold from the caller (50); checkout is forced.
    cyancoordinator/src/fs/merger.rs:230-237rust
    // Set up merge options with find_renames enabled
    let mut merge_opts = git2::MergeOptions::new();
    merge_opts.find_renames(true);
    merge_opts.rename_threshold(self.similarity_threshold);
    // Set up checkout options
    let mut checkout_opts = git2::build::CheckoutBuilder::new();
    checkout_opts.force();
  6. sulfone.iridium/cyancoordinator/src/fs/merger.rs:254-268Merge; on conflicts, keep the markers in the working files and do not commit.
    cyancoordinator/src/fs/merger.rs:254-268rust
    // Perform the merge
    repo.merge(
    &[&incoming_annotated],
    Some(&mut merge_opts),
    Some(&mut checkout_opts),
    )?;
    // Check if we have conflicts
    if repo.index()?.has_conflicts() {
    if self.debug {
    println!("⚠️ Merge resulted in conflicts");
    }
    // In case of conflicts, we keep the conflicts in the working directory
    } else {
  7. sulfone.iridium/cyancoordinator/src/fs/mod.rs:115-132Delete = local paths minus merged paths.
    cyancoordinator/src/fs/mod.rs:115-132rust
    fn cleanup_deleted_files(
    &self,
    target_dir: &Path,
    local_vfs: &VirtualFileSystem,
    merged_vfs: &VirtualFileSystem,
    ) -> Result<Vec<PathBuf>, Box<dyn Error + Send>> {
    let local_paths: HashSet<PathBuf> = local_vfs.get_paths().into_iter().collect();
    let merged_paths: HashSet<PathBuf> = merged_vfs.get_paths().into_iter().collect();
    let files_to_delete: Vec<PathBuf> =
    local_paths.difference(&merged_paths).cloned().collect();
    if !files_to_delete.is_empty() {
    self.writer.cleanup(target_dir, &files_to_delete)?;
    }
    Ok(files_to_delete)
    }

What happens to one file

The merge decides per file and per line. This table is every case, read as git reads it.

BaseLocalIncomingResult
absentabsentpresentwritten (new template file)
absentpresentabsentkept (user file)
presentsame as basechangedincoming written
presentchangedsame as baselocal kept
presentchanged linesother lines changedboth changes kept
presentchanged line Xchanged line Xconflict markers
presentsame as baseabsentdeleted (template dropped it)
presentchangedabsentgit modify/delete conflict; see the questions below
absentpresentpresent, differentadd/add conflict markers

Worked example: the shop, step 4

Step through cyanprint update ./shop (web 5 → 6) file by file. The sets are from the shop example.

Merging ./shop during update web 5 to 6
  1. README.md

    Base and incoming changed the same line differently from local, so git writes markers.

    baseRun: npm run dev
    localRun: npm run dev -- --port 3000
    incomingRun: bun dev
    resultconflict markers
  2. jest.config.js

    The user never touched it and web:6 does not make it: the merge drops it and the delete step removes it from disk.

    basepresent
    localsame as base
    incomingabsent
    resultdeleted
  3. notes.md

    Only the user has it.

    baseabsent
    localDeploy on Fridays.
    incomingabsent
    resultkept
  4. Dockerfile

    Only the template changed it (docker:7 to docker:8).

    baseFROM node:22-alpine
    localFROM node:22-alpine
    incomingFROM node:22-bookworm-slim
    resultincoming
  5. app.config.json

    The template's own value changed between runs because deterministic state is not saved. Base B (from cache), local A, incoming D.

    baseinstanceId B
    localinstanceId A
    incominginstanceId D
    resultconflict markers

The tree below is the folder before and after.

./shop around update web 5 to 6
Before
  • app.config.jsonkept
  • Dockerfilekept
  • jest.config.jskept
  • notes.mdkept
  • README.mdkept
After
  • app.config.jsonconflict
  • Dockerfilechanged
  • jest.config.jsremoved
  • notes.mdkept
  • README.mdconflictshow file
    # shop
    
    <<<<<<<
    Run: npm run dev -- --port 3000
    =======
    Run: bun dev
    >>>>>>>
    

Why it is built this way

Hard cases

  • The merge base is only as true as the replay. Any difference between what a template wrote last time and what it writes on replay looks like a template edit. Lost deterministic state is the common cause (client-side state). So is a newer CLI whose layering or resolver rules changed: the replay pins the template, not the CLI.
  • Every stacked template is replayed on every run, not only the one being changed. A rerun of ci rebuilds web too.
  • Ignored files are outside the merge, and get overwritten. The local walk skips gitignored paths, so a template file at an ignored path (say .env) is never in the local set. In the temp repo, add_all also skips files the sets' own .gitignore ignores, so they are never committed; they stay in the temp folder from the last write, which is the incoming set's. The merged set therefore holds the template's .env, and it is written over the user's. Found while reading; not run.
  • Conflict markers go to disk; there is no prompt. A headless run reports done with markers in files.
  • A first create into a non-empty folder has an empty merge base: every template file that already exists with other content is an add/add conflict.
  • The temp repo commits as "Merger". The commits never leave the temp folder; only the working files are read back.
  • Create has no dirty check. cyanprint update checks git status first; cyanprint create (add, upgrade or rerun) does not.

Questions for the designer

  1. Was rebuilding the merge base chosen over storing it for size, trust or portability, knowing the replay must be deterministic?
  2. A file the user changed and the template dropped is a git modify/delete conflict. What lands on disk then was not traced (libgit2 conflict checkout); what should the user see?
  3. Should gitignored paths that templates make (such as .env) be merged, skipped, or written only when absent?
  4. Why 50 percent for rename detection?