Doc system test

Conflict resolvers

One resolver artifact, called from two places with two different rules - the coordinator between processors, the CLI between template versions.

Meta block front matter of this doc
namespaces
  • durian.cloud/cyanprint/authoring
  • durian.cloud/cyanprint/composition
  • durian.cloud/cyanprint/coordinator
  • durian.cloud/cyanprint/project-lifecycle
  • durian.cloud/cyanprint/registry
type
design
related
issues
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+bad-glob-differs-cli-merger

The problem. Many writers produce package.json: two processors inside one template version, four nodes of web:5's tree, and two stacked templates in ./shop. Keeping only the last one throws away scripts and dependencies; the authors need a way to join the versions.

The idea. A conflict resolver is an artifact (a container on port 5553) that takes every version of one file and returns one. It is called from two places with two different rules: the coordinator's merger, between the processors of one template version, and the CLI, between template versions.

The picture shows the three clashes in the shop and who settles each.

flowchart TD
  subgraph COORD["coordinator: merger container (one node's build)"]
    P1["shop-render:1, invocation 0"] --> PM{"processor output merge"}
    P2["shop-render:1, invocation 1"] --> PM
  end
  subgraph CLI["CLI"]
    DL{"dependency layering<br/>(nodes of one tree)"}
    TS{"template stacking<br/>(stacked templates)"}
  end
  PM -->|"one matching declaration"| R(("json-merger:1<br/>shared container :5553"))
  DL -->|"all writers agree"| R
  TS -->|"all writers agree"| R
  PM -->|"web:5's file set"| DL
  DL -->|"web:5's tree set"| TS
Three kinds of clash, two callers

The two rules

Coordinator (processor output merge)CLI (dependency layering, stacking)
A clash is2+ processors of one node write a path2+ nodes, or 2+ stacked templates, write a path
Declarations usedthe node's own resolverseach writer's own (stacking: stacked templates only)
Picked byevery declaration with a matching globeach writer votes its first match; all must agree
No matchlast processor winslast layer wins
Two or more matchesthe build failslast layer wins, recorded as ambiguous
Some match, some not(not a case: one list)last layer wins, recorded as no consensus
Glob libraryGo doublestar: * stops at /Rust glob: * also matches / and dots
Bad glob in a declarationthe build fails (invalid pattern error)never matches: that writer votes none
Conflict resolver call failsthe build failsthe run fails

The processor output merge uses one author's declarations for one author's processors, so overlap is a mistake. Dependency layering and template stacking join several authors' files, so every author must consent (resolver agreement).

The request, the reply, the route

Both callers send the same body and expect the same reply:

jsonjson
{ "config": { "arrays": "concat" },
"files": [
{ "path": "package.json", "content": "{\"name\":\"shop\",\"type\":\"module\"}", "origin": { "template": "<id>", "layer": 0 } },
{ "path": "package.json", "content": "{\"scripts\":{\"dev\":\"node src/server.ts\"}}", "origin": { "template": "<id>", "layer": 1 } } ] }

Reply, HTTP 200: { "path": "package.json", "content": "{…merged…}" }.

The fields mean different things at each place:

FieldCoordinatorCLI
origin.templatea processor version ida template version id
origin.layerthe processor's index in the Cyanthe run order index, or stack index
configthe matching declaration's configthe first writer's declaration's config
reply pathmust equal the path, else the build failsignored

The routes differ too:

CoordinatorCLI
Routemerger container to http://cyan-resolver-<id>:5553/api/resolveCLI to POST /proxy/resolver/{id}/api/resolve on the coordinator, which forwards
Timeoutnone set (Go default client)proxy 30 s (timeout: 502); CLI client 600 s
Successexactly 200 and a reply whose path equals the path (an empty 200 body decodes to path "" and fails the build)any 2xx; else the body is read as problem details

Conflict resolver containers are shared: no session, warmed with the template, named by version id only.

Here is the real code at both places.

sulfone.borondocker_executor/merger.go
  1. lines 100-116Coordinator: every declaration with a matching doublestar glob is a match.
    docker_executor/merger.go:100-116go
    // findMatchingResolver finds resolvers whose file patterns match the given path
    func findMatchingResolver(path string, resolvers []ResolverRes) ([]ResolverRes, error) {
    var matches []ResolverRes
    for _, resolver := range resolvers {
    for _, pattern := range resolver.Files {
    matched, err := doublestar.Match(pattern, path)
    if err != nil {
    return nil, fmt.Errorf("invalid glob pattern '%s' in resolver '%s': %w", pattern, resolver.ID, err)
    }
    if matched {
    matches = append(matches, resolver)
    break // One pattern match per resolver is enough
    }
    }
    }
    return matches, nil
    }
  2. lines 419-438No match: copy the last processor's file, with its file mode.
    docker_executor/merger.go:419-438go
    for _, conflictPath := range conflicts {
    versions := fileMap[conflictPath]
    // Match resolvers using doublestar.Match()
    matchingResolvers, err := findMatchingResolver(conflictPath, m.Template.Resolvers)
    if err != nil {
    return err
    }
    if len(matchingResolvers) == 0 {
    // LWW: use last version
    fmt.Printf("Conflict detected for '%s': no resolver match, using last writer wins (layer %d)\n", conflictPath, versions[len(versions)-1].Layer)
    lastVersion := versions[len(versions)-1]
    destPath := filepath.Join(mergeDir, conflictPath)
    if err := os.MkdirAll(filepath.Dir(destPath), 0755); err != nil {
    return fmt.Errorf("failed to create directory for '%s': %w", conflictPath, err)
    }
    if err := copyFile(lastVersion.Path, destPath); err != nil {
    return fmt.Errorf("failed to copy file '%s': %w", conflictPath, err)
    }
  3. lines 439-462One match: send all versions in processor order; the reply path must be the same path.
    docker_executor/merger.go:439-462go
    } else if len(matchingResolvers) == 1 {
    // Call resolver with all versions
    resolver := matchingResolvers[0]
    files, err := buildResolverFiles(conflictPath, versions)
    if err != nil {
    return err
    }
    request := ResolverRequest{
    Config: resolver.Config,
    Files: files,
    }
    fmt.Printf("Calling resolver '%s' for conflict '%s' with %d versions\n", resolver.ID, conflictPath, len(files))
    response, err := callResolver(resolver.ID, m.SessionId, request)
    if err != nil {
    return err
    }
    // Verify resolver returned the expected path
    if response.Path != conflictPath {
    return fmt.Errorf("Resolver returned invalid path: expected '%s', got '%s'", conflictPath, response.Path)
    }
  4. lines 485-488Two or more matches: fail, even for the same conflict resolver.
    docker_executor/merger.go:485-488go
    } else {
    // Multiple resolvers match - ERROR
    return fmt.Errorf("Multiple resolvers match conflicting file '%s': [%s]. Template resolver configuration may be misconfigured.", conflictPath, strings.Join(getResolverIDs(matchingResolvers), ", "))
    }
  5. lines 119-136origin.template is the processor id; origin.layer the processor index.
    docker_executor/merger.go:119-136go
    func buildResolverFiles(relPath string, versions []processorFile) ([]ResolverFile, error) {
    var files []ResolverFile
    for _, version := range versions {
    content, err := os.ReadFile(version.Path)
    if err != nil {
    return nil, fmt.Errorf("Failed to read file '%s': %w", relPath, err)
    }
    files = append(files, ResolverFile{
    Path: relPath,
    Content: string(content),
    Origin: ResolverOrigin{
    Template: version.Template,
    Layer: version.Layer,
    },
    })
    }
    return files, nil
    }
  6. sulfone.boron/server.go:738-758Coordinator proxy for the CLI: forward to the shared container with a 30 s client; failure is 502.
    server.go:738-758go
    endpoint := fmt.Sprintf("http://%s:%d/api/resolve", docker_executor.DockerContainerToString(d), docker_executor.ResolverPort)
    fmt.Println("🌐 Upstream Endpoint:", endpoint)
    fmt.Println("πŸ†• Start forwarding request...")
    reqBody, err := io.ReadAll(c.Request.Body)
    if err != nil {
    c.JSON(http.StatusBadRequest, ProblemDetails{
    Title: "Read request failed",
    Status: http.StatusBadRequest,
    Detail: "Failed read the initial request body",
    Type: "https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400",
    TraceId: nil,
    Data: []string{err.Error()},
    })
    return
    }
    fmt.Println("πŸ“¦ Request Body Size:", len(reqBody), "bytes")
    client := &http.Client{Timeout: 30 * time.Second}
    resp, err := client.Post(endpoint, c.GetHeader("Content-Type"), bytes.NewBuffer(reqBody))
  7. sulfone.iridium/cyancoordinator/src/conflict_file_resolver/registry.rs:35-49CLI: each writer votes its first matching declaration.
    cyancoordinator/src/conflict_file_resolver/registry.rs:35-49go
    pub fn get_resolver_choice(&self, template_id: &str, path: &str) -> ResolverChoice {
    let resolvers = match self.template_resolvers.get(template_id) {
    Some(r) => r,
    None => return ResolverChoice::None,
    };
    // Find the first resolver that matches the file path
    for resolver in resolvers {
    if self.matches_any_pattern(path, &resolver.file_patterns) {
    return ResolverChoice::Some(resolver.clone());
    }
    }
    ResolverChoice::None
    }
  8. sulfone.iridium/cyancoordinator/src/conflict_file_resolver/consensus.rs:95-116CLI: unanimous agreement, else one of three last-layer-wins records.
    cyancoordinator/src/conflict_file_resolver/consensus.rs:95-116go
    // Case 1: All variations agree on same resolver
    if without_resolver.is_empty() && resolver_groups.len() == 1 {
    let (_, group) = resolver_groups.iter().next().unwrap();
    if group.len() == with_resolver.len() {
    // All have the same resolver
    let resolver = group[0].1.clone();
    return ConsensusResult::Agreed(resolver);
    }
    }
    // Case 2: All variations have no resolver
    if resolver_groups.is_empty() {
    return ConsensusResult::AllNone;
    }
    // Case 3: No consensus - some have resolver, some don't
    if !with_resolver.is_empty() && !without_resolver.is_empty() {
    return ConsensusResult::NoConsensus {
    with_resolver,
    without_resolver,
    };
    }
  9. sulfone.iridium/cyancoordinator/src/client.rs:181-201CLI: sanitise the id, POST to the proxy with a 600 s client.
    cyancoordinator/src/client.rs:181-201go
    pub fn resolve_files(
    &self,
    cyan_id: &str,
    input: &ResolverInput,
    ) -> Result<ResolverOutput, Box<dyn Error + Send>> {
    let host = (self.endpoint).to_string().to_owned();
    // Sanitize cyan_id to prevent path injection
    let sanitized_cyan_id: String = cyan_id
    .chars()
    .filter(|c| c.is_alphanumeric() || *c == '-' || *c == '_')
    .collect();
    let endpoint = host + "/proxy/resolver/" + &sanitized_cyan_id + "/api/resolve";
    let http_client = new_client()?;
    http_client
    .post(endpoint)
    .json(input)
    .send()
    .map_err(|x| Box::new(x) as Box<dyn Error + Send>)
    .and_then(|x| {
    if x.status().is_success() {
    x.json().map_err(|e| Box::new(e) as Box<dyn Error + Send>)

Worked example: package.json in the shop

web:5's build normally runs one processor, atomi/shop-render:1 (version id 6a2b8c10-5d4e-4f30-9a12-7b8c9d0e1f23), then the plugin atomi/shop-format:1, so its own build has no clash. Suppose its script returned shop-render:1 twice (two Cyan entries with different globs), and both invocations wrote package.json. web:5 declares json-merger:1 for package.json. Step through all three places one run touches.

package.json from processors to folder
  1. Inside web:5's build

    Each invocation writes to its own folder. The merger finds two versions of package.json; exactly one declaration matches, so json-merger is called with both. Both versions carry origin.template 6a2b…: the id cannot tell the two invocations apart, only layer can. shop-format:1 then runs on the merged folder.

    callercoordinator merger
    versionsshop-render:1 (layer 0), shop-render:1 (layer 1)
    resultone merged package.json
  2. Variation: two declarations

    If web:5 also declared json-merger for **/*.json with another config, two declarations would match: the build fails with Multiple resolvers match conflicting file 'package.json'.

    resultweb:5 build error
  3. web:5's tree (step 1)

    docker:7, lint:2, node-base:3 and web:5 all wrote package.json. lint:2 declares nothing, so it votes none.

    callerCLI
    votesjson-merger, none, json-merger, json-merger
    resultweb:5's file wins (lww_no_consensus)
  4. Stacking web and ci (step 3)

    web:5 and ci:2 both declare json-merger:1 with the same config: they agree. One call through the proxy; origin.template holds the template version ids.

    callerCLI via proxy
    versionsweb:5 (layer 0), ci:2 (layer 1)
    resultdev, test and ci scripts joined

The tree below shows package.json at the end of each stage.

package.json through the shop
Before
  • web5-tree/
    • package.jsonconflictstep 1: web:5's file won; eslint and docker:build lostshow file
      {
        "scripts": {
          "dev": "node src/server.ts",
          "test": "jest"
        }
      }
      
After
  • shop/
    • package.jsonchangedstep 3: json-merger joined web:5 and ci:2show file
      {
        "scripts": {
          "dev": "node src/server.ts",
          "test": "jest",
          "ci": "npm test"
        }
      }
      

Why it is built this way

Hard cases

  • One declaration, two meanings. files: ["*.json"] matches src/data.json in the CLI but not in the merger, so the same template can merge a file in one place and keep the last version in the other.
  • Same conflict resolver twice is fatal only in the coordinator. Two declarations of json-merger with overlapping globs fail the build, but in the CLI the first one is simply the vote.
  • Timeouts differ. A conflict resolver that takes 40 seconds works in the coordinator (no limit) but fails in the CLI (the proxy gives up at 30).
  • Text only, everywhere. Both send file content as strings; binary files are not safe to resolve.
  • Conflict order is hash order at both places, so conflict resolver calls for different paths come in no fixed order.
  • A cache hit warms no conflict resolver. Conflict resolver containers are started only in the coordinator's template warm (sulfone.boron/docker_executor/template_executor.go:387-420), and the CLI warms a template only on a cache miss (sulfone.iridium/cyancoordinator/src/template/executor.rs:93-99). A run where every node hits still sends an agreed vote through the proxy; if no earlier run left cyan-resolver-<id> up, the proxy answers 502 (server.go:758-768) and the run fails. Inferred from the code; found by reading; not run.
  • A bad glob has two outcomes. The merger returns an error and the build fails (merger.go:105-108, 423-426); the CLI treats the pattern as never matching (registry.rs:68-72), so the writer silently votes none.
  • Identical versions still count as a clash at both places and still call the conflict resolver.

Questions for the designer

  1. Should both places use the same rule (one match vs unanimous vote), and the same glob library?
  2. What constraint put processor clashes in the coordinator and template clashes in the CLI?
  3. Should the proxy's 30 second limit be raised to match the CLI's 600, or made the same for the merger?
  4. Should origin.template say what kind of id it holds?
  5. Should the CLI and the merger treat a bad glob the same way (fail, or never match)?
  6. Should the proxy validate the id like the CLI does, and should it need authentication?
  7. Should a run whose every node hits the cache still start the conflict resolver containers it needs?