Doc system test

Answer validation

How an answer is checked - the template's validator by replay for text, password and date, local shape checks in headless, and the discovery walk.

Meta block front matter of this doc
namespaces
  • durian.cloud/cyanprint/prompting
  • durian.cloud/cyanprint/authoring
  • durian.cloud/cyanprint/composition
type
algorithm
code
  • sulfone.iridium/cyanprompt/src/domain/services/template/validate.rs
  • sulfone.iridium/cyanprompt/src/domain/services/template/engine.rs
  • sulfone.iridium/cyanprompt/src/http/client.rs
  • sulfone.helium/sdks/node/src/domain/template/service.ts
  • sulfone.helium/sdks/python/cyanprintsdk/domain/template/service.py
  • sulfone.helium/sdks/dotnet/sulfone-helium/Domain/Template/Service.cs
issues
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+preset-and-saved-answers-not-validated
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+date-bounds-differ-interactive-headless
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+headless-sibling-map-differs
  • https://github.com/AtomiCloud/sulfone.iridium/issues?q=TBD+validate-error-body-accepted

The problem. A template wants to refuse bad input (a port below 1024, an empty name) with its own rule, written in its own language. But the rule lives in the template container, the input is typed at the CLI, and the template keeps no state between calls, so it does not even know which question is being answered.

The idea. Validation is one more stateless replay. The CLI sends the answers before the question plus the candidate text; the template replays, stops at its first unanswered question (which is, by construction, the one being answered) and runs that question's validator. Headless runs add local shape checks and validate every supplied answer in a separate discovery walk.

The picture shows both paths to the same validate call.

flowchart TD
  subgraph Interactive
    IQ["question reply: text, password or date"] --> IV["inquire validator<br/>answers and state from BEFORE this reply"]
    IV -->|on Enter| VC
  end
  subgraph Headless
    HQ["discovery walk reaches a supplied answer"] --> LC{"local check:<br/>type fits kind? option listed?<br/>date parses and in range?"}
    LC -->|no| HE["Err: invalid answer for question 'id'"]
    LC -->|"select, checkbox, confirm: yes"| OK["accepted, no call"]
    LC -->|"text, password, date: yes"| VC
  end
  VC["validate call<br/>siblings + state + candidate"] --> SDK["SDK replays to first unanswered question,<br/>runs its validate(candidate)"]
  SDK -->|"valid: null"| ACC["accepted"]
  SDK -->|"valid: reason"| REF["interactive: shown, prompt stays<br/>headless: redacted Err, run stops"]
Two paths to one validator

How it works

The template side (all three SDKs): build a new inquirer and determinism helper from the request, run the script, catch the checkpoint exception, call question.validate(candidate) if the question has one, and reply { "valid": reason }. No validator, or an empty reason, means valid: null. If the script finishes instead of stopping, the SDK throws "Not supposed to reach here for validation!".

Interactive (add_template_validator):

  1. Only text, password and date prompts get a validator. Select, checkbox and confirm have none.
  2. The validator captures the answer map and the deterministic state as they were before the current question's reply. The current id is therefore absent, and the replay stops at it.
  3. inquire runs the validator when the user presses Enter. A date is sent as YYYY-MM-DD.
  4. A reason is shown under the prompt and the prompt stays open. A transport error (no connection, a body that is not JSON) ends the prompt, and the loop, with Err. A JSON error body without valid is not a transport error: it counts as accepted (see the bug below).

Headless (validate_answer, called from both walks):

  1. Local check (validateable_value). The answer's type must fit the kind; a select value must be one of options; every checkbox value must be one of options; a date must parse as %Y-%m-%d and sit inside min_date/max_date (a bound that does not parse is ignored). Failure: invalid answer for question '<id>', no call made.
  2. Select, checkbox and confirm stop here, accepted.
  3. Text, password and date go to the validate call with a sibling map: the answers the walk has passed, without the current id, and the state from before the question.
  4. A refusal or transport error becomes an id-only message (secret redaction) and the run ends with Err.

The discovery walk (validate_supplied_answers) runs before the outcome walk and calls validate_answer for every supplied answer the template reaches, one question at a time. Without it, a template that finalizes once every id is present would accept a bad supplied answer as done.

The table compares the two modes.

AspectInteractiveHeadless
answers present at startnever validatedvalidated when reached
sibling mapevery answer, carried and preset includedonly answers this node revealed so far
kinds with a server checktext, password, datetext, password, date
other kindswidget allows only valid pickslocal type and option check
bad date bound in metadatathe prompt fails (Err)bound ignored
on refusalreason shown, try againid-only Err, run stops

Here is the real code: the template side, the interactive validator, the headless checks and the discovery walk.

sulfone.heliumsdks/node/src/domain/template/service.ts
  1. lines 37-52Replay to the first unanswered question; run its validator; falsy result → null. Finishing the script is an error.
    sdks/node/src/domain/template/service.ts:37-52typescript
    async validate(answer: TemplateValidateInput): Promise<string | null> {
    const i = new StatelessInquirer(answer.answers);
    const d = new StatelessDeterminism(answer.deterministicState);
    try {
    await this._template.template(i, d);
    throw new Error('Not supposed to reach here for validation!');
    } catch (e) {
    if (e instanceof OutOfAnswerException) {
    const q: Question = e.question;
    const validateResult = q.validate?.(answer.validate);
    return validateResult || null;
    }
    throw e;
    }
    }
  2. sulfone.iridium/cyanprompt/src/domain/services/template/validate.rs:234-256Interactive: text and password get a validator over the captured maps (date is the same, below).
    cyanprompt/src/domain/services/template/validate.rs:234-256typescript
    pub fn add_template_validator(
    p: Prompts,
    repo: Rc<dyn CyanRepo>,
    answers: HashMap<String, Answer>,
    deterministic_state: HashMap<String, String>,
    ) -> Prompts {
    match p {
    Prompts::Text(text) => Prompts::Text(text.with_validator(move |v: &str| {
    validate_template(
    v,
    Rc::clone(&repo),
    answers.clone(),
    deterministic_state.clone(),
    )
    })),
    Prompts::Password(pw) => Prompts::Password(pw.with_validator(move |v: &str| {
    validate_template(
    v,
    Rc::clone(&repo),
    answers.clone(),
    deterministic_state.clone(),
    )
    })),
  3. sulfone.iridium/cyanprompt/src/domain/services/template/validate.rs:14-30One validate call: reason → Invalid, null → Valid.
    cyanprompt/src/domain/services/template/validate.rs:14-30typescript
    fn validate_template(
    result: &str,
    repo: Rc<dyn CyanRepo>,
    answers: HashMap<String, Answer>,
    deterministic_state: HashMap<String, String>,
    ) -> Result<Validation, CustomUserError> {
    let input = TemplateValidateInput {
    answers: answers.clone(),
    deterministic_state: deterministic_state.clone(),
    validate: result.to_string(),
    };
    repo.validate_template(input).map(|r| {
    r.map_or(Validation::Valid, |x| {
    Validation::Invalid(ErrorMessage::Custom(x))
    })
    })
    }
  4. sulfone.iridium/cyanprompt/src/domain/services/template/validate.rs:94-119Headless local checks: option membership, Bool for confirm, any other pairing refused.
    cyanprompt/src/domain/services/template/validate.rs:94-119typescript
    (Question::Select(select), Answer::String(s)) => {
    if select.options.iter().any(|opt| opt == s) {
    Ok(None)
    } else {
    Err(format!("invalid answer for question '{}'", question.id()))
    }
    }
    // The interactive Checkbox (MultiSelect) picker only ever yields a subset of
    // the question's `options` — a value outside that set can never be selected.
    // Reject any element that is not a member (id-only message).
    (Question::Checkbox(checkbox), Answer::StringArray(arr)) => {
    if arr
    .iter()
    .all(|s| checkbox.options.iter().any(|opt| opt == s))
    {
    Ok(None)
    } else {
    Err(format!("invalid answer for question '{}'", question.id()))
    }
    }
    // Confirm has no structural constraint beyond the Bool discriminant — accepted
    // unconditionally (mirrors the interactive path's `default => default` arm).
    (Question::Confirm(_), Answer::Bool(_)) => Ok(None),
    // Any other (question, answer) pairing is a type mismatch the interactive prompt
    // could never produce — reject with an id-only message.
    _ => Err(format!("invalid answer for question '{}'", question.id())),
  5. sulfone.iridium/cyanprompt/src/domain/services/template/validate.rs:140-158Local check first; only text, password and date reach the call; sibling values captured for redaction.
    cyanprompt/src/domain/services/template/validate.rs:140-158typescript
    let value = match validateable_value(question, answer) {
    // A value to push through the coordinator validator.
    Ok(Some(value)) => value,
    // No interactive validator for this kind — accept (mirrors `add_template_validator`).
    Ok(None) => return Ok(()),
    // Type mismatch / malformed date — reject before contacting the coordinator.
    // The message references the question id only.
    Err(msg) => return Err(msg),
    };
    // The coordinator validator receives the full SIBLING answer map (every answer the
    // walk has progressed past), so a cross-field validator's rejection message or a
    // transport error can echo a SIBLING value — including an earlier Password — not just
    // the submitted value for this question. Capture the rendered sibling values BEFORE
    // the map is moved into `validate_template` so the redaction below can drop any
    // message echoing one of them. Without this the offending value for the CURRENT
    // question is redacted but a sibling secret echoed by a cross-field validator reaches
    // the headless error envelope verbatim, leaking a secret the secrecy contract forbids.
    let sibling_values = redact::answer_map_renderings(&answers);
    match validate_template(&value, repo, answers, deterministic_state) {
  6. sulfone.iridium/cyanprompt/src/domain/services/template/engine.rs:345-357Discovery: siblings are the answers revealed so far minus the current id; state is from before the question.
    cyanprompt/src/domain/services/template/engine.rs:345-357typescript
    let mut siblings = discovered.clone();
    siblings.remove(&question_id);
    crate::domain::services::template::validate::validate_answer(
    &q.question,
    answer,
    Rc::clone(&self.client),
    siblings,
    state_data.clone(),
    )?;
    // Valid — record it and thread deterministic state so the
    // next call advances the walk to the following question.
    discovered.insert(question_id, answer.clone());
    state_data = q.deterministic_state;
  7. sulfone.iridium/cyanprompt/src/domain/services/template/engine.rs:215-227Outcome walk: the same check with the walked map, as a second line of defence.
    cyanprompt/src/domain/services/template/engine.rs:215-227typescript
    match crate::domain::services::template::validate::validate_answer(
    &q.question,
    answer,
    Rc::clone(&self.client),
    walked.clone(),
    state_data.clone(),
    ) {
    Ok(()) => {
    // Answer already supplied and valid — record it in the
    // walk-order snapshot, re-derive deterministic state
    // from the coordinator's response, and continue.
    walked.insert(question_id, answer.clone());
    state_data = q.deterministic_state;

Worked example: a bad port in a headless run

The answers file for cyanprint create atomi/web:5 ./shop --headless has every answer, but port is 80. docker:7, lint:2 and node-base:3 pass. Step through web:5's discovery walk.

Discovery walk of web:5 with port 80
  1. Init call, empty map

    Discovery starts from nothing, even though four answers were supplied. d.get makes A; the replay stops at name.

    sent answers{}
    sent state{}
    replyquestion name (text), state {instance-id: A}
  2. Check name

    Text with a String: no local refusal. Validate call with siblings {} and the state from before name ({}).

    siblings{}
    state{}
    candidateshop
    replyvalid: null
    revealedname
  3. Init call with name

    Discovery now carries the name reply's state. d.get returns A; the replay stops at port.

    sent answersname
    sent state{instance-id: A}
    replyquestion port
  4. Check port

    Validate call with siblings {name} and the state from before port. The replay stops at port and runs its validator.

    siblings{name: shop}
    state{instance-id: A}
    candidate80
    replyvalid: Use a port from 1024 to 65535
  5. Refused

    The reason echoes neither 80 nor shop, so it is kept. The walk returns Err; the outcome walk never runs; ./shop is removed.

    outcomeErr
    message hasinvalid answer for question 'port': Use a port from 1024 to 65535
    exit1

Why it is built this way

Hard cases

  • The validate call has no question id. The map must stop the replay at the right question. A map that already answers it would make the template validate the next question, or throw at the end.
  • Answers present at the start are trusted interactively. Saved answers in an update, preset answers and answers from earlier nodes are never shown and never validated. Headless validates them when reached, so a stricter validator in web:6 can fail a headless update on an answer an interactive update would accept.
  • Headless re-checks saved answers of every re-executed template, not only the upgraded one. Each node runs start_headless over its whole map (executor.rs:208-211; the base rebuild at run.rs:140-157 too), so a stricter validator in any template that is not served from the cache can fail a headless run when the caller supplies nothing. Found while reading; not run.
  • A validator that throws. In the Node SDK that is a rejected handler, not a refusal.
  • One validator sees a fresh d.get value. The validate call sends the state from before the question's reply, so a value first made in the replay that stopped at this question (web:5's name) is made again, differently, for each validation.
  • Every attempt is a full replay. A user who types three bad ports costs three replays of the script.
  • Node empties to valid. Any falsy validator result ("", undefined) is valid. A validator that throws is an error, not a refusal.
  • Checkbox duplicates and empty picks pass the local check.
  • Cache hits skip all of this: a node served from the node cache runs no walk.

Questions for the designer

  1. Should select, checkbox and confirm questions be able to carry a server validator?
  2. Should interactive runs validate saved and preset answers against the template's current validators, as headless does?
  3. Should a validate call name the question id, so a mismatch is caught instead of validating the wrong question?
  4. Should validators receive the deterministic state from the question's own reply?
  5. Should the headless sibling map include carried and preset answers, as the interactive one does?
  6. Should a validate reply without a valid key (an error body) be refused instead of accepted?