Answer validation
How an answer is checked - the template's validator by replay for text, password and date, local shape checks in headless, and the discovery walk.
The problem. A template wants to refuse bad input (a port below 1024, an empty name) with its own rule, written in its own language. But the rule lives in the template container, the input is typed at the CLI, and the template keeps no state between calls, so it does not even know which question is being answered.
The idea. Validation is one more stateless replay. The CLI sends the answers before the question plus the candidate text; the template replays, stops at its first unanswered question (which is, by construction, the one being answered) and runs that question's validator. Headless runs add local shape checks and validate every supplied answer in a separate discovery walk.
The picture shows both paths to the same validate call.
flowchart TD
subgraph Interactive
IQ["question reply: text, password or date"] --> IV["inquire validator<br/>answers and state from BEFORE this reply"]
IV -->|on Enter| VC
end
subgraph Headless
HQ["discovery walk reaches a supplied answer"] --> LC{"local check:<br/>type fits kind? option listed?<br/>date parses and in range?"}
LC -->|no| HE["Err: invalid answer for question 'id'"]
LC -->|"select, checkbox, confirm: yes"| OK["accepted, no call"]
LC -->|"text, password, date: yes"| VC
end
VC["validate call<br/>siblings + state + candidate"] --> SDK["SDK replays to first unanswered question,<br/>runs its validate(candidate)"]
SDK -->|"valid: null"| ACC["accepted"]
SDK -->|"valid: reason"| REF["interactive: shown, prompt stays<br/>headless: redacted Err, run stops"]How it works
The template side (all three SDKs): build a new inquirer and determinism helper from the request, run the script, catch the checkpoint exception, call question.validate(candidate) if the question has one, and reply { "valid": reason }. No validator, or an empty reason, means valid: null. If the script finishes instead of stopping, the SDK throws "Not supposed to reach here for validation!".
Interactive (add_template_validator):
- Only text, password and date prompts get a validator. Select, checkbox and confirm have none.
- The validator captures the answer map and the deterministic state as they were before the current question's reply. The current id is therefore absent, and the replay stops at it.
- inquire runs the validator when the user presses Enter. A date is sent as
YYYY-MM-DD. - A reason is shown under the prompt and the prompt stays open. A transport error (no connection, a body that is not JSON) ends the prompt, and the loop, with Err. A JSON error body without
validis not a transport error: it counts as accepted (see the bug below).
Headless (validate_answer, called from both walks):
- Local check (
validateable_value). The answer's type must fit the kind; a select value must be one ofoptions; every checkbox value must be one ofoptions; a date must parse as%Y-%m-%dand sit insidemin_date/max_date(a bound that does not parse is ignored). Failure:invalid answer for question '<id>', no call made. - Select, checkbox and confirm stop here, accepted.
- Text, password and date go to the validate call with a sibling map: the answers the walk has passed, without the current id, and the state from before the question.
- A refusal or transport error becomes an id-only message (secret redaction) and the run ends with Err.
The discovery walk (validate_supplied_answers) runs before the outcome walk and calls validate_answer for every supplied answer the template reaches, one question at a time. Without it, a template that finalizes once every id is present would accept a bad supplied answer as done.
The table compares the two modes.
| Aspect | Interactive | Headless |
|---|---|---|
| answers present at start | never validated | validated when reached |
| sibling map | every answer, carried and preset included | only answers this node revealed so far |
| kinds with a server check | text, password, date | text, password, date |
| other kinds | widget allows only valid picks | local type and option check |
| bad date bound in metadata | the prompt fails (Err) | bound ignored |
| on refusal | reason shown, try again | id-only Err, run stops |
Here is the real code: the template side, the interactive validator, the headless checks and the discovery walk.
- lines 37-52Replay to the first unanswered question; run its validator; falsy result → null. Finishing the script is an error.
sdks/node/src/domain/template/service.ts:37-52typescript async validate(answer: TemplateValidateInput): Promise<string | null> {const i = new StatelessInquirer(answer.answers);const d = new StatelessDeterminism(answer.deterministicState);try {await this._template.template(i, d);throw new Error('Not supposed to reach here for validation!');} catch (e) {if (e instanceof OutOfAnswerException) {const q: Question = e.question;const validateResult = q.validate?.(answer.validate);return validateResult || null;}throw e;}} - sulfone.iridium/cyanprompt/src/domain/services/template/validate.rs:234-256Interactive: text and password get a validator over the captured maps (date is the same, below).
cyanprompt/src/domain/services/template/validate.rs:234-256typescript pub fn add_template_validator(p: Prompts,repo: Rc<dyn CyanRepo>,answers: HashMap<String, Answer>,deterministic_state: HashMap<String, String>,) -> Prompts {match p {Prompts::Text(text) => Prompts::Text(text.with_validator(move |v: &str| {validate_template(v,Rc::clone(&repo),answers.clone(),deterministic_state.clone(),)})),Prompts::Password(pw) => Prompts::Password(pw.with_validator(move |v: &str| {validate_template(v,Rc::clone(&repo),answers.clone(),deterministic_state.clone(),)})), - sulfone.iridium/cyanprompt/src/domain/services/template/validate.rs:14-30One validate call: reason → Invalid, null → Valid.
cyanprompt/src/domain/services/template/validate.rs:14-30typescript fn validate_template(result: &str,repo: Rc<dyn CyanRepo>,answers: HashMap<String, Answer>,deterministic_state: HashMap<String, String>,) -> Result<Validation, CustomUserError> {let input = TemplateValidateInput {answers: answers.clone(),deterministic_state: deterministic_state.clone(),validate: result.to_string(),};repo.validate_template(input).map(|r| {r.map_or(Validation::Valid, |x| {Validation::Invalid(ErrorMessage::Custom(x))})})} - sulfone.iridium/cyanprompt/src/domain/services/template/validate.rs:94-119Headless local checks: option membership, Bool for confirm, any other pairing refused.
cyanprompt/src/domain/services/template/validate.rs:94-119typescript (Question::Select(select), Answer::String(s)) => {if select.options.iter().any(|opt| opt == s) {Ok(None)} else {Err(format!("invalid answer for question '{}'", question.id()))}}// The interactive Checkbox (MultiSelect) picker only ever yields a subset of// the question's `options` — a value outside that set can never be selected.// Reject any element that is not a member (id-only message).(Question::Checkbox(checkbox), Answer::StringArray(arr)) => {if arr.iter().all(|s| checkbox.options.iter().any(|opt| opt == s)){Ok(None)} else {Err(format!("invalid answer for question '{}'", question.id()))}}// Confirm has no structural constraint beyond the Bool discriminant — accepted// unconditionally (mirrors the interactive path's `default => default` arm).(Question::Confirm(_), Answer::Bool(_)) => Ok(None),// Any other (question, answer) pairing is a type mismatch the interactive prompt// could never produce — reject with an id-only message._ => Err(format!("invalid answer for question '{}'", question.id())), - sulfone.iridium/cyanprompt/src/domain/services/template/validate.rs:140-158Local check first; only text, password and date reach the call; sibling values captured for redaction.
cyanprompt/src/domain/services/template/validate.rs:140-158typescript let value = match validateable_value(question, answer) {// A value to push through the coordinator validator.Ok(Some(value)) => value,// No interactive validator for this kind — accept (mirrors `add_template_validator`).Ok(None) => return Ok(()),// Type mismatch / malformed date — reject before contacting the coordinator.// The message references the question id only.Err(msg) => return Err(msg),};// The coordinator validator receives the full SIBLING answer map (every answer the// walk has progressed past), so a cross-field validator's rejection message or a// transport error can echo a SIBLING value — including an earlier Password — not just// the submitted value for this question. Capture the rendered sibling values BEFORE// the map is moved into `validate_template` so the redaction below can drop any// message echoing one of them. Without this the offending value for the CURRENT// question is redacted but a sibling secret echoed by a cross-field validator reaches// the headless error envelope verbatim, leaking a secret the secrecy contract forbids.let sibling_values = redact::answer_map_renderings(&answers);match validate_template(&value, repo, answers, deterministic_state) { - sulfone.iridium/cyanprompt/src/domain/services/template/engine.rs:345-357Discovery: siblings are the answers revealed so far minus the current id; state is from before the question.
cyanprompt/src/domain/services/template/engine.rs:345-357typescript let mut siblings = discovered.clone();siblings.remove(&question_id);crate::domain::services::template::validate::validate_answer(&q.question,answer,Rc::clone(&self.client),siblings,state_data.clone(),)?;// Valid — record it and thread deterministic state so the// next call advances the walk to the following question.discovered.insert(question_id, answer.clone());state_data = q.deterministic_state; - sulfone.iridium/cyanprompt/src/domain/services/template/engine.rs:215-227Outcome walk: the same check with the walked map, as a second line of defence.
cyanprompt/src/domain/services/template/engine.rs:215-227typescript match crate::domain::services::template::validate::validate_answer(&q.question,answer,Rc::clone(&self.client),walked.clone(),state_data.clone(),) {Ok(()) => {// Answer already supplied and valid — record it in the// walk-order snapshot, re-derive deterministic state// from the coordinator's response, and continue.walked.insert(question_id, answer.clone());state_data = q.deterministic_state;
Worked example: a bad port in a headless run
The answers file for cyanprint create atomi/web:5 ./shop --headless has every answer, but port is 80. docker:7, lint:2 and node-base:3 pass. Step through web:5's discovery walk.
- Init call, empty map
Discovery starts from nothing, even though four answers were supplied. d.get makes A; the replay stops at name.
sent answers {}sent state {}reply question name (text), state {instance-id: A} - Check name
Text with a String: no local refusal. Validate call with siblings {} and the state from before name ({}).
siblings {}state {}candidate shopreply valid: nullrevealed name - Init call with name
Discovery now carries the name reply's state. d.get returns A; the replay stops at port.
sent answers namesent state {instance-id: A}reply question port - Check port
Validate call with siblings {name} and the state from before port. The replay stops at port and runs its validator.
siblings {name: shop}state {instance-id: A}candidate 80reply valid: Use a port from 1024 to 65535 - Refused
The reason echoes neither 80 nor shop, so it is kept. The walk returns Err; the outcome walk never runs; ./shop is removed.
outcome Errmessage has invalid answer for question 'port': Use a port from 1024 to 65535exit 1
Why it is built this way
Hard cases
- The validate call has no question id. The map must stop the replay at the right question. A map that already answers it would make the template validate the next question, or throw at the end.
- Answers present at the start are trusted interactively. Saved answers in an update, preset answers and answers from earlier nodes are never shown and never validated. Headless validates them when reached, so a stricter validator in web:6 can fail a headless update on an answer an interactive update would accept.
- Headless re-checks saved answers of every re-executed template, not only the upgraded one. Each node runs
start_headlessover its whole map (executor.rs:208-211; the base rebuild atrun.rs:140-157too), so a stricter validator in any template that is not served from the cache can fail a headless run when the caller supplies nothing. Found while reading; not run. - A validator that throws. In the Node SDK that is a rejected handler, not a refusal.
- One validator sees a fresh d.get value. The validate call sends the state from before the question's reply, so a value first made in the replay that stopped at this question (web:5's
name) is made again, differently, for each validation. - Every attempt is a full replay. A user who types three bad ports costs three replays of the script.
- Node empties to valid. Any falsy validator result (
"",undefined) is valid. A validator that throws is an error, not a refusal. - Checkbox duplicates and empty picks pass the local check.
- Cache hits skip all of this: a node served from the node cache runs no walk.
Questions for the designer
- Should select, checkbox and confirm questions be able to carry a server validator?
- Should interactive runs validate saved and preset answers against the template's current validators, as headless does?
- Should a validate call name the question id, so a mismatch is caught instead of validating the wrong question?
- Should validators receive the deterministic state from the question's own reply?
- Should the headless sibling map include carried and preset answers, as the interactive one does?
- Should a validate reply without a
validkey (an error body) be refused instead of accepted?