Semantic Decision Node

Problem

When a general-purpose LLM is asked to make these branch decisions, the decision and the branch fuse into one generated answer. The model may name an option outside the allowed set, phrase its answer so that parsing it is itself a judgement, or state high certainty on a coin-flip. The threshold that decides when to act, when to ask a human and when to fall back sits implicitly inside the prompt, where it cannot be reviewed, tuned per model version or audited afterwards. The call also costs frontier-model latency and price for a question whose answer is a single bit, and in a harness that asks dozens of such questions per turn that cost dominates. Writing the conditions as hard-coded rules instead fails on exactly the fuzzy inputs that made a model necessary.

Solution

Declare each branch condition as a question with a fixed answer type: a yes/no probability, a choice from a closed set with a distribution over the options, or a score on declared levels. At the node, send the current state and the question to a decision model whose output is constrained to that type, so an answer outside the schema cannot occur. The model returns the answer, the probability or distribution, and a confidence; it never names the next step. Code holds the thresholds, kept in one reviewed file next to the questions and keyed by the model version that answered, and maps the result onto the branches: act above a high threshold, route to a human or a fallback in the uncertain band, take the other branch below a low one. A compound judgement is split into orthogonal questions answered independently over the same state, so one answer never becomes context for another, and code composes them. Every decision is logged with the question, the answer, the probability, the threshold and the model version, so a wrong branch can be traced to a wrong answer or a wrong threshold. Thresholds are tuned on labelled traffic as coverage-against-accuracy curves rather than set once by intuition.

When to use

• The harness control flow is explicit code and some branch conditions need semantic judgement that rules cannot express.
• The decision has a closed answer set: yes or no, one of a fixed list, or a position on a declared scale.
• Many such decisions happen per turn or per item, so latency and cost per decision matter.
• The team needs to review, audit and tune the thresholds that decide when the agent acts, asks or falls back.