Testing a Loop

You test a loop in two ways. You try the draft yourself on the Chat bench, and you declare test cases: example conversations that run against the draft and must pass before the version can publish.

Try the draft on the bench

A draft never serves a user, but Chat can run it. Start a new conversation and pick the loop and the version, draft included. Talk to it as your user would: Enter sends and Shift+Enter starts a new line. The line under your message reads “Thinking”, and after ten seconds it shows how long the step has taken. The run shows on Runs like any other, so you can read each turn, each tool call, and each record it wrote.

Declare test cases

A test case is a check in the loop’s record, with the kind test_case:

checks:
  - kind: test_case
    name: a_prospect_who_qualifies
    transcript:
      - role: user
        text: We are about forty people and we need to route inbound leads automatically.
    expect:
      ending: completed
      leads.email: present
  - kind: test_case
    name: a_visitor_who_does_not_qualify_leaves_no_lead
    transcript:
      - role: user
        text: Just browsing, and I do not want to be contacted.
    expect:
      ending: completed
      leads.email: absent

Key	What it says
`name`	The case’s name. A failure names it.
`transcript`	The user’s turns that the case plays to the loop.
`expect`	What must be true when the case run ends. At least one entry.
`inputs`	What the case answers when the loop shows its form, by input id. Test data, never a secret. A required input the form asks for and the case does not give declines the form, which is how a case tests the loop’s answer to a refusal.
`stubs`	What a tool answers in this case, by tool id.
`approve`	The person’s answer to the approval card on a step, by the step’s name in `gates:`: `true` approves it, `false` declines it, for example `approve: {issue_refund: true}`. A case run declines every approval a person would give that the case does not name, so this is how a case tests what happens after the person says yes. Publish refuses a step that `gates:` does not ask a user to approve, and a case whose run never shows a card it names fails, saying so.

What a case can expect

ending — how the run must end, for example completed.
<collection>.<field> — present or absent: whether the run wrote that field. A case that expects absent proves the loop can end without writing a record.
<collection>.<field>: {equals: <value>} — the run wrote exactly that value, for example leads.company_size: {equals: "11-50"}. Publish refuses it on a field the collection fills from an input, because that value never passes through the loop; expect present there.
tool:<tool_id> — called or not_called, or the arguments the call must carry, such as tool:create_lead: {plan: team}. Other arguments are ignored.

What a case may touch

Each call in a case run is checked exactly as in a real run. Then a call that would leave the platform is not made. A tool with external effects, and your own endpoint or code tool, answers the case’s stubs entry for it, or {ok: true, simulated: true} when there is none. The run’s outputs are recorded, not sent, so running the cases three times never puts three leads in your CRM.

A tool that only reads, such as a lookup, can run for real in test cases. Declare case_runs: live on it. The publish review then says the tool is “called for real by test cases”. Publish refuses live on a tool with external effects and on a tool that runs on your page.

Run the cases

On the loop’s page, press Publish v1… (or the draft’s version) to open the publish review. Under Test cases, press Run the test cases. Each case is a real model run, paid from the account’s wallet. Each result says passed or failed and names what did not hold, and Open the run shows the case’s conversation.

The results belong to the draft’s current text. When you edit the draft, its results are cleared, and the review says that no test cases have run against it. Run the cases again before you publish.

Publish waits for the cases

A draft that declares test cases publishes only after they ran on its current text. One failure refuses the publish, and the refusal names the case. A draft that declares no test cases needs none.