Testing a Loop

You test a loop in two ways. You try the draft yourself in My conversations, and you declare test cases: example conversations that run against the draft and must pass before the version can publish.

Try the draft on the bench

A draft never serves a user, but My conversations can run it. Start a new conversation and pick the loop and the version, draft included. Talk to it as your user would: Enter sends and Shift+Enter starts a new line. The line under your message reads “Thinking”, and after ten seconds it shows how long the step has taken. The run shows on All runs like any other, so you can read each turn, each tool call, and each record it wrote.

Declare test cases

A test case is a check in the loop’s record, with the kind test_case:

checks:
  - kind: test_case
    name: a_prospect_who_qualifies
    transcript:
      - role: user
        text: We are about forty people and we need to route inbound leads automatically.
    expect:
      ending: completed
      leads.email: present
  - kind: test_case
    name: a_visitor_who_does_not_qualify_leaves_no_lead
    transcript:
      - role: user
        text: Just browsing, and I do not want to be contacted.
    expect:
      ending: completed
      leads.email: absent

Key	What it says
`name`	The case’s name. A failure names it.
`transcript`	The user’s turns that the case plays to the loop.
`expect`	What must be true when the case run ends. At least one entry.
`inputs`	What the case answers when the loop shows its form, by input id. Test data, never a secret. A required input the form asks for and the case does not give declines the form, which is how a case tests the loop’s answer to a refusal.
`stubs`	What a tool answers in this case, by tool id.
`approve`	The person’s answer to the approval card on a step, by the step’s name in `gates:`: `true` approves it, `false` declines it, for example `approve: {issue_refund: true}`. A case run declines every approval a user would give that the case does not name, so this is how a case tests what happens after the user says yes. It approves every `approve: operator` card the case does not name, because you started the run, and the run’s steps say so (“Approved by default, as test case … runs”), so `false` is how a case tests what happens after your team says no. Publish refuses a step that `gates:` does not ask a user or the operator to approve, and a case whose run never shows a card it names fails, saying so.

What a case can expect

ending — how the run must end, for example completed.
<collection>.<field> — present or absent: whether the run wrote that field. A case that expects absent proves the loop can end without writing a record.
<collection>.<field>: {equals: <value>} — the run wrote exactly that value, for example leads.company_size: {equals: "11-50"}. Publish refuses it on a field the collection fills from an input, because that value never passes through the loop; expect present there.
tool:<tool_id> — called or not_called, or the arguments the call must carry, such as tool:create_lead: {plan: team}. Other arguments are ignored.

What a case may touch

Each call in a case run is checked exactly as in a real run. Then a call that would leave the platform is not made. A tool with external effects, and your own endpoint or code tool, answers the case’s stubs entry for it, or {ok: true, simulated: true} when there is none. The run’s outputs are recorded, not sent, so running the cases three times never puts three leads in your CRM.

A tool that only reads, such as a lookup, can run for real in test cases. Declare case_runs: live on it. The publish review then says the tool is “called for real by test cases”. Publish refuses live on a tool with external effects and on a tool that runs on your page.

Run the cases

On the loop’s page, press Publish… (or Publish the change…). Under Test questions, the card starts the cases itself when none have run on the loop’s current text, and says that each is a real conversation, paid from the account’s wallet. A set that already ran on this text is not run again. Each case is shown by the visitor’s first message, such as “My ticket is ABC123. Is it ready?”, rather than its name; two cases that open alike are numbered (1), (2) in the order they are declared, here and in the chat’s lines about the run, and a case that opens with no visitor message shows its name. Each result says passed or failed, and See the conversation shows the case’s conversation. A failed case says what the case required and, under What happened, what the run did in plain words, such as “The run saved no run id in repair requests.” When the loop’s own check refused a save, the line names the rule, such as “The loop’s own check refused the save in demo action items: each record needs a value for priority.” That reason comes first, before what the run then failed to do, and the chat’s summary of the results shows it. The check’s words below it names the exact expectation that did not hold, which is what to change in the case or in the loop.

When a spending limit stopped a case’s run, the line names that limit and its fix. If the account’s wallet could not cover the run, add credit on Wallet & budgets: a higher budget for the loop does not help, because each case run sets aside its budget for one run from the same wallet. If the loop’s budget for one run is less than one step costs, the line says so, and that budget is the one to raise.

The results belong to the draft’s current text. When you edit the draft, its results are cleared, and the next time you open Publish the card runs them again.

Publish waits for the cases

A draft that declares test cases publishes only after they ran on its current text and passed. Publish stays disabled while they run or after one fails, with the reason under it; Change this test question opens the case in the editor. A draft that declares no test cases needs none.