Skip to content

Chatbot testing

Test your AI chatbot before your customers do

Describe a customer and what a good chat looks like. An AI plays that customer against your real agent, an AI judge grades the chat pass or fail with a reason, and nothing is sent, booked or saved along the way.

Example: a test for a refund on a late order

Visitor: My order is a week late. I want my money back.

Agent: I'm sorry your order is late. Could you give me your order number so I can look into the refund for you?

Visitor: It's 1042.

Agent: Thank you. I'm starting the refund request for order 1042. The refund isn't final until it is confirmed.

One fix can quietly break another answer

You tighten the instructions so the agent stops promising refunds, and a week later it no longer asks for the order number. You add a new price list, and an old page now contradicts it. Most teams test a chatbot by typing a few questions after each change, which catches the problem you were fixing and misses the ones you weren't looking at. Testing an AI chatbot properly means replaying the conversations that matter every time something changes, with the same tools the agent uses for real, and without emailing your team or booking meetings while you do it.

Check the conversations that matter in one click

Tests with simulated customers sit in the Playground, below the chat. They are included from Starter.

  • Simulated customers

    For each test you describe who the customer is, how they behave and what they want. An AI plays them, in the language the description implies, so "a German customer" chats in German.

  • Your real agent

    The agent answers exactly as on your website: with its saved instructions, guardrails, knowledge, procedures and tools. You test what visitors would get, not a copy.

  • An AI judge with reasons

    When the chat ends, a judge reads it against your success criteria and grades it Pass or Fail, with the reason. You see the share of tests that passed at the top.

  • No side effects

    Actions, hand-offs, live chat requests, bookings, order lookups and form submissions are simulated. Nobody gets an email, no ticket is opened and nothing is booked.

  • Drafts side by side

    In the Playground, Compare asks your saved settings and an edited draft the same question, so you see which answers better before you change anything on your website.

How a test works

  1. Write the test

    Click Add test and fill in a name, the customer, their goal and the success criteria: what the agent must do, or never do. Optionally set a first message and Max turns, from 1 to 8 replies (6 by default).

  2. Run one or all

    Run starts a single test and Run all runs every test of the agent, one after another. The page shows each result as it finishes. Cancel stops the run at the next step.

  3. The customer chats until it ends

    The simulated customer writes, your agent replies with its real tools, and the chat goes on until the customer reaches their goal, gives up, or the agent has replied Max turns times.

  4. Read the verdict

    Each test shows Pass, Fail or Error. Open it to read the judge's reason and the whole chat, with the tools the agent used under each reply. Recent runs keeps your last 10 runs to compare.

Testing details

  • Up to 25 tests per agent

    Start with the conversations that matter most: your most common question, a refund or cancellation, a question the agent must decline, a request for a person. Edit or delete a test at any time; past results keep its name.

  • Read-only actions, if you want them

    API actions are not called in a test: the agent gets a result marked as simulated. Switch on Allow read-only actions and actions with the GET method really run, so answers can use your live data.

  • Procedures step by step

    Procedures run as usual, with their state kept only for the test. Their actions are simulated like any other, and client-side actions in the browser are skipped.

  • Nothing in your stats

    Tests never appear under Conversations, Leads, topics or dashboard stats. Lead forms and chat forms can appear, but nothing is submitted and no alert or webhook goes out. Results stay in the Tests section.

  • Try models and drafts

    The Playground answers with the instructions, guardrails and model on screen, saved or not. In Compare, A uses your saved settings and B your draft, each with its own model. Use B copies the draft into your settings.

  • Test as a signed-in visitor

    Enter a user ID, name, email and metadata to chat as a verified visitor in the Playground, without signing in on your site. Server-side actions get the same placeholders as for real visitors.

  • Why this answer, after launch

    Under each reply in a transcript, Why this answer shows a confidence level and the knowledge and actions the reply was built from, so you can find the source of a wrong answer.

  • History to undo a change

    Every save keeps a version of the agent's settings. History compares any version with the current settings line by line, and the owner or an admin can restore an earlier one in one step.

Frequently asked questions

How do I test an AI chatbot before it goes live?

Ask your visitors' most common questions in the Playground, which answers with the settings on screen, saved or not. Then add tests with simulated customers for the conversations that must work, run them all after every change and read the transcripts of the ones that fail. Use Compare to try a new version of the instructions next to the saved one before you save it.

What is a test with simulated customers?

A test describes a customer, their goal and what the agent must do for the chat to pass. An AI plays that customer and chats with your agent, which answers with its real settings and tools. When the chat ends, an AI judge reads it and grades it Pass or Fail, with a reason.

Do tests send emails, book meetings or call my APIs?

No. Hand-offs and live chat requests are simulated, so nobody gets an email, a Slack message or a helpdesk ticket. No calendar is asked and no order is looked up. API actions return simulated data, unless you switch on Allow read-only actions, which lets GET actions really run. Web search, when it is on, really searches.

Can I trust the judge?

Mostly, but not blindly. The judge is an AI too and can be wrong, and the simulated customer doesn't always behave like a real one. Read the transcript before you change your agent because of a fail, and write criteria that can be checked from the chat itself, such as "asks for the order number before looking anything up".

Do tests use my messages?

Each reply your agent gives in a test counts like a message in the Playground: one with the default model, more with a larger model. The simulated customer and the judge don't count. Before you run, the section shows the most messages a run can use with a one-message model, and a test often uses fewer because the customer ends the chat earlier.

Which plans include tests?

Tests with simulated customers are part of the Starter, Pro and Enterprise plans. On Free, the Tests section shows Included from Starter. The Playground, temporary chats and Compare are available on every plan.

Who on my team can run tests?

Editors, admins and the owner can add, edit, delete and run tests. Viewers see the tests and their results but can't change or run them. Only one run per agent can go at a time.

Can I compare two versions of my instructions?

Yes. Switch on Compare in the Playground: A answers with your saved settings and B with an edited draft of the instructions, guardrails and reply language, and each side can use its own model. Click Send to both, and Use B when B answers better. Both chats are temporary, and each send uses the messages of both sides.

Know your agent works before your visitors find out

Create your agent free, try it in the Playground, and add tests with simulated customers from Starter.

Create your agent free

Free plan, no credit card needed.