> For the complete documentation index, see [llms.txt](https://shoppad.gitbook.io/yedric/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://shoppad.gitbook.io/yedric/going-further/simulations.md).
# Simulations
Simulations is a dedicated testing sandbox that lets you experiment with your assistant's behavior before pushing changes live. Run a test message through your assistant, define what a good response looks like, and iterate with confidence.
## How Simulations Work
When you run a simulation, it uses your assistant's current configuration (including the model, tools, and page prompts) so you are testing against what your assistant would actually do in production. You can also test variations by adjusting the configuration before running.
## Saved vs. Ad-hoc Simulations
There are two ways to run tests.
**Saved simulations** are named test scenarios you create and store for ongoing use. You can run them at any time to check that your assistant is still behaving as expected after a change. The **Saved** tab lists all your saved simulations with their last run date and results.
**Ad-hoc runs** are one-off tests you run without saving. Use these when you are experimenting quickly and do not need to preserve the test case. Past ad-hoc runs appear in the **History** tab alongside your saved simulation runs.
## Creating a Simulation
To create a saved simulation, click **New Simulation**. You will fill in:
Create and run simulations.
**Prompt** The message the simulated user sends. Write this the way a real user would phrase the question or request.
**Persona** Optionally describe the type of user sending the message, for example "a first-time customer who has just installed the app" or "an advanced user looking for API documentation." This helps the model generate more realistic behavior.
**Number of runs** How many times to run the simulation. Since AI responses can vary slightly between runs, more runs give you a larger sample to evaluate how consistently your assistant behaves and to catch edge cases. A single run is fine for a quick check; 10-20 runs gives a more reliable picture of pass rate. You can set up to 500 runs.
**Preview page URL** The page URL to simulate the user being on. This affects which page prompt rule applies, so it is useful when you want to test behavior on a specific part of your app.
**Additional information** What the simulated user knows, wants, and will refuse when replying to the assistant's questions. This describes the person in the conversation, not the state of your store or app.
**World facts** Optional backend state that the fabricated tools must treat as already true, for example "The store already has a product 'Panda Socks' with variants Small and Large." Write flat facts about the store or app, not what the user wants or believes; that belongs in Additional information. World facts only work in the positive: a fact that rules something out is ignored, so if a scenario depends on something not existing, add that tool to the whitelist under Tool isolation so it runs for real instead. Leave the field blank and fabricated tools invent everything, as before.
**Halt condition** An optional piece of text that, if found in the assistant's response, stops the simulation at that point rather than continuing the conversation. This is useful when you only care whether a specific phrase or answer appears early in the exchange and do not need to evaluate what follows.
**Tool isolation** Optionally restrict which tools the assistant is allowed to use during this simulation. This lets you test behavior in a controlled environment without other tools interfering.
## Success Criteria
You can define one or more success criteria for a simulation. Each criterion is a rule the assistant's response must satisfy for the run to be counted as passing. The available criterion types are:
* **Tool called**: The assistant must have called a specific tool (optionally with matching arguments).
* **Tool not called**: The assistant must not have called a specific tool.
* **Mentioned**: The assistant's response must contain a specific piece of text.
* **Not mentioned**: The assistant's response must not contain a specific piece of text.
* **Completed within turns**: The conversation must have reached a conclusion within a set number of back-and-forth turns.
* **LLM judge**: Write a custom evaluation prompt that is run by a fast model to assess whether the response meets your criteria. Use this for nuanced judgments that are hard to capture with text matching. You can optionally pin a specific, stronger model to do the judging for that criterion when the default fast judge is not precise enough.
If all criteria pass, the run is marked as passing. Results show you the pass rate across all runs so you can see how consistently the assistant meets your expectations.
Where a criterion accepts a regular expression (a **Tool called** argument matcher, or a **Mentioned** or **Not mentioned** pattern), a pattern that cannot compile is reported as an **invalid regex** error on that criterion, naming the offending pattern, rather than quietly counting as a failed check. Patterns are JavaScript regular expressions, so syntax from other languages, such as the inline `(?i)` flag, is rejected here.
Simulation data is stored in the History tab.
## Suites
Suites let you group multiple saved simulations together and run them all at once. This is useful for regression testing. After making a change to your assistant, run your full suite to confirm that nothing broke across your key scenarios.
To create a suite, navigate to the **Suites** tab, click **New Suite**, and add the simulations you want to include. When you run a suite, each simulation runs independently and the results are aggregated into a summary showing total runs, pass rate, and any failures.
## History
The **History** tab shows all past runs, both saved simulations and ad-hoc tests. You can filter by status (completed, running, failed, cancelled) and expand any run to see the individual conversation results and whether each criterion passed or failed.
## When to Use Simulations
Use Simulations any time you are considering changes to your assistant's configuration and want to validate the impact before going live. They are especially valuable for testing prompt changes, confirming that specific tools are being called correctly, and catching regressions after an update.
