> ## Documentation Index > Fetch the complete documentation index at: https://docs.getnao.io/llms.txt > Use this file to discover all available pages before exploring further. # Evaluation > Test and evaluate your analytics agent with unit tests ## Overview The `nao test` command allows you to measure your agent's performance on a set of unit tests created by you. It's meant to help you monitor and improve your context's quality over time. `nao test` is your **offline eval**: a fixed suite you run before shipping and in CI to catch regressions against a benchmark you control. Its online counterpart is [Recommendations](/nao-agent/context-engineering/recommendations), which mines real production usage to surface the gaps you didn't think to test. Use both to close the feedback loop between shipping context and improving it. ## nao test The `nao test` command runs unit tests from your `tests/` folder, executes them against your agent, and compares results to verify correctness. ### Create unit tests Create a `tests/` folder in your project root: ``` your-project/ ├── nao_config.yaml ├── RULES.md ├── tests/ # Test folder │ ├── total_revenue.yml # Test file 1 │ ├── customer_metrics.yml # Test file 2 │ └── outputs/ # Test results (auto-generated) │ └── results_20250209_143022.json ``` Then create your test files. Each test is a YAML file in the `tests/` folder. Test files should have a `.yml` or `.yaml` extension. Test files follow this template: ```yaml theme={null} name: total_revenue prompt: What is the total revenue from all orders? sql: | SELECT SUM(amount) as total_revenue FROM orders ``` **Required fields:** * `name`: A descriptive name for the test * `prompt`: The question or prompt to test * `sql`: SQL query which produces the right data ### Launch nao test command Before running `nao test`: * **Start the nao chat server** (for example with `nao chat` or your usual local setup) so that the backend API is available. * On the **first `nao test` run**, the CLI will prompt you to log in in your browser — **use the same account you use in the local nao chat interface**, so tests run under the same project and permissions. Run all tests: ```bash theme={null} nao test ``` This will: * Discover all `.yml` and `.yaml` files in the `tests/` folder * Run each test against the configured models, `openai:gpt-4.1` by default * Display results in a summary table * Save detailed results to `tests/outputs/results_TIMESTAMP.json` **Specify LLM model to test:** ```bash theme={null} # Test with GPT 4.1 nao test -m openai:gpt-4.1 ``` Model format: `provider:model_id`. The provider must be one you configured in the `llm` block of `nao_config.yaml` or in the admin UI - see [LLM providers and models](/nao-agent/context-builder/configuration#llm-providers-and-models). Repeat the flag to compare several models on the same suite. Every test then runs once per model, and the summary table has one row per test and model: ```bash theme={null} nao test -m openai:gpt-4.1 -m anthropic:claude-sonnet-4-5 ``` A multi-model run also prints two comparison tables: **Performance by Model**, one row per model ranked by pass rate then cost, and **Pass / Fail by Test and Model**, a grid showing which model passes which test. Both are described in [Test Outputs](#test-outputs). **Run tests in parallel:** ```bash theme={null} # Run with 4 parallel threads nao test -t 4 ``` This speeds up execution when running many tests, but output may be interleaved. **Run specific tests:** ```bash theme={null} # Run one test by test name or YAML filename stem nao test -s total_revenue ``` You can also use the long flag: ```bash theme={null} nao test --select total_revenue ``` Pass a comma-separated list to run several named tests in one go. Selection order is preserved, and surrounding whitespace and duplicates are tolerated: ```bash theme={null} nao test -s total_revenue,customer_metrics ``` A selection also matches a subfolder of `tests/`, which runs every test it contains: ```bash theme={null} # Runs all tests under tests/contracts/ nao test -s contracts ``` An unknown name errors and prints the available tests. **Run non-interactively (CI):** By default, the first `nao test` run opens a browser to log in. For CI pipelines, pass credentials directly so the run never blocks on a prompt: ```bash theme={null} # Inline flags nao test -u user@example.com --password '' # Or via environment variables (recommended in CI - # the password never appears in the command line or in shell history) export NAO_USERNAME=user@example.com export NAO_PASSWORD='' nao test ``` The credentials are reused on session refresh if the auth token expires mid-run. The password is never echoed to stdout or stderr. ### Defaults in nao\_config.yaml Rather than repeating the same flags on every run, declare your defaults in the `test` block of `nao_config.yaml` and commit them with the suite: ```yaml theme={null} test: models: - openai:gpt-4.1 - anthropic:claude-sonnet-4-5 threads: 4 comparison: rtol: 0.00001 atol: 0.00000001 decimals: 2 ``` Every key is optional, and the matching command line flag overrides it for a single run: | Key | Default | Flag | Description | | ------------ | -------------------- | ----------------- | ----------------------------------------------------------- | | `models` | `["openai:gpt-4.1"]` | `-m`, `--model` | Models to run the suite against, as `provider:model_id` | | `threads` | `1` | `-t`, `--threads` | Test runs executed in parallel | | `comparison` | see below | — | Tolerances used when comparing results to the expected data | Which tests to run stays a per-run decision with `-s`/`--select`, and credentials stay out of the config: pass them with `-u`/`--password` or the `NAO_USERNAME` / `NAO_PASSWORD` environment variables. ### Comparison tolerances `test.comparison` controls how the agent's data is compared to the result of your `sql` query: | Key | Default | Description | | ---------- | ------------ | --------------------------------------------------------- | | `rtol` | `0.00001` | Relative tolerance for numeric values | | `atol` | `0.00000001` | Absolute tolerance for numeric values | | `decimals` | `2` | Decimals kept when rounding float values before comparing | Raise `decimals` when a test legitimately hinges on more precision than two decimals, and lower it when rounding differences between the agent's SQL and your reference SQL cause false failures. ### Test costs The cost of each run comes from nao's built-in price table for the model under test. When you declare a model under `llm.providers[].models` with a `costs` block, those prices win — which is how you get accurate costs for a model nao doesn't know, such as an alias exposed by a LiteLLM proxy: ```yaml theme={null} llm: providers: - provider: openai api_key: {{ env('LITELLM_API_KEY') }} base_url: http://0.0.0.0:4000 models: - id: my-proxy-alias costs: input_no_cache: 2.5 output: 10 ``` Then run the suite against it with `nao test -m openai:my-proxy-alias`. Models with no price anywhere report an empty cost, while tokens, duration and pass/fail are still recorded. See [Model costs](/nao-agent/context-builder/configuration#model-costs). The deprecated `llm.meta.costs` block is still read as a last-resort price for every model. Move those prices onto the models they belong to so a multi-model run reports real numbers. ### Test mode behavior When `nao test` runs a prompt, the agent operates in **test mode**. In this mode the clarification tool is removed from the agent's toolset, so the agent cannot ask follow-up questions. Instead, it makes reasonable assumptions or states that it cannot answer. This prevents tests from hanging on a clarification prompt and ensures every test produces a deterministic result. Interactive chat is unaffected: the clarification tool remains available in normal conversations. ### How It Works 1. **Test Discovery**: Scans the `tests/` folder for `.yml` and `.yaml` files 2. **Test Execution**: For each test: * Sends the prompt to your agent in test mode (clarification disabled, see above) * Captures the agent's full conversation history, tool calls, and response text 3. **Data Verification**: * **Extract actual data**: rather than asking the model to retype its answer as JSON (which loses precision on large or wide results), nao loads **every `execute_sql` result from the run into an in-memory DuckDB**, one table named after each query id, and asks the model for a **DuckDB query over those rows** that returns the final answer. The query runs locally against the data the agent already fetched.
Rows are loaded as newline-delimited JSON so DuckDB infers the column types instead of nao guessing them. If the model's query fails, **one repair attempt** is allowed: the error is sent back and the model gets a second try. * The verification query produces **structured data** matching the expected columns. * **Execute expected SQL**: the `sql` query from your test file is executed against your database to get the expected results * **Compare data**: the agent answer's data and expected data (from SQL execution) are compared 4. **Data Comparison Process**: * **Normalize datasets**: Both datasets are converted to DataFrames and normalized (resets index, infers types, and sorts columns alphabetically) * **Ignore row order**: Rows are sorted by all columns before comparison so equivalent results with different row order still pass * **Row count match**:If row count doesn't match, they are not compared * **Round floats**: Float columns are rounded to `comparison.decimals` (2 by default) so display-level precision differences don't fail a test * **Exact match**: First attempts exact equality comparison * **Approximate match**: For numeric columns, uses numpy's `allclose` with the `comparison.rtol` and `comparison.atol` tolerances (1e-5 and 1e-8 by default) to handle floating-point differences * **Diff generation**: If both comparisons fail, generates a detailed diff showing where values differ 5. **Result Collection**: Collects metrics including: * Pass/fail status of the data diff * Token usage and costs (inputs and outputs of the LLM), priced as described in [Test costs](#test-costs) * Execution duration * Tool call count ### Test Outputs **Console Output** The command displays a summary table with: * Test name * Model used * **Pass/fail** status * Message (e.g., "match", "values differ") * **Token usage** * **Cost** * **Execution time** * **Tool call count** * A final summary with total passed/failed counts **Example output:** nao test command output When the run covers more than one model, two extra tables follow it. **Performance by Model** aggregates every run of a model, ranked by pass rate and then by cost, so the cheapest model wins a tie. It reports pass rate, passed runs out of total, tokens, cost, average duration and average tool calls: ``` Performance by Model ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┓ ┃ ┃ Pass ┃ ┃ ┃ ┃ Avg ┃ Avg ┃ ┃ Model ┃ Rate ┃ Passed ┃ Tokens ┃ Cost ┃ Time (s)┃ Tools ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━┩ │ anthropic │ 100.0% │ 2/2 │ 46000 │ 0.089 │ 8.8 │ 5.5 │ │ claude-sonnet-4-5 │ │ │ │ │ │ │ │ openai │ 50.0% │ 1/2 │ 73390 │ 0.138 │ 13.6 │ 8.0 │ │ gpt-4.1 │ │ │ │ │ │ │ └───────────────────┴─────────┴────────┴────────┴─────────┴─────────┴──────────┘ ``` **Pass / Fail by Test and Model** puts one row per test and one column per model, which is how you spot a test that only one model gets right: ``` Pass / Fail by Test and Model ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓ ┃ ┃ openai ┃ anthropic ┃ ┃ Test ┃ gpt-4.1 ┃ claude-sonnet-4-5 ┃ ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩ │ total_revenue │ ✓ │ ✓ │ │ revenue_per_customer │ ✗ │ ✓ │ └──────────────────────┴─────────┴───────────────────┘ ``` With `-t`/`--threads`, live output is still interleaved but these tables always list runs grouped by model, in the order the models were configured. **JSON Results File** Detailed results are saved to `tests/outputs/results_TIMESTAMP.json`. Each result carries a `details.reference_sql` field holding the `sql` query from the test YAML file, including for tests that errored out: ```json theme={null} { "timestamp": "2025-02-09T14:30:22.123456", "results": [ { "name": "total_revenue", "model": "openai:gpt-4.1", "passed": true, "message": "match", "tokens": 1250, "cost": 0.0125, "duration_ms": 234, "tool_call_count": 1, "details": { "response_text": "...", "actual_data": [...], "expected_data": [...], "reference_sql": "SELECT SUM(amount) as total_revenue\nFROM orders", "tool_calls": [...] } } ], "summary": { "total": 3, "passed": 3, "failed": 0, "total_tokens": 3750, "total_cost": 0.0375, "total_duration_ms": 702, "total_duration_s": 0.7, "total_tool_calls": 3, "avg_duration_ms": 234, "avg_tool_calls": 1.0 }, "by_model": [ { "model": "openai:gpt-4.1", "total": 3, "passed": 3, "failed": 0, "pass_rate": 100.0, "total_tokens": 3750, "total_cost": 0.0375, "total_duration_ms": 702, "avg_duration_ms": 234, "total_tool_calls": 3, "avg_tool_calls": 1.0 } ] } ``` `summary` aggregates every run of the file, while `by_model` holds the same metrics per model, ranked exactly like the **Performance by Model** table. Use it to assert a per-model pass rate in CI instead of recomputing it from `results`. ## nao test server The `nao test server` command starts a web server to explore test results in a visual interface. The test server provides: * **Summary Dashboard**: Overview cards showing pass rate, total tests, tokens, costs, and duration * **Performance by Model**: for multi-model runs, one row per model with its pass rate, passed runs, tokens, cost, average duration and average tool calls. Click a row to filter the run table down to that model * **Pass / Fail by Test and Model**: for multi-model runs, a grid of tests against models. Each cell shows the status and duration, and opens the detailed view of that run * **Results Table**: Interactive table of all test runs with status, metrics, and details, filterable by model * **Detailed View**: Click any test to see: * Full response text * **Reference SQL**: the `sql` query defined in the test YAML file, so you can compare it side by side with what the agent ran. It is shown even when the test errored out. * Tool calls with arguments and results. `execute_sql` calls render their `sql_query` as formatted multi-line SQL instead of a JSON blob, with any remaining arguments listed separately. * Data comparison (actual vs expected) * Diff view for failed tests * Performance metrics **Test server UI** nao test server **Zoom on one test** nao test server zoom on one test Start the Server with: ```bash theme={null} nao test server ``` This will start the test server on `http://localhost:8765` The test server reads from `tests/outputs/`. Make sure you've run `nao test` at least once to generate result files. Result files written before per-model summaries existed still get the model views: the server computes `by_model` on the fly when a file doesn't carry it. ## Best Practices ### Creating Effective Tests 1. **Start with critical queries**: Test the most important questions your users ask 2. **Cover edge cases**: Include tests for boundary conditions and complex scenarios 3. **Keep tests focused**: Each test should verify one specific behavior 4. **Avoid overfitting and leakage**: Avoid including exact answers or overly specific details in your context that would allow the agent to "cheat" by pattern matching rather than actually understanding the context. ### Integrating into Workflow 1. **Version control**: Commit your `tests/` folder to git 2. **CI/CD integration**: Run tests automatically on context changes 3. **Regular evaluation**: Run tests weekly or after major context updates 4. **Track trends**: Monitor pass rates and costs over time Learn how to integrate testing into your context engineering workflow