Evaluation

Overview

The nao test command allows you to measure your agent’s performance on a set of unit tests created by you. It’s meant to help you monitor and improve your context’s quality over time. nao test is your offline eval: a fixed suite you run before shipping and in CI to catch regressions against a benchmark you control. Its online counterpart is Recommendations, which mines real production usage to surface the gaps you didn’t think to test. Use both to close the feedback loop between shipping context and improving it.

nao test

The nao test command runs unit tests from your tests/ folder, executes them against your agent, and compares results to verify correctness.

Create unit tests

Create a tests/ folder in your project root:

your-project/
├── nao_config.yaml
├── RULES.md
├── tests/                          # Test folder
│   ├── total_revenue.yml          # Test file 1
│   ├── customer_metrics.yml       # Test file 2
│   └── outputs/                   # Test results (auto-generated)
│       └── results_20250209_143022.json

Then create your test files. Each test is a YAML file in the tests/ folder. Test files should have a .yml or .yaml extension. Test files follow this template:

name: total_revenue
prompt: What is the total revenue from all orders?
sql: |
  SELECT SUM(amount) as total_revenue
  FROM orders

Required fields:

name: A descriptive name for the test
prompt: The question or prompt to test
sql: SQL query which produces the right data

Launch nao test command

Before running nao test:

Start the nao chat server (for example with nao chat or your usual local setup) so that the backend API is available.
On the first nao test run, the CLI will prompt you to log in in your browser — use the same account you use in the local nao chat interface, so tests run under the same project and permissions.

Run all tests:

nao test

This will:

Discover all .yml and .yaml files in the tests/ folder
Run each test against the configured models, openai:gpt-4.1 by default
Display results in a summary table
Save detailed results to tests/outputs/results_TIMESTAMP.json

Specify LLM model to test:

# Test with GPT 4.1
nao test -m openai:gpt-4.1

Model format: provider:model_id. The provider must be one you configured in the llm block of nao_config.yaml or in the admin UI - see LLM providers and models. Repeat the flag to compare several models on the same suite. Every test then runs once per model, and the summary table has one row per test and model:

nao test -m openai:gpt-4.1 -m anthropic:claude-sonnet-4-5

A multi-model run also prints two comparison tables: Performance by Model, one row per model ranked by pass rate then cost, and Pass / Fail by Test and Model, a grid showing which model passes which test. Both are described in Test Outputs. Run tests in parallel:

# Run with 4 parallel threads
nao test -t 4

This speeds up execution when running many tests, but output may be interleaved. Run specific tests:

# Run one test by test name or YAML filename stem
nao test -s total_revenue

You can also use the long flag:

nao test --select total_revenue

Pass a comma-separated list to run several named tests in one go. Selection order is preserved, and surrounding whitespace and duplicates are tolerated:

nao test -s total_revenue,customer_metrics

A selection also matches a subfolder of tests/, which runs every test it contains:

# Runs all tests under tests/contracts/
nao test -s contracts

An unknown name errors and prints the available tests. Run non-interactively (CI): By default, the first nao test run opens a browser to log in. For CI pipelines, pass credentials directly so the run never blocks on a prompt:

# Inline flags
nao test -u user@example.com --password '<password>'

# Or via environment variables (recommended in CI -
# the password never appears in the command line or in shell history)
export NAO_USERNAME=user@example.com
export NAO_PASSWORD='<password>'
nao test

The credentials are reused on session refresh if the auth token expires mid-run. The password is never echoed to stdout or stderr.

Defaults in nao_config.yaml

Rather than repeating the same flags on every run, declare your defaults in the test block of nao_config.yaml and commit them with the suite:

test:
  models:
    - openai:gpt-4.1
    - anthropic:claude-sonnet-4-5
  threads: 4
  comparison:
    rtol: 0.00001
    atol: 0.00000001
    decimals: 2

Every key is optional, and the matching command line flag overrides it for a single run:

Key	Default	Flag	Description
`models`	`["openai:gpt-4.1"]`	`-m`, `--model`	Models to run the suite against, as `provider:model_id`
`threads`	`1`	`-t`, `--threads`	Test runs executed in parallel
`comparison`	see below	—	Tolerances used when comparing results to the expected data

Which tests to run stays a per-run decision with -s/--select, and credentials stay out of the config: pass them with -u/--password or the NAO_USERNAME / NAO_PASSWORD environment variables.

Comparison tolerances

test.comparison controls how the agent’s data is compared to the result of your sql query:

Key	Default	Description
`rtol`	`0.00001`	Relative tolerance for numeric values
`atol`	`0.00000001`	Absolute tolerance for numeric values
`decimals`	`2`	Decimals kept when rounding float values before comparing

Raise decimals when a test legitimately hinges on more precision than two decimals, and lower it when rounding differences between the agent’s SQL and your reference SQL cause false failures.

Test costs

The cost of each run comes from nao’s built-in price table for the model under test. When you declare a model under llm.providers[].models with a costs block, those prices win — which is how you get accurate costs for a model nao doesn’t know, such as an alias exposed by a LiteLLM proxy:

llm:
  providers:
    - provider: openai
      api_key: {{ env('LITELLM_API_KEY') }}
      base_url: http://0.0.0.0:4000
      models:
        - id: my-proxy-alias
          costs:
            input_no_cache: 2.5
            output: 10

Then run the suite against it with nao test -m openai:my-proxy-alias. Models with no price anywhere report an empty cost, while tokens, duration and pass/fail are still recorded. See Model costs.

The deprecated llm.meta.costs block is still read as a last-resort price for every model. Move those prices onto the models they belong to so a multi-model run reports real numbers.

Test mode behavior

When nao test runs a prompt, the agent operates in test mode. In this mode the clarification tool is removed from the agent’s toolset, so the agent cannot ask follow-up questions. Instead, it makes reasonable assumptions or states that it cannot answer. This prevents tests from hanging on a clarification prompt and ensures every test produces a deterministic result. Interactive chat is unaffected: the clarification tool remains available in normal conversations.

How It Works

Test Discovery: Scans the tests/ folder for .yml and .yaml files
Test Execution: For each test:
- Sends the prompt to your agent in test mode (clarification disabled, see above)
- Captures the agent’s full conversation history, tool calls, and response text
Data Verification:
- Extract actual data: rather than asking the model to retype its answer as JSON (which loses precision on large or wide results), nao loads every execute_sql result from the run into an in-memory DuckDB, one table named after each query id, and asks the model for a DuckDB query over those rows that returns the final answer. The query runs locally against the data the agent already fetched.
  Rows are loaded as newline-delimited JSON so DuckDB infers the column types instead of nao guessing them. If the model’s query fails, one repair attempt is allowed: the error is sent back and the model gets a second try.
- The verification query produces structured data matching the expected columns.
- Execute expected SQL: the sql query from your test file is executed against your database to get the expected results
- Compare data: the agent answer’s data and expected data (from SQL execution) are compared
Data Comparison Process:
- Normalize datasets: Both datasets are converted to DataFrames and normalized (resets index, infers types, and sorts columns alphabetically)
- Ignore row order: Rows are sorted by all columns before comparison so equivalent results with different row order still pass
- Row count match:If row count doesn’t match, they are not compared
- Round floats: Float columns are rounded to comparison.decimals (2 by default) so display-level precision differences don’t fail a test
- Exact match: First attempts exact equality comparison
- Approximate match: For numeric columns, uses numpy’s allclose with the comparison.rtol and comparison.atol tolerances (1e-5 and 1e-8 by default) to handle floating-point differences
- Diff generation: If both comparisons fail, generates a detailed diff showing where values differ
Result Collection: Collects metrics including:
- Pass/fail status of the data diff
- Token usage and costs (inputs and outputs of the LLM), priced as described in Test costs
- Execution duration
- Tool call count

Test Outputs

Console Output The command displays a summary table with:

Test name
Model used
Pass/fail status
Message (e.g., “match”, “values differ”)
Token usage
Cost
Execution time
Tool call count
A final summary with total passed/failed counts

Example output:

When the run covers more than one model, two extra tables follow it. Performance by Model aggregates every run of a model, ranked by pass rate and then by cost, so the cheapest model wins a tie. It reports pass rate, passed runs out of total, tokens, cost, average duration and average tool calls:

                      Performance by Model
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┓
┃                   ┃ Pass    ┃        ┃        ┃         ┃ Avg     ┃ Avg      ┃
┃ Model             ┃ Rate    ┃ Passed ┃ Tokens ┃ Cost    ┃ Time (s)┃ Tools    ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━┩
│ anthropic         │ 100.0%  │ 2/2    │ 46000  │ 0.089   │ 8.8     │ 5.5      │
│ claude-sonnet-4-5 │         │        │        │         │         │          │
│ openai            │ 50.0%   │ 1/2    │ 73390  │ 0.138   │ 13.6    │ 8.0      │
│ gpt-4.1           │         │        │        │         │         │          │
└───────────────────┴─────────┴────────┴────────┴─────────┴─────────┴──────────┘

Pass / Fail by Test and Model puts one row per test and one column per model, which is how you spot a test that only one model gets right:

              Pass / Fail by Test and Model
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃                      ┃ openai  ┃ anthropic         ┃
┃ Test                 ┃ gpt-4.1 ┃ claude-sonnet-4-5 ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ total_revenue        │ ✓       │ ✓                 │
│ revenue_per_customer │ ✗       │ ✓                 │
└──────────────────────┴─────────┴───────────────────┘

With -t/--threads, live output is still interleaved but these tables always list runs grouped by model, in the order the models were configured. JSON Results File Detailed results are saved to tests/outputs/results_TIMESTAMP.json. Each result carries a details.reference_sql field holding the sql query from the test YAML file, including for tests that errored out:

{
  "timestamp": "2025-02-09T14:30:22.123456",
  "results": [
    {
      "name": "total_revenue",
      "model": "openai:gpt-4.1",
      "passed": true,
      "message": "match",
      "tokens": 1250,
      "cost": 0.0125,
      "duration_ms": 234,
      "tool_call_count": 1,
      "details": {
        "response_text": "...",
        "actual_data": [...],
        "expected_data": [...],
        "reference_sql": "SELECT SUM(amount) as total_revenue\nFROM orders",
        "tool_calls": [...]
      }
    }
  ],
  "summary": {
    "total": 3,
    "passed": 3,
    "failed": 0,
    "total_tokens": 3750,
    "total_cost": 0.0375,
    "total_duration_ms": 702,
    "total_duration_s": 0.7,
    "total_tool_calls": 3,
    "avg_duration_ms": 234,
    "avg_tool_calls": 1.0
  },
  "by_model": [
    {
      "model": "openai:gpt-4.1",
      "total": 3,
      "passed": 3,
      "failed": 0,
      "pass_rate": 100.0,
      "total_tokens": 3750,
      "total_cost": 0.0375,
      "total_duration_ms": 702,
      "avg_duration_ms": 234,
      "total_tool_calls": 3,
      "avg_tool_calls": 1.0
    }
  ]
}

summary aggregates every run of the file, while by_model holds the same metrics per model, ranked exactly like the Performance by Model table. Use it to assert a per-model pass rate in CI instead of recomputing it from results.

nao test server

The nao test server command starts a web server to explore test results in a visual interface. The test server provides:

Summary Dashboard: Overview cards showing pass rate, total tests, tokens, costs, and duration
Performance by Model: for multi-model runs, one row per model with its pass rate, passed runs, tokens, cost, average duration and average tool calls. Click a row to filter the run table down to that model
Pass / Fail by Test and Model: for multi-model runs, a grid of tests against models. Each cell shows the status and duration, and opens the detailed view of that run
Results Table: Interactive table of all test runs with status, metrics, and details, filterable by model
Detailed View: Click any test to see:
- Full response text
- Reference SQL: the sql query defined in the test YAML file, so you can compare it side by side with what the agent ran. It is shown even when the test errored out.
- Tool calls with arguments and results. execute_sql calls render their sql_query as formatted multi-line SQL instead of a JSON blob, with any remaining arguments listed separately.
- Data comparison (actual vs expected)
- Diff view for failed tests
- Performance metrics

Test server UI

Zoom on one test

Start the Server with:

nao test server

This will start the test server on http://localhost:8765

The test server reads from tests/outputs/. Make sure you’ve run nao test at least once to generate result files.

Result files written before per-model summaries existed still get the model views: the server computes by_model on the fly when a file doesn’t carry it.

Best Practices

Creating Effective Tests

Start with critical queries: Test the most important questions your users ask
Cover edge cases: Include tests for boundary conditions and complex scenarios
Keep tests focused: Each test should verify one specific behavior
Avoid overfitting and leakage: Avoid including exact answers or overly specific details in your context that would allow the agent to “cheat” by pattern matching rather than actually understanding the context.

Integrating into Workflow

Version control: Commit your tests/ folder to git
CI/CD integration: Run tests automatically on context changes
Regular evaluation: Run tests weekly or after major context updates
Track trends: Monitor pass rates and costs over time

Context Engineering Playbook

Learn how to integrate testing into your context engineering workflow

Get Started

Context Builder

Context Engineering

Chat Interface

Connectors

Self-Hosting

nao Cloud

nao Enterprise

Developers

Overview

nao test

Create unit tests

Launch nao test command

Defaults in nao_config.yaml

Comparison tolerances

Test costs

Test mode behavior

How It Works

Test Outputs

nao test server

Best Practices

Creating Effective Tests

Integrating into Workflow

Context Engineering Playbook

​Overview

​nao test

​Create unit tests

​Launch nao test command

​Defaults in nao_config.yaml

​Comparison tolerances

​Test costs

​Test mode behavior

​How It Works

​Test Outputs

​nao test server

​Best Practices

​Creating Effective Tests

​Integrating into Workflow

Context Engineering Playbook

Overview

nao test

Create unit tests

Launch nao test command

Defaults in nao_config.yaml

Comparison tolerances

Test costs

Test mode behavior

How It Works

Test Outputs

nao test server

Best Practices

Creating Effective Tests

Integrating into Workflow