> ## Documentation Index
> Fetch the complete documentation index at: https://docs.timbal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Overview

> Automated testing framework for Timbal agents and runnables

## What are Evals?

Evals are automated tests that validate your agent's behavior, outputs, and execution patterns. They help you ensure your agents perform correctly and consistently across different scenarios.

## Why Evals Matter

AI agents are non-deterministic - the same input can yield different results. Evals help you:

* **Validate outputs**: Ensure agents produce correct responses
* **Check tool usage**: Verify agents use the right tools with correct inputs
* **Monitor performance**: Track execution time and token usage
* **Catch regressions**: Prevent breaking changes during development
* **Test execution patterns**: Validate sequential and parallel tool execution

## How Evals Work

Timbal's eval system uses a YAML-based test definition format with a powerful validator system:

```yaml theme={"dark"}
- name: time_in_madrid
  description: Test that agent returns the time in Madrid
  runnable: agent.py::agent
  tags: ["datetime", "smoke"]
  timeout: 30000
  
  params:
    prompt: "what time is it in madrid"
  
  output:
    type!: "string"
    contains!: ":"
    pattern!: "\\d{1,2}:\\d{2}"
  
  elapsed:
    lt!: 6000
  
  seq!:
    - llm
    - get_datetime
    - llm
```

### Core Components

<CardGroup cols={2}>
  <Card title="Validators" icon="check" href="/evals/validators">
    20+ validators for checking outputs, patterns, types, and more
  </Card>

  <Card title="Flow Validators" icon="arrow-right" href="/evals/validators/flow">
    Validate execution sequences and parallel tool calls
  </Card>

  <Card title="LLM Validators" icon="brain" href="/evals/validators/llm">
    AI-powered semantic validation for natural language
  </Card>

  <Card title="CLI" icon="terminal" href="/evals/running-evals">
    Command-line interface for running and discovering evals
  </Card>
</CardGroup>

## Prerequisites

Before running evals, ensure the following environment variables are set:

* `TIMBAL_API_KEY` - Your Timbal API key
* `TIMBAL_API_HOST` - The Timbal API host URL
* `TIMBAL_ORG_ID` - Your organization ID

These can be set in your environment or in a `.env` file in your project directory.

## Quick Start

### 1. Create a test file

Create a file named `eval_greeting.yaml`:

```yaml theme={"dark"}
- name: greeting_test
  description: Verify the agent greets users appropriately
  runnable: agent.py::my_agent
  
  params:
    prompt: "Hi there!"
  
  output:
    not_null!: true
    type!: "string"
    prompt!: "The response greets the user politely"
  
  elapsed:
    lt!: 5000
```

### 2. Run your evals

```bash theme={"dark"}
python -m timbal.evals.cli path/to/eval_greeting.yaml
```

### 3. View results

The CLI displays pytest-style output with pass/fail status, duration, and detailed failure information:

```
========================= timbal evals =========================
collected 1 eval

eval_greeting.yaml
  greeting_test ......................................... PASSED (0.45s)
    tags: greeting, basic
    ├── output
    │   ├── not_null! ✓
    │   ├── type! ✓
    │   └── prompt! ✓
    └── elapsed
        └── lt! ✓

========================= 1 passed in 0.45s =========================
```

## Eval Structure

Each eval consists of:

| Field         | Description                            | Required |
| ------------- | -------------------------------------- | -------- |
| `name`        | Unique identifier for the eval         | Yes      |
| `runnable`    | Path to the runnable (`file.py::name`) | Yes      |
| `params`      | Input parameters for the runnable      | No       |
| `description` | Human-readable description             | No       |
| `tags`        | List of tags for filtering             | No       |
| `timeout`     | Maximum execution time in milliseconds | No       |
| `output`      | Validators for the final output        | No       |
| `elapsed`     | Validators for total execution time    | No       |
| `seq!`        | Sequence flow validator                | No       |

<Note>
  Eval names must be unique across all eval files. The CLI will error if duplicate names are found.
</Note>

See [Writing Evals](/evals/writing-evals) for the complete syntax reference.

## Next Steps

<CardGroup cols={2}>
  <Card title="Writing Evals" icon="pen" href="/evals/writing-evals">
    Learn the full eval syntax and best practices
  </Card>

  <Card title="Validators Reference" icon="book" href="/evals/validators">
    Complete reference for all validators
  </Card>

  <Card title="Running Evals" icon="play" href="/evals/running-evals">
    CLI options and CI/CD integration
  </Card>
</CardGroup>