Skip to main content
Input validation allows you to verify that input values match expected criteria. You can validate values set by pre_hooks, use default values from memory, and ensure your agent receives the correct input data.

​
Points System Agent

agent.py
from timbal import Agent
from timbal.state import get_run_context

async def pre_hook():
    """Set premium_user flag based on prompt content."""
    span = get_run_context().current_span()
    prompt = span.input.get("prompt").content[0].text
    if prompt and "premium" in prompt.lower():
        span.input["premium_user"] = True
    return None

async def get_premium_user():
    """Get premium_user status from input."""
    span = get_run_context().current_span()
    return span.input.get("premium_user", False)

agent = Agent(
    name="points_agent",
    model="openai/gpt-4.1-mini",
    system_prompt="""You are a points system assistant.
Premium users have 10 points.
Premium user: {::get_premium_user}""",
    pre_hook=pre_hook,
)

​
Running Evaluations

python -m timbal.eval --fqn agent.py::agent --tests evals.yaml

​
Example: Two Tests Demonstrating Value Persistence

This example demonstrates how to validate input values and use default values from memory. The first test validates that the input has a premium_user key with the correct value, and the second test shows how to use that value as a default in a subsequent test.

​
Test 1: Validate Input Key Value

evals.yaml
- name: eval_input_has_points
  description: Validate that the input has premium_user key set to True
  turns:
    - input:
        prompt: "Hello, I'm a premium user"
        premium_user:
          validators:
            equals: True
      output: "Hi! Nice to meet you."

​
Test 2: Use Default Value from Memory

evals.yaml
- name: eval_input_receives_points_from_memory
  description: Input receives premium_user as default value from memory
  turns:
    - input:
        prompt: "How many points do I have?"
        premium_user: True
      output:
        content:
          validators:
            contains: ["10"]

​
How It Works

  1. First Test: The pre_hook function detects “premium” in the prompt and sets premium_user: True in the input. The validator checks that this value is correctly set.
  2. Second Test: The input uses premium_user: True as a default value. This value is passed to the agent without validation, allowing the agent to use it in the system prompt.

​
Evaluation Results

​
Successful Validation

When both tests pass:
summary.json
{
  "total_files": 1,
  "total_tests": 2,
  "total_validations": 2,
  "inputs_passed": 1,
  "inputs_failed": 0,
  "outputs_passed": 1,
  "outputs_failed": 0,
  "steps_passed": 0,
  "steps_failed": 0,
  "execution_errors": 0,
  "tests_failed": []
}

​
Failed Validation

When the input validation fails or the output doesn’t match expectations:
summary.json
{
  "total_files": 1,
  "total_tests": 2,
  "total_validations": 2,
  "inputs_passed": 0,
  "inputs_failed": 1,
  "outputs_passed": 0,
  "outputs_failed": 1,
  "steps_passed": 0,
  "steps_failed": 0,
  "execution_errors": 0,
  "tests_failed": [
    {
      "test_name": "eval_input_has_points",
      "test_path": "evals.yaml::eval_input_has_points",
      "input": {
        "prompt": [
          "Hello, I'm a premium user"
        ],
        "premium_user": {
          "validators": {
            "equals": true
          }
        }
      },
      "reason": [
        "input"
      ],
      "execution_error": null,
      "input_passed": false,
      "input_explanations": [
        "Input key 'premium_user': Message does not exactly match expected output. Expected: 'True', Got: 'False'"
      ],
      "output_passed": null,
      "output_explanations": [],
      "actual_output": {
        "text": "Actually, you are not currently marked as a premium user. If you believe this is a mistake or need assistance with your premium status, please let me know!",
        "files": []
      },
      "expected_output": {
        "content": [
          "Hi! Nice to meet you."
        ]
      },
      "steps_passed": null,
      "steps_explanations": [],
      "actual_steps": [],
      "expected_steps": null
    },
    {
      "test_name": "eval_input_receives_points_from_memory",
      "test_path": "evals.yaml::eval_input_receives_points_from_memory",
      "input": {
        "prompt": [
          "How many points do I have?"
        ],
        "premium_user": true
      },
      "reason": [
        "output"
      ],
      "execution_error": null,
      "input_passed": null,
      "input_explanations": [],
      "output_passed": false,
      "output_explanations": [
        "Validator contains: Message does not contain '10'."
      ],
      "actual_output": {
        "text": "You currently have 0 points. If you need any help or want to know more about earning points, feel free to ask!",
        "files": []
      },
      "expected_output": {
        "content": {
          "validators": {
            "contains": [
              "10"
            ]
          }
        }
      },
      "steps_passed": null,
      "steps_explanations": [],
      "actual_steps": [],
      "expected_steps": null
    }
  ]
}

​
Key Features

  • Input Validation: Validate input key values using validators (equals, contains, regex)
  • Pre-hook Validation: Validate values set by pre_hook functions
  • Default Values: Pass default values to the agent (key: value)
  • Type Support: Validators support strings, booleans, integers, and floats