Execution Behavior

​

Example

​

Eval Configuration

- name: eval_travel_assistant_workflow
  description: Fetch data in parallel then processes sequentially
  runnable: agent.py::agent
  params:
    prompt: "I'm planning a trip to Madrid. What's the current time, the weather forecast, and flight prices?"
  seq!:
    - llm
    - parallel!:
        - get_datetime:
            input:
              timezone:
                eq!: "Europe/Madrid"
        - get_weather:
            input:
              city:
                eq!: "Madrid"
    - search_flights:
        input:
          destination:
            contains!: "Madrid"
    - llm

​

Agent Implementation

from timbal import Agent

async def get_datetime(timezone: str) -> str:
    """Get current datetime for a timezone."""
    return f"2024-01-15 14:30:00 {timezone}"

async def get_weather(city: str) -> str:
    """Get weather forecast for a city."""
    weather_data = {
        "Madrid": "Sunny, 18°C",
        "Barcelona": "Cloudy, 15°C"
    }
    return weather_data.get(city, "Weather data unavailable")

async def search_flights(destination: str) -> str:
    """Search for flights to a destination."""
    return f"Found flights to {destination}"

agent = Agent(
    name="travel_assistant",
    model="openai/gpt-5.2",
    tools=[get_datetime, get_weather, search_flights],
)

​

Running Evaluations

python -m timbal.evals.cli evals.yaml

​

How It Works

1. Initial Processing: The agent starts with an LLM call to understand the request.
2. Parallel Execution: Two tools (get_datetime and get_weather) execute in parallel using parallel!, improving efficiency by fetching independent data simultaneously.
3. Sequential Processing: After parallel execution completes, search_flights runs sequentially, as it may depend on the previous results.
4. Final Processing: A final LLM call synthesizes all the gathered information into a response.
5. Validation: The seq! validator ensures:
   • Tools execute in the correct order
   • Parallel tools run simultaneously (overlapping execution times)
   • Tool inputs match expected values using nested validators

​

Evaluation Results

​

Successful Validation

──────────────────── Timbal Evals ────────────────────
collected 1 evals from 1 file

 PASSED  evals.yaml::eval_travel_assistant_workflow [4.65s]
└── travel_assistant
    └── ✓ seq!
        ├── llm
        ├── ✓ parallel!
        │   ├── get_datetime
        │   │   └── ✓ input.timezone.eq! ("Europe/Madrid")
        │   └── get_weather
        │       └── ✓ input.city.eq! ("Madrid")
        ├── search_flights
        │   └── ✓ input.destination.contains! ("Madrid")
        └── llm

============================= 1 passed in 4.65s ==============================

​

Failed Validation

──────────────────── Timbal Evals ────────────────────
collected 1 evals from 1 file

 FAILED  evals.yaml::eval_travel_assistant_workflow [4.36s]
└── travel_assistant
    └── ✗ seq!
        ├── llm
        ├── ✗ parallel!
        │   ├── get_datetime
        │   │   └── ✓ input.timezone.eq! ("Europe/Madrid")
        │   └── get_weather
        │       └── ✓ input.city.eq! ("Madrid")
        ├── search_flights
        │   └── ✓ input.destination.contains! ("Madrid")
        └── llm

!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 1 failed in 4.36s !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

​

Key Features

• Nested Flow Validation: Combine seq! and parallel! to validate complex execution patterns
• Parallel Execution: Use parallel! to ensure independent tools run simultaneously for better performance
• Sequential Processing: Use seq! to enforce order when tools depend on previous results
• Tool Input Validation: Validate tool input parameters using nested validators within flow validators
• Span Validation: Validate individual span properties (input, output, elapsed, usage) within sequences
• Workflow Compliance: Ensure agents follow expected execution patterns and optimize tool usage