Skip to main content
Validators are the core of Timbal’s eval system. They check specific properties of your agent’s execution and determine whether an eval passes or fails.

​
Validator Syntax

All validators use the name! suffix convention: 
output:
  type!: "string"
  contains!: "hello"
  min_length!: 10
The ! suffix distinguishes validators from regular YAML keys.

​
Transforms

Transforms allow you to normalize values before validation. This is useful for case-insensitive matching or handling whitespace. 
output:
  contains!:
    value: "success"
    transform: lowercase

​
Available Transforms

TransformDescriptionExample
lowercaseConvert to lowercase"Hello" → "hello"
uppercaseConvert to uppercase"Hello" → "HELLO"
trimRemove leading/trailing whitespace" hi " → "hi"
collapse_whitespaceReplace multiple spaces with single"a b" → "a b"

​
Chaining Transforms

Apply multiple transforms in order: 
output:
  contains!:
    value: "hello world"
    transform: [trim, lowercase, collapse_whitespace]
Transforms are applied left-to-right: first trim, then lowercase, then collapse_whitespace.

​
Negation

Most validators support negation using the not_ prefix or the negate field.

​
Using Aliases

output:
  not_contains!: "error"
  not_starts_with!: "Error:"
  ne!: "failed"

​
Using the negate Field

output:
  contains!:
    value: "error"
    negate: true

​
Combining Negation with Transforms

output:
  not_contains!:
    value: "ERROR"
    transform: uppercase

​
Validator Categories

Comparison

String matching, patterns, and equality checks

Type

Type checking, JSON, and format validation

Length

Length constraints and bounds

LLM

AI-powered semantic validation

Flow

Execution sequence and parallelism

​
Quick Reference

ValidatorCategoryDescriptionExample
eq!ComparisonExact equalityeq!: "hello"
ne!ComparisonNot equalne!: "error"
contains!ComparisonSubstring/item checkcontains!: "world"
not_contains!ComparisonAbsence checknot_contains!: "error"
contains_all!ComparisonContains all itemscontains_all!: ["a", "b"]
not_contains_all!ComparisonMissing at least onenot_contains_all!: ["x", "y"]
contains_any!ComparisonContains at least onecontains_any!: ["a", "b"]
not_contains_any!ComparisonContains nonenot_contains_any!: ["x", "y"]
pattern!ComparisonRegex matchpattern!: "\\d+"
not_pattern!ComparisonRegex non-matchnot_pattern!: "^Error"
starts_with!ComparisonPrefix checkstarts_with!: "Hello"
not_starts_with!ComparisonNo prefix matchnot_starts_with!: "Error"
ends_with!ComparisonSuffix checkends_with!: "."
not_ends_with!ComparisonNo suffix matchnot_ends_with!: "error"
lt!ComparisonLess thanlt!: 100
lte!ComparisonLess than or equallte!: 100
gt!ComparisonGreater thangt!: 0
gte!ComparisonGreater than or equalgte!: 1
type!TypeType checkingtype!: "string"
not_type!TypeType exclusionnot_type!: "null"
json!TypeValid JSONjson!: true
email!TypeEmail formatemail!: true
not_null!TypeNon-null checknot_null!: true
length!LengthExact lengthlength!: 10
min_length!LengthMinimum lengthmin_length!: 5
max_length!LengthMaximum lengthmax_length!: 100
semantic!LLMSemantic matchsemantic!: "greeting"
not_semantic!LLMSemantic non-matchnot_semantic!: "rude"
language!LLMLanguage detectionlanguage!: "en"
not_language!LLMLanguage exclusionnot_language!: "fr"
seq!FlowExecution sequenceSee Flow
parallel!FlowParallel executionSee Flow

​
Where Validators Apply

​
Output Validation

output:
  type!: "string"
  not_null!: true
  contains!: "success"

​
Timing Validation

elapsed:
  lt!: 5000
  gte!: 100

​
Span Validation

get_weather:
  input:
    city:
      eq!: "Madrid"
  output:
    type!: "object"
  elapsed:
    lt!: 2000

​
Usage Validation

llm:
  usage:
    input_tokens:
      lte!: 500
    output_tokens:
      lte!: 1000

​
Combining Validators

​
Multiple Validators on One Target

Apply multiple validators to the same target: 
output:
  not_null!: true
  type!: "string"
  min_length!: 10
  contains!: "success"
  not_contains!: "error"
All validators must pass for the eval to succeed.

​
Nested Validation

Validate nested properties in span inputs/outputs: 
search_products:
  input:
    query:
      contains!: "laptop"
    options:
      limit:
        lte!: 100
      category:
        eq!: "electronics"
  output:
    results:
      min_length!: 1

​
Error Messages

When a validator fails, it provides detailed error information: 
FAILED: output
  Validator: contains!
  Expected: "success"
  Actual: "The operation failed due to network error"
  Error: Value does not contain expected substring
For semantic validators: 
FAILED: output
  Validator: semantic!
  Expected: "A polite greeting"
  Actual: "What do you want?"
  Reason: The response is curt and lacks politeness