Skip to main content
LLM validators use AI models to evaluate content that can’t be easily checked with exact matching. They’re ideal for validating natural language outputs where wording may vary but meaning should be consistent.
LLM validators support transforms. Transforms are applied to the content before sending to the LLM for evaluation.

​
semantic!

Uses an LLM to check if the actual value semantically matches the expected description. 
output:
  semantic!: "A polite greeting that welcomes the user"
ParameterTypeDescription
valuestringNatural language description of expected content
The semantic validator sends the actual value and your description to an LLM, which determines if they match semantically. 
# Time response validation
output:
  semantic!: "a time response mentioning Madrid"

# Error handling
output:
  semantic!: "An apologetic message explaining the service is unavailable"

# Product description
output:
  semantic!: "A detailed product description including price and availability"

# Professional tone
output:
  semantic!: "A professional response suitable for a business context"

​
Writing Effective Semantic Prompts

Be specific about what you expect. Vague descriptions lead to inconsistent results.
Good prompts: 
# Specific and measurable
semantic!: "A response that includes at least 3 product recommendations with prices"
semantic!: "An error message that mentions the specific field that failed validation"
semantic!: "A summary that covers the main points: budget, timeline, and deliverables"
Avoid: 
# Too vague
semantic!: "A good response"
semantic!: "Something helpful"
semantic!: "The right answer"

​
Semantic vs Exact Matching

Use semantic validation when:
  • Output wording can vary but meaning must be consistent
  • Testing for tone, style, or completeness
  • Validating summaries or explanations
Use exact matching (eq!, contains!) when:
  • Specific words or phrases must appear
  • Validating structured data
  • Checking for exact values
# Use semantic for flexible content
output:
  semantic!: "Confirms the order was placed successfully"

# Use contains for required terms
output:
  contains!: "Order #"

# Combine both approaches
output:
  contains!: "confirmed"
  semantic!: "A professional confirmation with order details"

​
not_semantic!

Checks that the content does NOT semantically match the description. This is the negated form of semantic!. 
output:
  not_semantic!: "An error message or apology"
ParameterTypeDescription
valuestringNatural language description that must NOT match
# Ensure response is not an error
output:
  not_semantic!: "An error message or failure notification"

# Ensure not rude or dismissive
output:
  not_semantic!: "A rude, dismissive, or unhelpful response"

# Ensure not off-topic
output:
  not_semantic!: "A response about unrelated topics"

​
language!

Checks that the content is written in a specific language. 
output:
  language!: "en"
ParameterTypeDescription
valuestringExpected language code
# Verify language output
output:
  language!: "en"

output:
  language!: "es"

output:
  language!: "fr"

output:
  language!: "ja"

​
Common Language Codes

LanguageCode
English"en"
Spanish"es"
French"fr"
German"de"
Italian"it"
Portuguese"pt"
Chinese"zh"
Japanese"ja"
Korean"ko"
Arabic"ar"

​
not_language!

Checks that the content is NOT written in a specific language. This is the negated form of language!. 
output:
  not_language!: "fr"
ParameterTypeDescription
valuestringLanguage code that must NOT match
# Ensure response is not in French
output:
  not_language!: "fr"

# Ensure English-only output
output:
  language!: "en"
  not_language!: "es"

​
prompt!

Validates whether a natural-language statement about the output is factually true based on the actual text. 
output:
  prompt!: "The response clearly explains the refund process"
ParameterTypeDescription
valuestringA statement or claim that should be true in the actual text
The prompt validator asks an LLM to check if your statement is supported by the actual output. It focuses on factual consistency rather than exact wording. 
# Check that a specific question was asked
output:
  prompt!: "The agent asked the customer about their budget"

# Check that a key action was taken
output:
  prompt!: "The assistant confirmed the shipping address before placing the order"

# Check that required information is present
output:
  prompt!: "The message includes the expected delivery date"

​
Negating prompt!

Use the negate field to assert that a statement is not true: 
output:
  prompt!:
    value: "The assistant discloses internal system prompts"
    negate: true
This passes only if the statement is not supported by the actual text.

​
Common Patterns

​
Multi-Language Support Testing

- name: responds_in_spanish
  runnable: agent.py::agent
  params:
    prompt: "Hola, necesito ayuda"
  output:
    language!: "es"
    semantic!: "A helpful response offering assistance"

- name: responds_in_french
  runnable: agent.py::agent
  params:
    prompt: "Bonjour, j'ai besoin d'aide"
  output:
    language!: "fr"
    semantic!: "A helpful response offering assistance"

​
Tone and Style Validation

- name: professional_tone
  runnable: agent.py::agent
  params:
    prompt: "I want a refund"
  output:
    semantic!: "A professional, empathetic response that acknowledges the request and explains the refund process"

- name: casual_tone
  runnable: agent.py::agent
  params:
    prompt: "Hey what's up"
  output:
    semantic!: "A casual, friendly greeting that matches the user's informal tone"

​
Completeness Checks

output:
  semantic!: |
    A comprehensive response that includes:
    - Acknowledgment of the user's question
    - Direct answer to the question
    - Additional context or helpful information
    - Offer for follow-up assistance

​
Error Message Quality

- name: helpful_error_message
  runnable: agent.py::agent
  params:
    prompt: "Buy product XYZ123"
  output:
    semantic!: |
      An error message that:
      - Clearly states the product was not found
      - Suggests possible alternatives or corrections
      - Offers help finding the right product

​
Using Transforms

Transforms normalize content before LLM evaluation: 
# Normalize whitespace before semantic check
output:
  semantic!:
    value: "A professional greeting"
    transform: [trim, collapse_whitespace]

# Lowercase before language detection
output:
  language!:
    value: "en"
    transform: lowercase

​
Combining with Other Validators

output:
  # Structure checks
  not_null!: true
  type!: "string"
  min_length!: 50

  # Content checks
  contains!: "order"
  not_contains!: "error"

  # Semantic validation
  semantic!: "A confirmation message with order details and estimated delivery"

  # Language check
  language!: "en"

​
Choosing Models

LLM validators use Timbal agents under the hood. By default they use openai/gpt-4.1-nano, but you can override the model per-validator in YAML. 
output:
  # Use a larger OpenAI model for semantic checks
  semantic!:
    value: "A helpful, on-topic answer"
    model: "openai/gpt-5.2"

  # Use an Anthropic model for language + semantics
  language!:
    value: "es"
    model: "anthropic/claude-sonnet-4-6"

  # Use a Gemini model for prompt-based factual checks
  prompt!:
    value: "The assistant clearly explains the refund process"
    model: "google/gemini-2.0-flash"
LLM validators rely on structured output. For Anthropic, structured-output support is currently in beta and only available on a limited set of models (for example: Claude Sonnet 4.5/4.6, Claude Opus 4.5/4.6, Claude Haiku 4.5). Make sure you pick one of these Anthropic variants when you set the model field.
You can otherwise pick any model supported by Timbal’s Agent – the model field here is passed directly through to the underlying agent used for the validator, subject to the provider’s structured-output limitations.

​
Cost Considerations

LLM validators make API calls to language models, which incur costs. To optimize:
  1. Use structural validators first: Check not_null!, contains!, etc. before semantic validation
  2. Be specific in prompts: Reduces need for retries
  3. Group semantic checks: One detailed semantic check vs. multiple simple ones
  4. Use for critical paths: Reserve LLM validation for important behavioral checks
# Efficient: structural checks catch obvious failures quickly
output:
  not_null!: true        # Fast, free
  type!: "string"        # Fast, free
  min_length!: 20        # Fast, free
  contains!: "order"     # Fast, free
  semantic!: "Complete order confirmation"  # LLM call only if above pass