← Back to Blog

Using Promptfoo to Prevent Ticket Routing Regressions

Test ticket routing with Promptfoo: compare two prompts, catch billing misclassification and case mismatches, and inspect a small real API evaluation.

We use synthetic ticket negative controls to catch routing errors prior to rollout.

The candidate is deliberately broken as a negative control, not a customer incident.

The Configuration

We use promptfoo to compare our baseline prompt against a candidate version. We target the Gemini 3.8 Flash model via the APIBox gateway. Ensure your APIBOX_API_KEY is exported in your environment variables rather than hardcoded. Always include your .env and results.json files in .gitignore to protect sensitive data.

description: Ticket routing regression example
prompts:
  - id: baseline
    raw: 'Classify the ticket as billing or technical. Payment, invoices and refunds are billing. Crashes and connectivity are technical. Return only the label. Ticket: {{ticket}}'
  - id: candidate
    raw: 'Classify the ticket as billing or technical. If it mentions API, choose technical even for an invoice dispute; otherwise payment and refunds are billing. Return only the label. Ticket: {{ticket}}'
providers:
  - id: openai:chat:gemini-3.8-flash
    config:
      apiBaseUrl: https://api.apibox.cc/v1
      apiKeyEnvar: APIBOX_API_KEY
      temperature: 0
      max_tokens: 256
      timeout: 30000
      maxRetries: 0
tests:
  - vars:
      ticket: 'I was charged twice for my API subscription. Please refund the duplicate payment.'
    assert:
      - type: equals
        value: billing
  - vars:
      ticket: 'The API endpoint returns connection refused.'
    assert:
      - type: equals
        value: technical
  - vars:
      ticket: 'Please send me an invoice for last month.'
    assert:
      - type: equals
        value: billing

Running the Test

Execute the test suite using the CLI. We use --no-cache to ensure fresh inference and --max-concurrency 1 for stable, sequential evaluation:

npx --yes promptfoo@0.125.1 eval --no-cache --max-concurrency 1 -o results.json

Analysis of Results

6 inference evaluations using Promptfoo 0.125.1 via APIBox gemini-3.8-flash show: Baseline returned ‘Billing’ for the duplicate-charge ticket, failing on strict equality with ‘billing’. Candidate returned ‘technical’ for the same ticket, failing due to semantic misrouting. The other 4 evaluations passed; errors were 0. The process exited with code 100 due to assertion failures, not provider failure.

Small sample sizes do not prove production accuracy or deterministic model outputs, even at temperature 0 with no cache. —no-cache only ensures fresh requests; normalization should be evaluated separately only if supported by downstream business contracts.

Add anonymized boundary tickets, inspect failures for each label, and repeat evaluation before rollout.

Get a key from the APIBox console; check pricing and your actual bill.

References: OpenAI-compatible provider · Assertions · APIBox model directory

Try it now, sign up and start using 30+ models with one API key

Sign up free →