Agent Response Regression Kit

What it does

Evaluate recorded agent replies against explicit required, forbidden and exact text expectations without calling a model.

How to use it

Give the assistant the concrete task and required inputs. This fictional request illustrates the contract; use your selected files and parameters for real work.

{
  "cases": [
    {
      "id": "timezone",
      "actual": "09:00 UTC",
      "mustContain": [
        "UTC"
      ],
      "mustNotContain": [
        "API_KEY"
      ]
    },
    {
      "id": "exact",
      "actual": "done",
      "exact": "done"
    }
  ]
}

Output and verification

Creates a JSON report and, when applicable, CSV, Markdown or document artifacts in the current workspace. Inputs remain intact. Five offline tests cover the example, invalid input, actual CLI output, path boundaries and non-overwrite behavior. Live account, browser and visual results require separate verification.

Scope

These are deterministic assertions on supplied replies, not a semantic intelligence score. Generate realistic cases, keep held-out cases separate, and report model execution separately. Do not automatically rewrite agent instructions.