Evals

An eval runs an agent scenario and checks the result. Define cases in evals/. Deterministic assertions inspect replies, tool events, and run metrics. Prose assertions use an LLM judge or TypeSafe Jev.

Use deterministic checks for observable facts and judged prose for qualities that need interpretation.

Layout

evals/
├── refund-over-cap/
│   ├── eval.json
│   └── eval.md
├── ingest-report/
│   ├── eval.json
│   ├── eval.ts
│   └── report.pdf
├── triage.md
└── README.md

A directory case uses eval.md or eval.ts, with an optional eval.json manifest and fixtures. A flat Markdown file works for a case without supporting files. README.md is skipped.

Discovery is recursive. The ID is the path relative to evals/, without the file extension or trailing /eval: refund-over-cap or journeys/onboarding/first-run. Moving a case changes its ID, which identifies baselines and trends.

Top-level evals/traces/ is reserved for online trace evaluators and excluded from ordinary eval discovery.

eval.json: metadata only

{
  "tags": ["refusal", "write-path"],
  "objectives": { "turns": { "max": 4 }, "hitlRequests": { "min": 1 } },
  "runs": 5,
  "passRate": 0.8,
  "timeoutMs": 60000
}

The same fields can be written in YAML frontmatter. Declaring a field in both frontmatter and eval.json is a load error.

eval.md: the case

# Eval: Looks up the order before answering
 
Check that the answer comes from the order system.
 
## Setup
 
Context: The operator is viewing order ORD-1001.
 
## Query
 
"What's the status of ORD-1001?"
 
## Assertions
 
- tool_called_with: lookup_order {"order_id": "ORD-1001"}
- contains: delivered
- no_failed_actions
- Should state the order status clearly.
- Should NOT invent a delivery date the tool did not return.

This case assumes a lookup_order tool and a fixture order whose status is delivered.

Objectives belong in metadata. A ## Objectives section is a load error.

Grading with TypeSafe Jev

In Studio, open Run settings on the Evals page and select TypeSafe Jev under Grading. Choose Statement probability to pass each prose assertion at a probability threshold, or Scored rubric to grade each assertion against ordered levels. A score at or above the passing value passes. Deterministic checks and named scorers keep their own rules.

Enable the fallback judge to send uncertain answers to an LLM. Statement probabilities within the uncertainty bounds, including both endpoints, use the fallback. Rubrics use the fallback when confidence is below the configured value. Results retain Jev's probabilities and identify the deciding stage. A failed grading call is reported as Not graded.

Run settings apply to every run started from the page, whether one case or Run all. A grader selected there overrides the saved grading settings for that run. Saved with each case uses the saved grader, or the configured LLM judge. Code-authored evals execute their own assertions and do not use this prose grader.

Save grading settings in eval.json or YAML frontmatter to use them in Studio and amodal eval:

{
  "grader": {
    "kind": "typesafe",
    "model": "jev-latest",
    "passAbove": 0.7,
    "fallback": {
      "model": "anthropic/claude-haiku-4-5-20251001",
      "uncertain": { "low": 0.3, "high": 0.7 }
    }
  }
}

For scored grading, use rubric: { "levels": ["Not satisfied", "Partly satisfied", "Fully satisfied"], "passAt": 2 } instead of passAbove. Levels are numbered from zero. A rubric accepts 2–10 nonempty levels. Its fallback uses confidenceBelow, such as 0.6, instead of uncertain.

The trace export dialog includes these settings in the generated eval file. Hosted runs use the platform's TypeSafe configuration and attribute usage to the authenticated deployment. Standalone runtimes require TYPESAFE_API_KEY. Each Jev request has a 30-second ceiling and respects the eval deadline and cancellation. Responses over 100,000 characters require an LLM judge.

Repeating an eval

{ "runs": 5, "passRate": 0.8 }

At least four of the five declared runs must pass. The required count is ceil(runs × passRate). With the default passRate: 1, every run must pass.

Execution stops when the verdict is settled. For this example, a second failure makes a pass impossible, so the runner skips remaining attempts. Each attempt may contain several agent or judge calls; runs is not a model-call count.

Each repeat gets its own timeoutMs, bounded by the time left in the suite's --timeout. A deadline that stops the repeats does not reduce the declared number needed to pass. The result reports attempted, passed, required, and total runs, and includes the first failing attempt when the case fails.

The session an eval runs in

An eval without agent uses the chat route's default selection: agents/default/ if present, otherwise unscoped chat. Set session fields to test a specific named agent or data scope:

{
  "agent": "request-intake",
  "scopeId": "user-a",
  "maxSessionTokens": 4000,
  "scope": { "mission_id": "m-42" }
}

Each attempt creates a fresh session. Setting scopeId does not supply authenticated user claims.

agent, scopeId, and maxSessionTokens also work on code-authored EvalDefinition objects. Set scope in the manifest or per turn with t.send(message, {scope}).

The answer model is selected for the whole suite with amodal eval --model. A case cannot pin an answer model or force a skill. Use skill_loaded: to check whether the model chose the expected skill. judgeModel controls the grader independently.

Assertions

Deterministic keys

Known key: value bullets are checked against the run's reply and event facts:

An unknown key shaped like snake_case: is a load error. Other text is judged prose.

Negating a key

- not tool_called_with: issue_refund {"amount": 940}
- not tool_returned: check_conflicts {"ok": false}
- not skill_loaded: escalation

Negation passes when the corresponding positive check fails. Should NOT <key>: … is also supported. Use an argument-level negative when the tool may run for other records:

- tool_called_with: update_request {"request_id": "R-16"}
- not tool_called_with: update_request {"request_id": "R-11"}

Use no_failed_actions to catch broken tools even when the model produces a plausible reply. Pair tool_succeeded with an output check when the returned value matters:

- tool_returned: combine {"sum": 14}

A JSON subset distinguishes 14 from 1400; a substring check does not. Non-object handler returns are wrapped under result:

- tool_returned: summarize {"result": "Two requests remain."}

Judged prose

Keep judged checks specific enough to assess from the run. A deterministic failure includes the observed facts, such as the tools that actually ran.

Conversation

## Conversation
 
- user: "Refund ORD-1002 in full, $940."
- deny: issue_refund
- user: "What would it take to approve it?"

Confirmations default to denied. The first approve: or deny: after a user message controls that turn's confirmation requests. Its tool name is a label, not an enforced selector. Use separate user turns for actions needing different decisions, and assert the actual operation and arguments.

For a native OpenAPI write, denial happens before execution:

## Conversation
 
- user: "Record that Lee Park is adopting Biscuit."
- deny: shelter\_\_adoptResident
 
## Assertions
 
- hitl_requested: /residents/{residentId}/adopt
- not tool_called: shelter\_\_adoptResident
- not tool_succeeded: shelter\_\_adoptResident
- no_failed_actions

This assumes a shelter connection with confirmation required on the adoption operation. Direct the agent to use that operation. A confirmation question in ordinary prose does not exercise the runtime's approval gate.

Setup context is included only in the first user turn.

Calling a tool directly

Use a tool: step to test a handler without relying on the model to choose it:

## Conversation
 
- tool: build_from_packet {"packet_id": "PK-1"}
 
## Assertions
 
- subagent_called: package-reviewer
- tool_returned: build_from_packet {"status": "staged"}
- no_failed_actions

Arguments must be a JSON object and can be omitted for a tool with no parameters. Missing required arguments fail validation before execution. Unknown arguments are rejected only if the schema disallows them. Nested tool and subagent events are recorded for assertions.

A script can contain only tool steps, or use them to prepare a later conversation:

- tool: seed_request {"id": "R-1"}
- user: "What is in request R-1?"

Code evals use t.callTool:

const built = await t.callTool('build_from_packet', { packet_id: 'PK-1' });
built.calledSubagent('package-reviewer');
built.calledTool('build_from_packet', { output: { status: 'staged' } });

calledTool supports input and output matchers. An output matcher can be a JSON subset or a predicate over raw output text.

This lane is ungated

Direct invocation skips the outer permission checker, preToolUse / postToolUse hooks, and requiresConfirmation. It can execute a write without asking. The handler's own checks still run.

A direct call tests execution, not model selection or the outer approval gate. Test approval with user: and deny:. An approval directive after a tool: step is a load error.

Attachments

Declare fixture files in the manifest:

{
  "attachments": [{ "path": "report.pdf" }, { "name": "annex", "path": "fixtures/annex-b.pdf" }]
}

- user: "Summarize this report and its annex." [attach: report.pdf, annex]

Paths resolve against the eval directory. A name defaults to the filename. Only user turns can attach files.

MIME type is inferred for .pdf, .png, .jpg, .jpeg, .gif, .webp, .txt, .md, .csv, .docx, .pptx, .xlsx, .odt, .odp, .ods, and .rtf. Other extensions require mimeType. Office and OpenDocument files are converted to text at ingestion, and their embedded images are attached after the text under the same attachment name.

Fixtures use file paths, not inline base64. Undeclared attachment names, unreadable files, and unknown extensions without a MIME type fail loading. Assert a fact found only inside the fixture to verify that the document reached the model.

Objectives

Objectives are soft bounds used for ranking, not pass/fail:

{ "objectives": { "turns": { "max": 4 }, "hitlRequests": { "min": 1 } } }

Supported metrics are turns, userTurns, toolCalls, redundantToolCalls, subagentCalls, hitlRequests, failedActions, replyChars, latencyMs, and costMicros. Each can have a min or max. An unknown metric is a load error.

Use an assertion when a bound must fail the case. For example, max_turns: 4 is a gate; an objective on turns is a ranking signal.

Code-authored evals

Use eval.ts for branching conversations, custom predicates, or assertions scoped to one turn:

export const readsBothDocuments = {
  description: 'Read facts from documents as they arrive.',
  async test(t) {
    const first = await t.send('Read this report.', { attachments: ['report.pdf'] });
    t.require('report reached the model', () => /Project Cedar/.test(first.reply ?? ''));
    first.satisfies('does not invent the annex amount', () => !/1572/.test(first.reply ?? ''));
 
    const second = await t.send('Read its annex.', { attachments: ['annex.pdf'] });
    second.satisfies('reads the annex amount', () => /1572/.test(second.reply ?? ''));
    t.noFailedActions();
  },
};

Declare both attachments in the sibling manifest. This example assumes the report contains Project Cedar and only the annex contains 1572.

One file can export several cases. A named export contributes to the case ID, such as ingest/readsBothDocuments.

The context t and returned turn objects expose calledTool, notCalledTool, calledSubagent, loadedSkill, toolOrder, messageIncludes, maxTurns, maxToolCalls, noFailedActions, succeeded, and satisfies(label, predicate). Methods on a returned turn inspect only that turn.

Ordinary assertions collect failures and continue. t.require stops the script when subsequent checks would have no valid setup.

Scorers

A scorer is a reusable check in scorers/<name>/scorer.ts. Reference it with scorer: <name>.

const known = new Set(['1', '2.1', '2.2']);
 
export default {
  description: 'Every cited section exists in the reference index.',
  requires: ['finalMessage'],
  preprocess: ({ facts }) => {
    const cited = [...(facts.finalMessage ?? '').matchAll(/§\s*([\d.]+)/g)].map((m) => m[1]);
    return { cited, missing: cited.filter((id) => !known.has(id)) };
  },
  score: ({ pre }) => (pre.cited.length === 0 ? 1 : 1 - pre.missing.length / pre.cited.length),
  reason: ({ pre }) =>
    pre.missing.length ? `Unknown sections: ${pre.missing.join(', ')}` : 'All cited sections exist',
};

Replace the example section set with your reference index. This scorer checks citation validity; it does not require a citation. Add that check separately when needed.

requires declares the facts a scorer reads. Unsupported fact requirements fail registration.

Running

amodal eval
amodal eval refund
amodal eval journeys/
amodal eval '/^(smoke|journeys)\//'
amodal eval --tag safety
amodal eval --tag safety --tag p0
amodal eval journeys/ --tag safety
amodal eval --json
amodal eval --record-dir out/
amodal eval --url https://your-runtime.example.com

The positional filter matches an ID, directory prefix, substring, or slash-delimited regex. Multiple tags are OR'd; the tag filter and positional filter are AND'd. Malformed regexes fail.

Without --url, the CLI starts the local runtime used by amodal dev on an ephemeral port. With --url, it runs evals provided by that runtime. Agent and judge calls use its configured providers.

The command exits nonzero if any case fails or no cases run. When the suite exceeds --timeout, the report retains completed results and identifies the incomplete run.

Running in Amodal

Open the agent's Evals page, expand a case, and select Run, or select Run all. Filter the list by status or search it first, and Run all becomes Run N shown and runs only those cases. Select several agent models under Run settings to compare them on the same cases. Runs continue when you navigate between pages within the same agent workspace. Keep the browser tab open and remain signed in to that workspace. Refreshing, closing the tab, signing out, or leaving the agent workspace aborts unfinished runs.

Stop on a case stops it, including its remaining models. If the case belongs to a batch, it also stops the batch. Stop run in the progress banner stops the active case and skips the queued cases. Completed results are kept.

Completed results saved to the agent's history remain available after a reload. Expand a case and open Run history to revisit them. Expand a result to inspect its assertions and response, then open Tool activity and an individual call to inspect its input and output.

A history-save warning is separate from the grade. The result remains visible in the current tab, but saving to history is unconfirmed.

Deployment triggers

Deployment evals are opt-in. Adding evals/ to an agent does not enable automatic runs. Run suites explicitly from the Evals page or with amodal eval.

For GitHub-connected agents, set ci.evals.runOn in the agent's amodal.json to enable evals after a successful deployment. For example, add this ci block to run evals on production deployments:

{
  "ci": {
    "evals": {
      "runOn": "production"
    }
  }
}

Cloud reads the configuration at the deployed commit. Missing or unreadable configuration, invalid JSON, and an unrecognized runOn value skip evals. Setting only runs, passRate, or judgeModel does not enable automatic execution. Skipped evals do not post an amodal/evals GitHub check.

This setting controls deployment triggers only. Manual eval runs remain available. Amodal-hosted agents run evals on demand. Online trace evaluators are configured separately under Traces → Evaluators.

Hosted evals use temporary sessions that are not saved to conversation history. Saved eval result history is separate from these conversation sessions. Agent and judge calls still produce traces and incur model usage charges.