Evals

Run eval suites over LLM-producing targets and gate on the scores.

An instruction class's system prompt changes, its expertModelId changes, or its temperature moves — and nothing in the workspace says what broke. Assertion tests cannot answer this: LLM output is not verified with equals, it is measured as a distribution. This extension is the measuring instrument.

What you can do

• Author suites in .punica/evals.yaml — the file lives in git, is edited by hand, and is the single source of truth: each suite names a target (an instruction class or a capability), a jsonl dataset, how many samples to run per case, which engines score the output, and the thresholds a run must meet.
• Run the target through the gateway — k repeats per case with the LLM cache skipped, so variance is real variance and every call lands in the audit trail under one trace.
• Score with installed engines — engines are separate extensions discovered from the capability catalog by their evals.<engine>.score capability. The hub imports nothing and bundles no metric library.
• Read the aggregate — per metric: mean, p50/p95, pass rate against the threshold, and variance across the k repeats. One metric is the hub's own: target.okRate, the share of invocations that produced output at all — threshold it when a model that only sometimes answers must not pass the gate.
• Hold a pipeline gate — evals.suite.run answers { pass, reason }, so a stage in .punica/mlops.yaml with kind: gate and capability: evals.suite.run actually stops a pipeline, with the failing metric named in the reason.
• Track a baseline — the last run is written to .punica/evals/baseline.json and the next run shows the delta per metric; history is mirrored to MLflow when extension-experiments is installed.

Eval engines

An engine registers evals.<engine>.score and answers { scores: Record<string, number>, pass?: boolean, reason?: string, detail?: unknown }. An engine that answers only { pass, reason } is a valid engine — it shows a verdict with no score rows. The first engine is extension-evals-manifest, which scores generator output against the flow validator.

Requirements

• An eval engine, e.g. extension-evals-manifest.
• For instruction-class targets: a registered LLM profile the instruction class can bind to.

Getting started

1. Open the Evals panel and create the starter .punica/evals.yaml.
2. Install an engine (extension-evals-manifest).
3. Add a suite when its target is finished — each generator brings its own suite, dataset and thresholds as part of its completion:

version: 1
suites:
  - id: my-generator
    target:
      kind: instructionClass   # instructionClass | capability
      id: ivy.node.generator
    dataset: .punica/evals/my-generator.jsonl
    samples: 5
    engines: [manifest]
    thresholds:
      manifest.valid: 0.9
    onFailure: stop
baseline: .punica/evals/baseline.json

Dataset lines are jsonl: { "id": "...", "input": "<userPrompt>", "expect": { ... } } with expect optional. Run from the panel, the palette (Run Eval Suite) or a pipeline gate.