Evals

Run eval suites over LLM-producing targets and gate on the scores.

An instruction class's system prompt changes, its expertModelId changes, or its temperature moves — and nothing in the workspace says what broke. Assertion tests cannot answer this: LLM output is not verified with equals, it is measured as a distribution. This extension is the measuring instrument.

What you can do

• Author suites in .punica/evals.yaml — the file lives in git, is edited by hand, and is the single source of truth: each suite names a target (an instruction class or a capability), a jsonl dataset, how many samples to run per case, which engines score the output, and the thresholds a run must meet.
• Run the target through the gateway — k repeats per case with the LLM cache skipped, so variance is real variance and every call lands in the audit trail under one trace.
• Score with installed engines — engines are separate extensions discovered from the capability catalog by their evals.<engine>.score capability. The hub imports nothing and bundles no metric library.
• Read the aggregate — per metric: mean, p50/p95, pass rate against the threshold, and variance across the k repeats. One metric is the hub's own: target.okRate, the share of invocations that produced output at all — threshold it when a model that only sometimes answers must not pass the gate.
• Hold a pipeline gate — evals.suite.run answers { pass, reason }, so a stage in .punica/mlops.yaml with kind: gate and capability: evals.suite.run actually stops a pipeline, with the failing metric named in the reason.
• See what is waiting for a person — an answer no engine could judge (the case declares no expectation, or every engine declined) sits in the panel's "Awaiting a person" list beside the capability calls that are blocked on an approval, because both are the same question. A decision is right-clicked onto the row and written back into the dataset line, so it travels with the golden set and shows up in a diff. It never invents an expectation: what an expectation looks like is the engine's vocabulary, and the hub speaks none.
• Track a baseline — the last run is written to .punica/evals/baseline.json and the next run shows the delta per metric; history is mirrored to MLflow when extension-experiments is installed.

Eval engines

An engine registers evals.<engine>.score and answers { scores: Record<string, number>, pass?: boolean, reason?: string, detail?: unknown }. An engine that answers only { pass, reason } is a valid engine — it shows a verdict with no score rows. The first engine is extension-evals-manifest, which scores generator output against the flow validator.

Requirements

• An eval engine, e.g. extension-evals-manifest.
• For instruction-class targets: a registered LLM profile the instruction class can bind to.

Getting started

1. Open the Evals panel and create the starter .punica/evals.yaml.
2. Install an engine (extension-evals-manifest).
3. Add a suite for anything that produces text from a prompt — an instruction class or a capability — with its own dataset and thresholds:

version: 1
suites:
  - id: my-target
    target:
      kind: instructionClass   # instructionClass | capability
      id: my.instruction.class
    dataset: .punica/evals/my-target.jsonl
    samples: 5
    engines: [an-engine]
    thresholds:
      a.metric: 0.9
    onFailure: stop
baseline: .punica/evals/baseline.json

Dataset lines are jsonl: { "id": "...", "input": "<userPrompt>", "expect": { ... } } with expect optional. Run from the panel, the palette (Run Eval Suite) or a pipeline gate.

What the hub does not know

Every engine receives the same five fields — output, expect, caseId, suite and target — and nothing else the hub invented. The hub scores nothing and names no metric, so what a metric means, what an artifact is, and what an engine needs to know are the engine's business:

    engines:
      - id: an-engine
        options:
          anything: the engine documents

Options travel to the engine verbatim, after the five fields, so an option always wins over whatever the engine would have worked out for itself. The one number the hub does compute is target.okRate, the share of invocations that answered at all — that is a property of the run, not of the artifact.