Writing an eval suite

A suite is a jsonl file of cases and a block in .punica/evals.yaml that names a target, the dataset, an engine and thresholds. A suite with no engine is silently dropped by the hub, so the engine is the first thing you check, not the last.

1. Which engines exist

evals.engines first, with no arguments. Read engines: for answers that are text (a hint, a subject line, a JSON verdict) the engine is text; for a node or agent manifest it is manifest; for retrieval it is rag. If catalogReadable is false or the engine you need is absent, say so and stop: a suite cannot be bound to an engine that is not installed.

2. Read what exists

fs.readFile on .punica/evals.yaml. If it exists, you add a suite to it and keep every other suite as it is; if it does not, you create it.

Then read the target itself. A workspace instruction class <id> lives at .punica/instruction-classes/<id>.ic.yaml; a prompt lives where the person said. Read its system prompt: it says what the target is given (a policy text, a document, a record) and what it must never do. When it names material the target answers from, find that material in the project: fs.list with path: "." shows the top of the project, and the policy, document or record files are in the folder whose name says so (docs, policies, content); fs.list that folder, then read each file in it with fs.readFile. Never guess a path and never write the material yourself: the cases come from those files, not from what an assistant of this kind usually says, and if the project holds no such file, say so and ask for it before writing a case.

3. Write the cases

Read the exact shape from the file suite-template.md of this skill (ask for it with { "file": "suite-template.md" }), then write .punica/evals/<suite-id>.jsonl with fs.writeFile: one JSON object per line, four to eight cases, each with an id, an input (the whole prompt the target gets, as text: when the target is given a policy or a document and a question, the input holds that text and the question, in the order the system prompt describes) and an expect object holding only checks the engine can run (contains, notContains, matches, maxLines, maxChars, requiredKeys) and a note saying what a person would look for. An expectation is a fact the material states (a number of days, a price, a city, a word the target must not use), so a wrong answer fails and a vague one does not pass; at least one case asks something the material does not answer and expects the target to say so. Cases come from the person's examples when they gave any; otherwise write them from the material and say they are theirs to edit.

4. Write the suite

fs.writeFile .punica/evals.yaml with the block from the template: id, target (kind and id), dataset, samples: 1, engines with the engine id from step 1, thresholds with expect.satisfied at the bar the person wants (1.0 unless they said otherwise), onFailure: stop. Keep version: 1 and baseline at the top level.

5. Run it once

evals.suite.run with suite set to the new id. Read pass, reason and report: how many cases ran, the expect.satisfied mean, which cases failed and on which check. A run that invokes a local model takes minutes; wait for it.

6. Report

• The files written, with the case count and the engine.
• The run: pass or fail, the mean against the bar, each failing case by id with its first failing check.
• What to do with a failing case: fix the target, or change the case when the expectation was wrong; never lower the bar to make the run green.
• Answer in the language the person wrote in.