Text Score

Scores an LLM answer against the assertions its eval case declares.

The engine of the Evals hub for answers that are not artifacts — a tutor hint, a commit subject, a JSON verdict. The question there is not "is this a valid manifest" but "did this answer do what the case asked of it", and only the case can say. So the case says it: a hint that must contain no code fence, a subject that must fit 72 characters, a diagnosis whose rationaleCorrect must be false. The engine checks the declaration and knows nothing else.

What you can do

• Score any answer against its case — the hub calls evals.text.score per case with { output, expect, … }; the answer is { scores, pass, reason, detail } in the hub's engine contract.
• Gate on the assertions — expect.satisfied is the fraction of a case's checks that held, so expect.satisfied: 0.9 in a suite means nine of ten declared checks must hold across the run.
• Gate on shape — json.parse is 0 or 1 when the suite sets json: true, and schema.requiredCoverage counts the suite's required dot paths present in the parsed answer.
• Read regression direction — text.chars is a counter, never a thresholded score. A stable satisfaction rate with a rising character count is an answer getting wordier, which is a signal and not a verdict.
• Send what cannot be asserted to a person — an expect that is prose declares nothing, so the case is scored by nothing and reaches the adjudication queue for a human verdict. Inside an object expect, note does the same thing beside the checks.

The assertions

Suite-wide options live in the engine binding; per-case ones on the dataset line's expect.

Key	Holds when
contains	every string appears in the answer
notContains	none of them appears
matches / notMatches	the regular expression matches / does not
minChars / maxChars	the answer's length is within the bound
maxLines	the answer has no more lines than that
requiredKeys	every dot path exists in the parsed answer
note	never checked — it is the sentence the adjudicator reads

Three rules hold throughout. Absent is not zero: a metric nothing declared is missing from the answer rather than a 0 that drags a mean down. A check that could not run is not a pass: required keys against text that is not JSON fail with the reason naming why, and so does a broken regular expression. The reason is a gate message — one sentence about the first failure, never a raw error object.

Requirements

• extension-evals (the hub) — this engine is discovered from the capability catalog and is invisible without it. Install the hub first.

Getting started

1. Install extension-evals, then this engine.
2. The hub's panel lists the engine automatically — discovery is by catalog, nothing is imported or configured.
3. Reference it from a suite in .punica/evals.yaml:

engines:
  - id: text
    options:
      json: true
thresholds:
  expect.satisfied: 0.9
  json.parse: 0.9

1. Declare what each case expects, on its own dataset line:

{"id":"one-file-bugfix","input":"…","expect":{"maxLines":1,"maxChars":72,"note":"one conventional subject, no body"}}

License

MIT