Manifest Score

Scores LLM output as a manifest so eval suites can gate on structural validity.

The first engine of the Evals hub. Every LLM-producing surface in the fleet answers with a manifest, and the first question about that output is not "how faithful is it" — it is: did it parse as JSON, is it a valid manifest, and how many of k attempts at the same prompt got through. That measurement is deterministic, and the measurer already exists: the substrate's own validators. This engine delegates to them and adds nothing.

What you can do

• Score any target's output — the hub calls evals.manifest.score per case with { output, kind }; the answer is { scores, pass, reason, detail } in the hub's engine contract.
• Gate on validity — manifest.valid is 0 or 1 per attempt, so a suite threshold like manifest.valid: 0.9 means "at least 90% of attempts must produce a valid manifest".
• Separate the two failure modes — json.parse failing is the prompt's formatting instruction breaking (responseFormat, "return ONLY raw JSON"); manifest.valid failing on parsed output is the prompt's schema explanation breaking. They have different fixes, so they are different metrics.
• Read regression direction — manifest.errorCount is a counter, never a thresholded score: a stable validity rate with a rising error count means the output is getting worse.
• Measure shape coverage — schema.requiredCoverage (0..1) counts the required fields present, measured against the validator's own required list (ivy.node, ivy.mcp; workflow has no measured list, so the metric is absent there).

Manifest kinds

kind	Validator
ivy.node	punica.flow.validator.validate — the substrate's v0.2 manifest validator
workflow	punica.flow.validateWorkflowDocument (strict)
ivy.mcp	Structural shape checks mirroring the generation pipeline's own
ivy.agent	No validator yet — answers an honest error until the agent generator ships

Markdown-fence tolerance is exactly one strip: raw JSON is tried first, then a single ```json block is unwrapped and tried once. Two fences, or broken JSON inside the fence, is a parse failure — every further repair would corrupt the thing being measured. A stripped fence is flagged in detail so "the model does not return clean JSON" is never lost.

Requirements

• extension-evals (the hub) — this engine is discovered from the capability catalog and is invisible without it. Install the hub first.

Getting started

1. Install extension-evals, then this engine.
2. The hub's panel lists the engine automatically — discovery is by catalog, nothing is imported or configured.
3. Reference it from a suite in .punica/evals.yaml:

engines:
  - id: manifest
    options:
      kind: ivy.node   # which manifest the target's output is expected to be
thresholds:
  manifest.valid: 0.9

License

MIT