Manifest Score

Scores LLM output as a manifest so eval suites can gate on structural validity.

The first engine of the Evals hub. Every LLM-producing surface in the fleet answers with a manifest, and the first question about that output is not "how faithful is it" — it is: did it parse as JSON, is it a valid manifest, and how many of k attempts at the same prompt got through. That measurement is deterministic, and the measurer already exists: the substrate's own validators. This engine delegates to them and adds nothing.

What you can do

• Score any target's output — the hub calls evals.manifest.score per case with { output, kind }; the answer is { scores, pass, reason, detail } in the hub's engine contract.
• Gate on validity — manifest.valid is 0 or 1 per attempt, so a suite threshold like manifest.valid: 0.9 means "at least 90% of attempts must produce a valid manifest".
• Separate the two failure modes — json.parse failing is the prompt's formatting instruction breaking (responseFormat, "return ONLY raw JSON"); manifest.valid failing on parsed output is the prompt's schema explanation breaking. They have different fixes, so they are different metrics.
• Read regression direction — manifest.errorCount is a counter, never a thresholded score: a stable validity rate with a rising error count means the output is getting worse.
• Measure shape coverage — schema.requiredCoverage (0..1) counts the required fields present, measured against the validator's own required list (ivy.node, ivy.mcp, ivy.agent; workflow has no measured list, so the metric is absent there).

Manifest kinds

The kind comes from the suite's target.id — ivy.node.*, ivy.mcp.*, ivy.agent.* and workflow.* name their own kind, and the engine derives it from the target every score call carries. A target whose id names no kind declares it explicitly, and an explicit kind option always wins:

    engines:
      - id: manifest
        options:
          kind: ivy.node

These names are the engine's, not the hub's: the hub scores nothing and knows no product's artifact names.

kind	Validator
ivy.node	punica.flow.validator.validate — the substrate's v0.2 manifest validator
workflow	punica.flow.validateWorkflowDocument (strict)
ivy.mcp	Structural shape checks mirroring the generation pipeline's own
ivy.agent	punica.flow.validateWorkflowDocument (best-effort) over the same envelope ivy.agent.run builds

Why an agent is judged best-effort and a workflow strictly. An ivy.agent's graph is a workflow graph, so it is validated as one — but its nodes name ivy.node.* capabilities that only exist on a machine where those nodes were generated. Strict mode turns each of those into an error, which would score the workspace rather than the generator. Best-effort keeps the structural checks (a missing graph is still an error in both modes) and moves the unresolved references to warnings; detail.unknownCapabilities counts them, so they are visible without being scored.

Markdown-fence tolerance is exactly one strip: raw JSON is tried first, then a single ```json block is unwrapped and tried once. Two fences, or broken JSON inside the fence, is a parse failure — every further repair would corrupt the thing being measured. A stripped fence is flagged in detail so "the model does not return clean JSON" is never lost.

Requirements

• extension-evals (the hub) — this engine is discovered from the capability catalog and is invisible without it. Install the hub first.

Getting started

1. Install extension-evals, then this engine.
2. The hub's panel lists the engine automatically — discovery is by catalog, nothing is imported or configured.
3. Reference it from a suite in .punica/evals.yaml:

engines: [manifest]     # the engine reads the kind off the suite's target id
                        # (needs this engine 1.1.2 or newer — before it, `kind`
                        #  was a required input and the gateway refused the call)
thresholds:
  manifest.valid: 0.9

License

MIT