LLMOps

One live map of your whole LLMOps lifecycle: data, models, retrieval, agents, quality gates, release, monitoring, feedback and the improve loop, each stage read from the extension that owns it.

What you can do

• See every lifecycle stage on one flow, with a status dot per stage: green when healthy, amber when something deserves a look, red when a gate the workspace itself declared is failing.
• Read each stage's live numbers: knowledge bases and indexed chunks, AI cores and instruction classes, the last measured retrieval scores, governed agent runs, suite results against their own thresholds, export targets, call and error counts, and what waits for a person.
• Jump from any stage into the panel that owns the work: Knowledge, AI Cores, Instruction Classes, Monitor, Evals, Guardrails, Policy Export, Evidence.
• Watch the continuous-controls lane over the flow: how many capabilities the gate governs, what the workspace policy declares, and what the audit trail holds.
• Run the process. Press Run and each stage performs the capability actions it declares and then runs its suite gates, in the order the file lists them, stopping at the first gate that does not pass.
• Describe the process and have it written for you. Say what your process should be — what it is built from, what validates it, what stops a bad release — and the whole file is generated from that description, ready to read before anything is saved.

Every action the map offers is a jump into the owning panel, and every step a run performs is a call to the owning extension — so the work happens under that panel's own permissions and approvals either way. A run never decides a verdict for itself: it produces the evals baseline and then reads the gate from it, exactly as the map reads it without a run.

Honest numbers

• Counts come from this machine's audit trail, inside its retention window; they are not usage of the product elsewhere.
• A count read through a capped window shows as a lower bound (≥ n), never as a total.
• A source that cannot be read says "cannot tell"; it is never drawn as empty.
• When guardrail scores are marker-only or the embedder is the hash placeholder, the stage says so on its face.

Getting started

1. Open the LLMOps icon in the activity bar: the sidebar lists every stage with its status, and draws the flow top to bottom.
2. Click a stage row to open its owner full-size in the content drawer (Evals, Knowledge, the LLM viewer, Guardrails) or in its own panel (Monitor); right-click for "Show on map" and the secondary owners.
3. Read the flow in the LLMOPS sidebar: every stage with its state, its owners and a chevron that opens its numbers. Press the design button in the header to declare the process itself — or describe the one you want and let it be written.

Generating it from a description

Describe your process in the Design panel and press Generate: a model authors the whole file and it appears as a pending draft you read before saving. It needs an AI core installed and enabled; the panel says so when there is none rather than offering a button that cannot work.

What comes back is checked against your machine before you see it. A measurer that does not exist is removed rather than swapped for a similar one, so the stage draws no number instead of a made-up one. A capability nothing here provides is dropped from a gate, a run step or a click. An eval suite you have not written yet is kept, with a note telling you to create it in Evals — you asked for that gate, so it stays. Everything removed is listed above the draft, and the description you typed is written into the file as a comment, so the process explains where it came from.

Nothing is written until you press Save, and a generated process replaces the file rather than merging into it — so the panel asks you to save or discard pending changes first. Your description stays in the field if the model does not answer.

Declaring the process

The file is the process. .punica/llmops.yaml says which stages exist, the order they run in, their edges, what each is called, what measures it, where its click lands and which extensions own it. Nothing outside the file adds a stage.

Open LLMOps in a workspace that has no file and the product's own ten-stage process is already there as a pending draft — every stage, edge and owner, visible before it exists. Save writes it, and from then on that file is what LLMOps reads. Design it in the panel or edit it by hand; your changes are collected and written when you press Save in the header — nothing is written while you are still choosing — and your comments survive.

What the product still owns is the code that measures. A stage names a built-in measurer with metric:, and the reading stays in the extension, because a number's source has to be something the map can be held to. A stage that names no measurer draws no numbers at all, and never draws "N/A". A metric: is not a capability id — it names a measurer, and no measurer runs anything.

What the file does say about running is run: and its suite gates. A stage's run: lists capability calls it makes, in order; a suite gate is runnable as written. A suite gate still takes its verdict from the evals baseline and turns its stage red when it fails, and a capability gate is only observed — it shows when the audit trail last saw the call, carries no arguments, and is never called by a run.

Version 1 files — an overlay that hid stages, renamed them and pinned their gates — keep working exactly as they did. The panel offers to rewrite one as a full declaration, keeping your declarations and comments, and tells you what changes before it writes anything: under version 1 the stage order meant nothing, and under version 2 the order is the flow.

Declaring the governed model does not change which model answers: that stays the one control in the LLM Capabilities panel, and the map reports when the declared model and the answering one differ.

version: 2

model: # the subject: what this process is a claim about
  providerId: local:qwen3:8b
  title: Tutor core

inputs: # the material, as references to what their owners hold
  suites: [smoke, tutor-hint]
  knowledge: [handbook]
  packs: [jailbreak-basic]

stages: # list order IS the flow, top to bottom
  - id: controls
    title: Continuous Controls
    lane: aside # a lane beside the lifecycle, not a step in it
    metric: controls.gate

  - id: quality
    title: Quality & Red Team
    metric: evals.suites # a built-in measurer, not a capability
    primaryOpen: { label: Open Evals, capability: evals.open }
    gates:
      - { suite: smoke, title: Smoke suite } # verdict from .punica/evals.yaml and its baseline
      - { capability: guardrails.check } # shown with its last audited run, no verdict claimed
    extensions:
      - name: extension-acme-redteam
        title: Acme Red Team
        role: The in-house attack harness this team gates on.
        open: { capability: acme.redteam.open } # verified against the live catalog before it becomes a click

  - id: handover # a stage the product never shipped
    title: Handover to Support
    metric: evals.queue

edges: # only what the order cannot say: a label, and an edge against the stream
  - { from: quality, to: handover, label: signed off }
  - { from: handover, to: quality, loopback: true }

A missing file means the product's own process, and the panel offers to write it. A malformed one keeps the product's process, and the sidebar says so rather than passing it off as yours. This file is deliberately separate from .punica/mlops.yaml: that one is the MLOps hub's runnable pipeline, and the two never reference each other.

The product's own process, exactly as the panel seeds it, is docs/llmops.seed.yaml. A shorter annotated example that declares its own stages is docs/llmops.example.yaml.

Requirements

The map works with whatever is installed and says what is missing. Stages light up as their owners are present: Knowledge and Vectors for data and retrieval, Evals and Guardrails for quality, Policy Export and Evidence for release, Monitor for runs.

Generating a process from a description additionally needs an AI core installed and enabled.