LLMOPS
A process you can read, measured stage by stage
One file declares the process. Every stage is measured by the extension that owns it, and the first gate that does not pass stops the run.
LLMOps is the path a model-backed product walks before and after release: the data it knows, the suites that grade it, the guardrails that hold it and the runs that show how it behaves. IVYX makes that path one file, .punica/llmops.yaml, and draws it as a live map you can run.
Click any block to see what it does. Run the process to watch the order become the run.
# .punica/llmops.yaml: the whole process, in one file.# Stages run top to bottom; the first gate that does not pass ends the run.model: the subject of the process
Names the governed core this process is about. Declaring it changes nothing about which model answers, which is the LLM panel’s one switch; it lets the designer say when the two disagree.
The file says what runs; the owners say what it measured
The process is one file
.punica/llmops.yaml names the stages, their order, their gates and what measures each one. A stage the file omits is gone, a stage you add is drawn, and the comments you write survive the designer’s edits.
Run walks it; the owners measure it
Run performs each stage’s capability calls and then its suite gates, in the file’s order. The verdict comes back from the owner’s own record, so a run cannot hand itself a pass.
The process, one step at a time
From the file to a measured model and back. Each card is a stage or the panel that owns it; LLMOps does no measuring of its own. Hover to hold a step, or click a card to jump to it.
.punica/llmops.yaml · the process
The file is the process
Process designer · cards in file order
The file is the process
- Every card is a stage declared in .punica/llmops.yaml, in the order the file gives them.
- A stage the file omits is gone from the process, and each card’s tag says what its stage does.
The process
What each stage is measured with
The process names the stages; these are the extensions that measure them, and what each one deliberately does not do.
Eval suites
Three engines: a generator's manifest validity, contains and matches over text classes, and recall@k for retrieval. Runs repeat k times, every suite has a baseline, a regression is a delta, and the cases no engine could judge wait in an adjudication queue. Groundedness is deliberately absent.
Guardrails
Three shipped packs, jailbreak, PII leak and prompt injection, each run as a suite with promptfoo as the detector. promptfoo can run fully local at a quality cost it states; its hosted default is surfaced, never shipped silently.
Knowledge
A knowledge base over local embeddings: chunking, a local index, and which chunk came back with which score, visible in the product.
Monitoring
The runs that called the governed model, drawn live as a span tree, and one OpenTelemetry trace per run for an external collector. Not every call on the machine.
Voice
Dictate to the assistant through whisper.cpp and hear the answer read sentence by sentence as it streams, through Kokoro, in English only. Hold to talk, or a conversation mode.
Outside by decision
An MCP gateway proxy, SSO and RBAC, a prompt-registry team platform, an LLM routing product, production monitoring, a metric library of its own, semantic threat detection, fine-tuning orchestration and a labelling tool.
Governed, not just automated
A run is calls through the same gateway a person’s clicks go through: policy decides, an approval is asked where a rule says so, and every call lands on the audit trail under the run’s own trace.
Capabilities, never commands
a run action names a capability. A command is refused in run: and told why, because it would run outside the gateway, with no policy decision and no audit record.
Measured, not self-graded
a suite gate is run through evals.suite.run, and its verdict is read back from the evals baseline afterwards. A gate whose reading cannot be found has not been satisfied.
A model proposes, a person saves
no LLMOps capability writes the process file. A model can draft it from a sentence and run it, and only a person’s Save in the designer writes it, because the process is what grades the model.
The rules that do not drift
No work of its own
LLMOps does no measuring itself. Every step of a run is a call to the extension that owns it, under that extension’s own permissions and approval rules.
A run follows the saved file
an edit in the designer is a draft until you press Save, and a run reads the saved file, not the draft.
Skipped is not passed
the first gate that does not pass stops the run, and the stages after it are shown as skipped, never as passed.
Two processes, never one file
llmops.yaml and mlops.yaml never reference each other, and neither extension reads the other’s file. A model’s process and a pipeline’s stay separate by decision.
“Cannot tell” is not empty
counts come from this machine’s audit trail within its retention window, and a source that cannot be read shows as “cannot tell”, never as zero.
A weak measure says so
a stage says on its face when guardrail scores are marker-only or the embedder is the hash placeholder, so a number is never better than what produced it.