MLOps Drift

The Evidently engine for the MLOps hub: compare the data your model sees now against the data it was trained on, and stop the pipeline when the distribution has moved.

Evidently is the real upstream tool. This extension manages it, it does not imitate it. The verdict on every column is Evidently's own.

What you can do

• Compare the data your model sees today against the data it was trained on
• Stop an MLOps pipeline when the distribution has moved, with the columns that moved named
• Catch a column that quietly disappeared or appeared, not just one whose numbers drifted
• Let a drift result trigger a retrain, rather than finding out weeks later

Requirements

The MLOps extension, the desktop app, and Evidently in the Python environment your workspace already uses.

Getting started

This is a gate you add once the pipeline runs. The walkthrough in the MLOps extension's README builds that pipeline from an empty folder; come back here to add drift to it.

What a drift gate is here

Two datasets and a bar. The reference is the data the model was trained against; the current dataset is whatever arrived since. Wire it into .punica/mlops.yaml and the hub runs it in order:

stages:
  - id: drift
    kind: gate
    capability: mlops.evidently.report
    input:
      reference: data/train.csv
      current: data/latest.csv
      driftShare: 0.5
    onFailure: stop
  - id: train
    kind: pipeline
    onFailure: stop

A failing gate marked stop ends the run, and train is recorded as skipped.

You can also point the panel at two files and measure drift with no pipeline at all. That pair lives in settings; a gate carries its own, in the file that is in git.

What the gate fails on

Three things, and they are not the same thing:

• Drift. Enough columns moved that Evidently's dataset test fails. driftShare is the bar, 0.5 by default, which is Evidently's own.
• A column that stopped being produced. This one is ours. Evidently measures drift on the columns both datasets have, so a reference column the current data no longer carries simply disappears from the report and the report passes. A schema change is the loudest drift signal there is, so the gate fails and names the column.
• A column that was not measured. Evidently's status vocabulary has five values, and ERROR or SKIPPED means the test did not answer. An unreadable verdict is not a green one.

A column that is new in the current data is reported and does not fail anything. There is no reference distribution to compare it against.

Why nothing here compares a score to a threshold

Evidently's per column number is a p value for K-S p_value and chi-square p_value, where drift is below the threshold, and a distance for methods like PSI and Wasserstein, where drift is above it. Reading the number without knowing the method inverts the answer for half the catalogue, so the report is run with tests included and Evidently states the verdict itself.

How it runs

The interpreter comes from Python Environments (python.env.resolve) and the drift driver is handed to it with -c. There is deliberately no bare python3, no PATH lookup, and no scratch file written into your workspace to run a gate.

The range this provider is written against is evidently>=0.7,<0.8, measured on Evidently 0.7.21 with pandas 3.0.5. The range is narrow because 0.7 moved the report surface to evidently.Report and made the per column verdict readable; an older release imports fine and answers a different shape.

Two measured numbers worth knowing before you wire a gate: importing Evidently costs about three seconds before any data is read, and its full HTML report is several megabytes because it carries its own rendering bundle. The HTML is therefore opt in, through the html input or the row menu.

Capabilities

Capability	Policy	What it does
mlops.evidently.report	medium / none	Compares two datasets and reports which columns drifted, by which test, and what moved
mlops.evidently.engine	low / none	The installed Evidently and pandas versions
mlops.evidently.reveal	low / none	Reveal the Drift panel

mlops.evidently.report answers pass: false with a reason when the data has moved. That is the verdict the MLOps hub records as a failed stage. An unusable environment (no interpreter, no Evidently, a dataset that is not there, two datasets sharing no columns) throws instead, with a different sentence for each, because each has a different fix.

Limits worth knowing

• Electron only: running processes is not something a browser host does.
• Whole file comparison. limit reads the first N rows of each file, and a sample can move the verdict, so it is for a quick look rather than for a gate.
• Drift on tabular columns. Embedding drift, text descriptors and the rest of Evidently's catalogue are not wired up here.
• The panel reads its gate list from the hub's mlops.pipeline.describe. With no MLOps hub installed it says so, and still lets you compare two files.

Verification

• npm run check:evidently replays measured driver envelopes (Evidently 0.7.21 / pandas 3.0.5, over a clean pair, a shifted pair, a pair missing a column, a pair with an extra one, a one row dataset and two disjoint frames) through the shipped parsers and panel logic. 97 checks.
• npm run probe:evidently -- /path/to/python builds throwaway datasets and drives the shipped engine against a live Evidently: a shifted dataset fails and names the columns that moved, a dropped column fails a report Evidently itself passes, an added column does not, and each situation that is not a verdict throws with its own kind. 25 checks. Re-run this after an Evidently or pandas upgrade, because a renamed status passes the harness and fails in the app.