MLOps Data Contracts

The pandera engine for the MLOps hub: draft a data contract from a dataset you already have, then stop the pipeline when the data stops matching it. The contract is a real pandera schema in your repository, code you edit, with git as its history.

What you can do

• Draft a data contract from a dataset you already have, instead of writing one from a blank file
• Keep that contract as code in your repository, edited by you and versioned in git
• Stop an MLOps pipeline the moment your data stops matching the contract, with the offending column named
• Check a dataset against its contract before anything downstream sees it
• See every contract gate in your pipeline in one panel, and validate them all at once

Requirements

• The MLOps extension, which holds the pipeline the gates belong to.
• Python Environments, which supplies the interpreter, and a Python 3.10 or newer environment for pandera and pandas.
• Data Profiler, to draft a contract from a dataset.
• The desktop app. Running Python is not something a browser can do.

Getting started

Open the Data Contracts panel with MLOps: Data Contracts Panel from the command palette. If the workspace has no contract yet, click Draft from a dataset… and pick a file. The data is profiled and a draft is written to the contracts/ folder. Add it to your pipeline as a gate, and the run stops there when the data breaks it.

Each gate in the panel has a right-click menu with Validate now, Open contract and Redraft from the dataset…. A gate whose contract file does not exist yet offers Draft this contract. If pandera is missing from your Python environment, the panel offers Install pandera.

Good to know

• A drafted contract is deliberately cautious. It checks columns, missing values, numeric types and ranges, and a list of allowed values only when it saw every value. Whatever it could not prove is written into the file as a note rather than guessed, so read the draft before you trust it.
• The whole file is validated at once. A dataset larger than memory needs a row limit, and the gate then says what it checked.
• Without the MLOps extension the panel says so rather than showing an empty list.