ivyxivyx
← All skills
IVYXSTUDIO · SKILL
E

Experiment comparison

experiment-comparison · v1.0.0

ivyx

Use when someone asks which of their experiment runs (deney, MLflow koşuları) is best, how the runs compare, or which parameter (learning rate, depth, estimators) moved the metric; reads the runs and measures the parameter shares instead of guessing.

Who is asked

Risk low: it loads without asking anyone. Each tool it then calls still meets your policy.

Tools it may call

While this skill is active, a call to any other tool is refused. These still meet every policy rule; the list narrows, it approves nothing.

experiments.listexperiments.getmlops.params.importancemlops.metric.history

What it tells the agent

The published SKILL.md, as written.

---
name: experiment-comparison
description: Use when someone asks which of their experiment runs (deney, MLflow koşuları) is best, how the runs compare, or which parameter (learning rate, depth, estimators) moved the metric; reads the runs and measures the parameter shares instead of guessing.
allowed-tools: experiments.list experiments.get mlops.params.importance mlops.metric.history
metadata:
  risk: low
---

Comparing experiment runs

The runs are in the project's tracking store. Read them, name the metric the person cares about, and measure which parameter moved it. You rank nothing by eye and you never say a parameter seems to matter: the share comes from mlops.params.importance or it is not said.

1. Read the runs

experiments.list first, with no arguments. Each run comes with its params and its last metric values. Keep the runs that have the metric the person named (or, when they named none, the one metric every run shares; say which you took). Drop runs with no value for it and say how many you dropped. experiments.get with a runId only when you need one run's full metric series or its artifacts.

2. Measure which parameter moved the metric

mlops.params.importance with:

  • runs: one entry per run kept, { "label": "<name or short id>", "params": {...}, "metrics": {...} } copied from the list;
  • objective: the metric's name;
  • direction: max for accuracy, f1, recall, precision; min for loss or error. If the metric's name does not say, ask the person before calling.

Read the answer:

  • parameters: each with its share of the metric's spread. Report them in that order with the share as a percentage, and the value at the best end.
  • excluded: parameters the measurement refused, with the reason it gives (one distinct value per run, the same value in every run, too few runs). Report every exclusion as the capability states it. A parameter with a distinct value in every run is not the most important one; it is unmeasurable, and a run id or seed is the usual case.
  • refusal, when the whole measurement was refused (too few runs): say so and compare the runs by the metric alone, saying that no share can be given.
  • belowResidual: a parameter with no measurable share; say no measurable share, not no effect.

3. Report

  • The runs: how many, the metric, its best and worst value and the run names at each.
  • The parameters in order of share, each with its share and its best-end value, and the excluded ones with their reason.
  • One sentence on what to try next, only from the measured shares (move the top parameter further toward its best end); nothing about parameters that were excluded or below the residual.
  • mlops.metric.history when the person asks how the pipeline's own gate metric moved across runs; otherwise leave it out.
  • Answer in the language the person wrote in.