ivyxivyx
← All skills
IVYXSTUDIO · SKILL
T

Train/test leakage check

train-test-leakage-check · v1.0.0

ivyx

Use when someone asks whether their train and test (or validation) data files overlap, leak (sızıntı), share rows, have duplicates, or whether the test set looks like the training set; measures it with queries and reports counts.

Who is asked

Risk low: it loads without asking anyone. Each tool it then calls still meets your policy.

Tools it may call

While this skill is active, a call to any other tool is refused. These still meet every policy rule; the list narrows, it approves nothing.

duckdb.inspectduckdb.registerSourceduckdb.queryprofiler.profile

What it tells the agent

The published SKILL.md, as written.

---
name: train-test-leakage-check
description: Use when someone asks whether their train and test (or validation) data files overlap, leak (sızıntı), share rows, have duplicates, or whether the test set looks like the training set; measures it with queries and reports counts.
allowed-tools: duckdb.inspect duckdb.registerSource duckdb.query profiler.profile
metadata:
  risk: low
---

Does the test set leak from the training set?

Every number in your answer is the result of a query in this run. Do not read the files with a file tool, do not judge from the first rows, and never write probably about a count you can measure.

1. Register both files

  1. duckdb.inspect on each file (the train file, then the test file) with its path as file: columns, row count, first rows. Note the id column (a column with a distinct value per row, such as PassengerId) and the label column.
  2. duckdb.registerSource twice, with path and an explicit viewName: "train" for the training file and "test" for the test file. Every query below uses exactly these two names; do not invent others and do not write register inside SQL.

2. Measure, in this order

One duckdb.query per line, with sql only (no file), copied as written with the id column's and the label column's names filled in:

  1. Exact row leakage, ignoring the id: select count(*) as leaked from (select * exclude (<id>) from test intersect select * exclude (<id>) from train)
  2. Id reuse: select count(*) as same_id from test where <id> in (select <id> from train) Report both; an id match with no row match means the id was reused, not the row.
  3. Duplicates inside each file: select count(*) - (select count(*) from (select distinct * from test)) as dup_test and the same with train.
  4. Label balance in each file, as counts: select <label>, count(*) as n from train group by 1 order by 1 and the same for test. Turn the counts into percentages yourself; a difference above 10 percentage points is worth a sentence.
  5. Examples of the leak, five rows: select * exclude (<id>) from test intersect select * exclude (<id>) from train limit 5
  6. Only when the person asks about a column's coverage: for a numeric column select min(<c>), max(<c>) from test against train; for a categorical one select distinct <c> from test except select distinct <c> from train.

If a query fails, read the error, fix that one query once and go on; do not rewrite the plan. profiler.profile with the test file's path only when the person asks about null shares or distributions.

3. Report

  • Leakage: the count of test rows also in train, as a count and a share of the test rows, with the examples.
  • Duplicates: per file.
  • Distribution: the target balance and any range or category the test set has that train does not.
  • What it means for a model score: say plainly that a score measured on leaked rows is too high, and by how many rows. Do not edit either file; say what the person could do (drop the leaked rows from test, or re-split from the union by id) and let them choose.
  • Answer in the language the person wrote in.