Engineering note

Model evaluation: split by scenario

Define what must remain genuinely unseen, isolate groups, and keep preprocessing away from the test set.

Type
Engineering note
Project
Cascade-GNN
Status
Documented method
Updated
2026-11-15
Read time
4 min
Language
EN
Markdown
In this document
  1. Define what must be new
  2. Split at the right level
  3. Fit transformations on training data
  4. Make the result inspectable

Define what must be new#

A score should answer a precise question. Are we predicting a new observation of a known scenario, or an entirely unseen scenario? These two goals do not necessarily need the same data split.

For a power-network simulation dataset, I would first identify what connects rows: scenario, topology, shared parameters, or time. This is a proposed method, not a numerical result from Cascade-GNN.

Split at the right level#

When observations share a group, splitting by group keeps that group from appearing on both sides. GroupKFold supports cross-validation by group; GroupShuffleSplit supports random group splits. The grouping choice depends on the evaluation question.

scikit-learn · Cross-validation with grouped data
  • Document the group key and retain identifiers for each partition.
  • Explicitly check that training and test group sets have an empty intersection.
  • If the question concerns the future, also check time ordering: grouping alone is not sufficient.

Fit transformations on training data#

Scaling, imputation, and feature selection must be fitted on training data. A Pipeline connects these steps to the model and helps avoid leakage during cross-validation. It cannot correct bad grouping or a feature that already contains the answer.

scikit-learn · Common pitfalls and data leakage

Make the result inspectable#

Before publishing a performance claim, prepare a small record: dataset version, split rule, a simple baseline, per-class metrics, and known limitations. Retain the configuration and code version for each run.

In my portfolio, Cascade-GNN remains a prototype. Validation figures are not displayed while the protocol and report are still being consolidated.

IN 60 SECONDS

Elmehdi Essamit

I build complete systems, from the ESP32 sensor to the power-flow solver, and validate every result against a reference.

Profile
Engineering student, year 1 · EHTP
Looking for
Introductory internship · 1 to 2 months
Period
Summer 2027
Mobility
Morocco or abroad
Languages
Arabic · French · English
Download the CVessamit.elmehdi@proton.meLinkedInFull recruiter page

Search the portfolio