# Model evaluation: split by scenario

Define what must remain genuinely unseen, isolate groups, and keep preprocessing away from the test set.

https://elmehdi-essamit.me/en/notes/evaluer-sans-fuite/

## Define what must be new

A score should answer a precise question. Are we predicting a new observation of a known scenario, or an entirely unseen scenario? These two goals do not necessarily need the same data split.

For a power-network simulation dataset, I would first identify what connects rows: scenario, topology, shared parameters, or time. This is a proposed method, not a numerical result from Cascade-GNN.

## Split at the right level

When observations share a group, splitting by group keeps that group from appearing on both sides. GroupKFold supports cross-validation by group; GroupShuffleSplit supports random group splits. The grouping choice depends on the evaluation question.

[scikit-learn · Cross-validation with grouped data](https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-iterators-for-grouped-data)

- Document the group key and retain identifiers for each partition.
- Explicitly check that training and test group sets have an empty intersection.
- If the question concerns the future, also check time ordering: grouping alone is not sufficient.

## Fit transformations on training data

Scaling, imputation, and feature selection must be fitted on training data. A Pipeline connects these steps to the model and helps avoid leakage during cross-validation. It cannot correct bad grouping or a feature that already contains the answer.

[scikit-learn · Common pitfalls and data leakage](https://scikit-learn.org/stable/common_pitfalls.html#data-leakage)

## Make the result inspectable

Before publishing a performance claim, prepare a small record: dataset version, split rule, a simple baseline, per-class metrics, and known limitations. Retain the configuration and code version for each run.

In my portfolio, Cascade-GNN remains a prototype. Validation figures are not displayed while the protocol and report are still being consolidated.

