Engineering note
Model evaluation: split by scenario
Define what must remain genuinely unseen, isolate groups, and keep preprocessing away from the test set.
In this document
Define what must be new#
A score should answer a precise question. Are we predicting a new observation of a known scenario, or an entirely unseen scenario? These two goals do not necessarily need the same data split.
For a power-network simulation dataset, I would first identify what connects rows: scenario, topology, shared parameters, or time. This is a proposed method, not a numerical result from Cascade-GNN.
Split at the right level#
When observations share a group, splitting by group keeps that group from appearing on both sides. GroupKFold supports cross-validation by group; GroupShuffleSplit supports random group splits. The grouping choice depends on the evaluation question.
scikit-learn · Cross-validation with grouped data- Document the group key and retain identifiers for each partition.
- Explicitly check that training and test group sets have an empty intersection.
- If the question concerns the future, also check time ordering: grouping alone is not sufficient.
Fit transformations on training data#
Scaling, imputation, and feature selection must be fitted on training data. A Pipeline connects these steps to the model and helps avoid leakage during cross-validation. It cannot correct bad grouping or a feature that already contains the answer.
scikit-learn · Common pitfalls and data leakageMake the result inspectable#
Before publishing a performance claim, prepare a small record: dataset version, split rule, a simple baseline, per-class metrics, and known limitations. Retain the configuration and code version for each run.
In my portfolio, Cascade-GNN remains a prototype. Validation figures are not displayed while the protocol and report are still being consolidated.