End-of-lesson Quiz
5 questions · Cross-Validation & Overfitting
1
of 5
Your intrusion detector scores 100% on training data and 74% on validation. The security team says 'the model works.' What's actually happening?
A 26-point train/test gap is a textbook overfitting signal. The model has memorised training patterns including noise. In production it will miss novel attacks because it never learned the underlying generalisable patterns. Fix: reduce model complexity until the gap shrinks to 2-3 points.
2
of 5
A depth-1 tree and a depth-50 tree both fail in production. Why?
This is the bias-variance tradeoff. Too simple (high bias) underfits both training and test data. Too complex (high variance) overfits training and fails on test. The sweet spot lies in the middle — complex enough to learn real patterns, simple enough to generalise.
3
of 5
Why is K-Fold Cross-Validation more reliable than a single train/test split?
A single split could be lucky (or unlucky). K-Fold CV trains K times on different splits, so you get K performance estimates. The mean is more reliable than any one split, and the standard deviation tells you how stable the model is across different data slices.
4
of 5
You get 5-fold CV scores of
[0.98, 0.71, 0.95, 0.96, 0.94]. What should you do?
One fold dropping 27 points below the others is a red flag, not noise. There's something distinctive about that subset — maybe a class imbalance, a unique attack family, or a data quality issue. A single train/test split would never have caught this.
5
of 5
A validation curve shows accuracy peaks at
max_depth=5 and drops off after that. Your manager wants max_depth=15 'because more is better.' What's your response?
The validation curve is your evidence. Past the peak, the training curve keeps climbing toward 100% but the validation curve drops — the model is memorising noise. In a security context, this means more false positives on benign traffic and missed novel attacks. Data beats opinions.