Class Imbalance in Security
In real-world security monitoring, attacks are rare:
| Dataset | Typical attack rate |
|---|---|
| Intrusion detection (CICIDS) | 1–20% |
| Fraud detection | 0.1–1% |
| Phishing detection | 5–30% |
| Malware in enterprise telemetry | 0.01–5% |
The Accuracy Trap
If 95% of events are benign, a model that simply predicts "benign" for everything achieves 95% accuracy — without ever detecting a single attack.
from sklearn.dummy import DummyClassifier
# A "model" that always predicts the majority class
dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train)
accuracy = dummy.score(X_test, y_test)
print(f"Accuracy: {accuracy:.1%}") # 95%+ but catches ZERO attacksThis is the accuracy trap: high accuracy hiding zero detection.
How to Spot It
Always check recall for the minority class (the class you actually care about detecting):
from sklearn.metrics import classification_report
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred,
target_names=['benign', 'attack']))
# Look at the "attack" row:
# precision recall f1-score support
# attack 0.00 0.00 0.00 500 ← ZERO recall!If recall on the attack class is 0.00, the model is useless regardless of overall accuracy.
Loading...
Loading...
Loading...
Think Deeper
Try this:
Your malware detector has 99.9% accuracy on enterprise telemetry where 0.05% of events are malicious. How many real threats does it catch?
If the model always predicts 'benign', it gets 99.95% accuracy — even better than 99.9%. Your 99.9% model might be catching zero threats. The only way to know is to check recall: what fraction of actual malware was flagged?
Cybersecurity tie-in: When a vendor claims "99.9% accuracy" for their threat detection,
your first question should be: "What's the class distribution, and what's the recall on threats?"
A model with 99.9% accuracy and 0% recall is an expensive way to write
return "benign".