Step 4: Threshold Tuning

predict_proba() vs predict()

model.predict(X) returns hard labels (0 or 1) using the default 0.5 threshold.

model.predict_proba(X) returns probabilities — you can then apply any threshold yourself:

probs = model.predict_proba(X_test_scaled)[:, 1]   # P(phishing)

# Custom threshold
threshold = 0.3
y_pred_custom = (probs >= threshold).astype(int)

print(f"Default (0.5): {(probs >= 0.5).sum()} flagged")
print(f"Lower   (0.3): {(probs >= 0.3).sum()} flagged")

The Precision-Recall Tradeoff

This is a fundamental tradeoff — you cannot have both perfect precision and perfect recall unless your model is perfect.

Choosing the Right Threshold

The optimal threshold depends on your operational priorities:

from sklearn.metrics import precision_recall_curve

# precision_recall_curve sweeps every threshold from 0 to 1 and reports
# the precision and recall you'd get at each one — perfect for finding
# the trade-off point that matches your operational priorities.
precisions, recalls, thresholds = precision_recall_curve(y_test, probs)

# Find threshold that maximises F1
f1_scores = 2 * (precisions * recalls) / (precisions + recalls + 1e-8)
best_idx = f1_scores.argmax()
print(f"Best threshold: {thresholds[best_idx]:.2f}")
print(f"Precision: {precisions[best_idx]:.3f}")
print(f"Recall:    {recalls[best_idx]:.3f}")