Step 2: Train a Random Forest

RandomForestClassifier Parameters

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(
    n_estimators=200,      # number of trees in the forest
    max_depth=None,        # let individual trees grow fully
    max_features='sqrt',   # features considered at each split
    oob_score=True,        # free validation using out-of-bag samples
    n_jobs=-1,             # use all CPU cores
    random_state=42
)
rf.fit(X_train, y_train)

Feature Subsampling

At each split, the random forest considers only a random subset of features (not all). This is the key difference from plain bagging.

Default: max_features='sqrt' means √(n_features) features per split. For 7 PE file features, that means each split considers ~3 features randomly.

This forces different trees to use different primary splits, producing diverse, uncorrelated trees. The ensemble benefit comes from this diversity — if all trees made the same mistakes, averaging would not help.

Out-of-Bag (OOB) Score

Each tree is trained on a bootstrap sample (~63% of data). The remaining ~37% — the out-of-bag samples — were never seen by that tree.

rf = RandomForestClassifier(n_estimators=200, oob_score=True,
                             random_state=42)
rf.fit(X_train, y_train)

print(f"OOB accuracy:  {rf.oob_score_:.3f}")
print(f"Test accuracy: {rf.score(X_test, y_test):.3f}")
# These should be close — both estimate generalisation

OOB accuracy is a free cross-validation estimate. If it closely matches test accuracy, your model is generalising well.

Tree vs Forest Comparison

The forest's test accuracy is significantly higher and its overfitting gap is smaller. The ensemble diversity pays off.