Step 4: Architecture Search

Architecture as a hyperparameter

When people say "hyperparameter tuning," they often mean learning rate and batch size. But the network architecture itself is also a hyperparameter -- and often has a bigger impact on performance.

The grid search strategy

A grid search tries every combination in a defined search space. Systematic and thorough, but the number of combinations grows quickly.

import itertools
import pandas as pd
from tensorflow import keras

# Define the search space
units_options = [32, 64, 128]
depth_options = [1, 2, 3]

# Total combinations: 3 x 3 = 9 models
results = []

for units, depth in itertools.product(units_options, depth_options):
    # Build model with this configuration
    layers = [keras.layers.Dense(units, activation='relu', input_shape=(10,))]
    for _ in range(depth - 1):
        layers.append(keras.layers.Dense(units, activation='relu'))
    layers.append(keras.layers.Dense(1, activation='sigmoid'))

    model = keras.Sequential(layers)
    model.compile(optimizer='adam', loss='binary_crossentropy',
                  metrics=['accuracy'])

    history = model.fit(X_train, y_train,
                        validation_data=(X_val, y_val),
                        epochs=50, batch_size=32, verbose=0)

    best_val_loss = min(history.history['val_loss'])
    n_params = model.count_params()

    results.append({
        'units': units, 'depth': depth,
        'params': n_params, 'best_val_loss': round(best_val_loss, 4),
    })

df = pd.DataFrame(results).sort_values('best_val_loss')
print(df.to_string(index=False))

Grid search vs random search

Reading the results table

When comparing architecture search results, look for these patterns: