What Dense "sees" in an image
A 28x28 greyscale image has 784 pixels. A Dense layer receives these as a flat vector: [0.0, 0.0, 0.12, 0.85, 0.95, ...]. It has no concept that pixel 3 is adjacent to pixel 4 -- it only knows statistical correlations between positions.
| Property | Dense layer | Conv2D layer |
|---|---|---|
| Input format | Flat 1D vector (784,) | 2D grid (28, 28, 1) |
| Spatial awareness | None -- every pixel equally distant | Full -- uses local 3x3 neighbourhoods |
| Translation invariance | None -- must relearn for every position | Built-in -- same filter slides everywhere |
| Parameters for 128 outputs | 784 x 128 + 128 = 100,480 | Depends on filters, but far fewer |
The shuffled-pixels experiment
The definitive proof that Dense ignores spatial structure: randomly shuffle every pixel in every image, then retrain. Dense accuracy barely changes because it never used spatial relationships in the first place.
import numpy as np
from tensorflow import keras
# Load MNIST
(X_train, y_train), (X_test, y_test) = keras.datasets.mnist.load_data()
X_train = X_train.reshape(-1, 784).astype('float32') / 255.0
X_test = X_test.reshape(-1, 784).astype('float32') / 255.0
# Create a fixed random permutation
perm = np.random.permutation(784)
# Shuffle every image the same way
X_train_shuffled = X_train[:, perm]
X_test_shuffled = X_test[:, perm]
# Dense model gets ~97.8% on normal AND shuffled images
# CNN would collapse from ~99.2% to ~random on shuffled imagesThe parameter cost of spatial blindness
Because Dense treats every pixel independently, it needs separate weights for every pixel-to-neuron connection:
| Layer type | Input | Output units | Parameters |
|---|---|---|---|
| Dense | 784 pixels (flattened) | 128 | 784 x 128 + 128 = 100,480 |
| Conv2D (32 filters, 3x3) | 28x28x1 | 32 feature maps | 3 x 3 x 1 x 32 + 32 = 320 |
The Conv2D layer achieves comparable feature extraction with 314x fewer parameters because it reuses the same 3x3 filter at every position (weight sharing).
When does Dense still work?
Dense layers still get ~97-98% on MNIST because the dataset is simple: centred, normalised digits with little variation. The statistical correlations alone are sufficient. But on harder image tasks (CIFAR-10, medical imaging, malware visualisation), Dense performance degrades sharply while CNNs maintain accuracy.
Think Deeper
You shuffle every pixel in an MNIST image randomly. Dense accuracy barely changes. Why does this prove Dense ignores spatial structure?