Evaluation, Overfitting,
and Underfitting

validation - generalization - regularization - early stopping

Learning Outcomes

1. Explain train/validation/test split and why each matters

2. Diagnose overfitting and underfitting from training curves

3. Apply regularization: L1/L2, Dropout, Early Stopping

4. Choose the right strategy to improve generalization

Week 5 -- 1145208 Deep Learning

Today's Roadmap

Part 1 - Evaluation

How to measure performance

train/validation/test, cross-validation, learning curves

Part 2 - Overfitting

When the model memorizes

symptoms, causes, overfitting vs underfitting

Part 3 - Regularization

Fighting overfitting

L1/L2, Dropout, Early Stopping, data augmentation

Part 4 - Wrap up

Summary + Homework

strategy cheat sheet, exercises

Part 1: Why three splits?

Train

60%

Model learns from this data

weights updated here

Validation

20%

Check during training

tune hyperparameters

Test

20%

Final evaluation only

never touch during training

Test set = final exam. Peeking at it = cheating.

Validation in Keras

# Option 1: validation_split (simple)
model.fit(X_train, y_train,
          epochs=50,
          validation_split=0.2)
# Last 20% of X_train used as validation

# Option 2: pass validation data (better)
model.fit(X_train, y_train,
          epochs=50,
          validation_data=(X_val, y_val))

# Option 3: K-Fold (most robust)
from sklearn.model_selection import KFold
kfold = KFold(n_splits=5)
for train_idx, val_idx in kfold.split(X_train):
    model.fit(X_train[train_idx],
              y_train[train_idx],
              validation_data=(
                  X_train[val_idx],
                  y_train[val_idx]))

- validation_split=0.2 = last 20% is validation

- Warning: data must not be shuffled before split (Keras uses last 20%)

- Better: split manually, then use validation_data

- K-Fold: most robust but slower (trains K models)

[Tip] For time series: never shuffle! Use past for train, future for val.

Reading Learning Curves

Good fit

loss ----

val_loss ----

Both decrease, stay close

Overfitting

loss ------

val_loss / (goes up!)

Train good, validation bad

Underfitting

loss ---- (stays high)

val_loss ---- (stays high)

Both stay high, no learning

Gap between train and val = overfitting. Both high = underfitting.

Plotting learning curves

import matplotlib.pyplot as plt

history = model.fit(X_train, y_train,
                    epochs=50,
                    validation_split=0.2)

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Loss
axes[0].plot(history.history['loss'],
             label='Train')
axes[0].plot(history.history['val_loss'],
             label='Val')
axes[0].set_title('Loss')
axes[0].legend()

# Accuracy
axes[1].plot(history.history['accuracy'],
             label='Train')
axes[1].plot(history.history['val_accuracy'],
             label='Val')
axes[1].set_title('Accuracy')
axes[1].legend()

plt.tight_layout()
plt.show()

- Plot both loss and accuracy side by side

- What to look for:

- Are both curves going down?

- Is the gap between train/val growing?

- Is val_loss starting to increase?

- Early stopping point: when val_loss stops decreasing

[Tip] Save best model only: keras.callbacks.ModelCheckpoint('best.keras', save_best_only=True)

Part 2: Overfitting -- model memorizes

Causes

- Model too complex (too many parameters)
- Training too long (too many epochs)
- Too little training data
- Noisy labels in training data

Symptoms

- Train accuracy high, val accuracy low
- Train loss low, val loss high
- Gap grows with more epochs
- Model works on training, fails on new data

Analogy: student who memorizes answers but can't solve new problems

Underfitting -- model too simple

Causes

- Model too simple (few parameters)
- Not enough training epochs
- Regularization too strong
- Features not informative enough

Solutions

- Add more layers or neurons
- Train longer (more epochs)
- Reduce regularization
- Add better features

Both train and val performance are poor. The model hasn't learned the pattern.

Model Capacity and the sweet spot

Low capacity

1 Dense(8)

164 parameters

Underfitting

Right capacity

2 Dense(64)

24,960 parameters

Good fit

High capacity

5 Dense(256)

500k+ parameters

Overfitting

Start small, increase until val performance stops improving

Part 3: L1/L2 Regularization

from keras import regularizers

model = keras.Sequential([
    layers.Dense(64, activation='relu',
                 input_shape=(784,),
                 kernel_regularizer=
                   regularizers.l2(0.01)),
    layers.Dense(64, activation='relu',
                 kernel_regularizer=
                   regularizers.l2(0.01)),
    layers.Dense(10, activation='softmax')
])

- Regularization adds penalty to loss function

- L2: \( +\lambda \sum w^2 \) = push weights toward zero

- L1: \( +\lambda \sum |w| \) = can make weights exactly zero (sparse)

- 0.01 = lambda (regularization strength)

- Larger lambda = stronger regularization = simpler model

Dropout -- random deactivation

model = keras.Sequential([
    layers.Dense(128, activation='relu',
                 input_shape=(784,)),
    layers.Dropout(0.5),  # 50% off
    layers.Dense(64, activation='relu'),
    layers.Dropout(0.3),  # 30% off
    layers.Dense(10, activation='softmax')
])

- During training: randomly set X% of neurons to zero

- Forces network to not rely on any single neuron

- Dropout(0.5): 50% of neurons randomly off each batch

- Only active during training, not during inference

- Like training an ensemble of many smaller networks

[Tip] Typical: 0.2-0.5 for Dense layers. 0.1-0.3 for input.

Early Stopping -- stop when val stops improving

early_stop = keras.callbacks.EarlyStopping(
    monitor='val_loss',    # what to watch
    patience=5,            # wait 5 epochs
    restore_best_weights=True
)

model.fit(X_train, y_train,
          epochs=100,       # train "forever"
          validation_split=0.2,
          callbacks=[early_stop])

- monitor='val_loss' = watch validation loss

- patience=5 = stop if no improvement for 5 epochs

- restore_best_weights=True = revert to best epoch

- Train many epochs, but only keep the best version

[Tip] Always use EarlyStopping + ModelCheckpoint together for best results

ModelCheckpoint -- save best model

checkpoint = keras.callbacks.ModelCheckpoint(
    'best_model.keras',
    monitor='val_loss',
    save_best_only=True,  # overwrite only better
    save_weights_only=False
)

model.fit(X_train, y_train,
          epochs=100,
          validation_split=0.2,
          callbacks=[checkpoint, early_stop])

# Load best model later
model = keras.models.load_model(
    'best_model.keras')

- Saves model to file after each epoch

- save_best_only=True = only save if val improved

- Without this: final epoch saved (may be overfit!)

- .keras format = full model (architecture + weights + optimizer)

Data Augmentation -- more training data

# Image augmentation with Keras
data_augmentation = keras.Sequential([
    layers.RandomFlip("horizontal"),
    layers.RandomRotation(0.1),
    layers.RandomZoom(0.2),
    layers.RandomContrast(0.2),
])

# Use in model
model = keras.Sequential([
    data_augmentation,      # <-- first layer
    layers.Flatten(input_shape=(28,28)),
    layers.Dense(128, activation='relu'),
    layers.Dense(10, activation='softmax')
])

# Or use on-the-fly during training
datagen = keras.preprocessing.image\
    .ImageDataGenerator(
        rotation_range=10,
        width_shift_range=0.1,
        horizontal_flip=True)

- Creates new training samples from existing ones

- For images: flip, rotate, zoom, shift, contrast

- 1000 original images -> thousands of variations

- Only apply to training, not validation/test

- Greatly reduces overfitting when data is limited

[Tip] Augmentation = free data. Use it whenever you have less than ~10k images per class

Putting it all together

from keras import regularizers

model = keras.Sequential([
    layers.Dense(128, activation='relu',
        input_shape=(784,),
        kernel_regularizer=regularizers.l2(1e-4)),
    layers.Dropout(0.3),
    layers.Dense(64, activation='relu',
        kernel_regularizer=regularizers.l2(1e-4)),
    layers.Dropout(0.3),
    layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

callbacks = [
    keras.callbacks.EarlyStopping(
        monitor='val_loss', patience=5,
        restore_best_weights=True),
    keras.callbacks.ModelCheckpoint(
        'best.keras', save_best_only=True)
]

model.fit(X_train, y_train, epochs=200,
          validation_split=0.2,
          callbacks=callbacks)

- L2 regularization: small penalty (1e-4)

- Dropout: 30% after each hidden layer

- EarlyStopping: patience=5, restore best

- ModelCheckpoint: save best to file

- Train 200 epochs, but only keep the best version

[Tip] Start with this recipe, then tune if needed

Strategy Cheat Sheet

Overfitting? (gap between train/val)

- Add Dropout
- Add L2 regularization
- Reduce model size
- Add more training data
- Data augmentation
- Early stopping

Underfitting? (both metrics poor)

- Add more layers/neurons
- Train longer (more epochs)
- Reduce regularization
- Better features / preprocessing
- Lower learning rate
- Check data quality

Summary + Homework

What we covered

- Train/validation/test splits
- Overfitting vs Underfitting: how to diagnose
- L1/L2 regularization
- Dropout
- EarlyStopping + ModelCheckpoint
- Data augmentation

Homework (due next week)

- Take your wk04 Fashion MNIST classifier
- Add Dropout(0.3) after each hidden layer
- Add EarlyStopping(patience=5)
- Plot train/val curves: before and after
- Compare: does regularization help val accuracy?
- Use uv init --no-package wk05-hw

Next week: Feature Engineering and preprocessing

Evaluation, Overfitting,and Underfitting

Learning Outcomes

Today's Roadmap

Part 1: Why three splits?

Train

Validation

Test

Validation in Keras

Reading Learning Curves

Good fit

Overfitting

Underfitting

Plotting learning curves

Part 2: Overfitting -- model memorizes

Causes

Symptoms

Underfitting -- model too simple

Causes

Solutions

Model Capacity and the sweet spot

Low capacity

Right capacity

High capacity

Part 3: L1/L2 Regularization

Dropout -- random deactivation

Early Stopping -- stop when val stops improving

ModelCheckpoint -- save best model

Data Augmentation -- more training data

Putting it all together

Strategy Cheat Sheet

Overfitting? (gap between train/val)

Underfitting? (both metrics poor)

Summary + Homework

What we covered

Homework (due next week)

Evaluation, Overfitting,
and Underfitting