Transfer Learning & Fine-Tuning (NumPy) — Student Lab

We simulate transfer learning with a small MLP on sklearn.datasets.load_digits.

Source task (pretrain): even vs odd. Target task (fine-tune): 10-class digit classification.

Section 0 — Data splits (simulate low-data target)

We create:

• ●a large pool for source pretraining
• ●a small target training set (e.g., 200 samples)
• ●a held-out validation set

FAANG gotcha: standardize using train stats only to avoid leakage.

Section 1 — Baseline: LogisticRegression on small target set

This baseline is intentionally strong and hard to beat with tiny models.

Task 1.1

Train LogisticRegression on the small target train split and report train/validation accuracy.

Section 2 — A small MLP (feature extractor + head)

We define MLP helper functions used in later sections.

Task 2.1

Implement init_mlp, forward, backward, accuracy, and train_mlp.

Section 3 — Train from scratch on the small target set

Task 3.1

Train an MLP from scratch on y10 using only the small target training set.

Section 4 — Pretrain on the source task (even vs odd)

We pretrain the same MLP but with a 2-class head.

Task 4.1

Train a 2-class MLP on the source task.

Section 5 — Transfer: frozen features vs fine-tuning

We reuse (W1, b1) from pretraining and create a new 10-class head.

Task 5.1

Frozen features: update only (W2,b2) (new head).

Task 5.2

Fine-tuning: update all parameters starting from the pretrained init (usually smaller LR).

Section 6 — Compare results

Which is best on validation: scratch, frozen, or finetune?