Why deep nets fight back

A shallow network with one or two hidden layers is forgiving. Stack ten, twenty, or a hundred layers and things change: gradients can vanish or explode on their way back through backpropagation, training can crawl to a halt, and a model with millions of parameters can memorize the training set instead of learning from it.

In this phase, you’ll learn the toolkit that makes deep networks actually trainable:

• how backpropagation computes gradients, and which optimizer (SGD, momentum, RMSProp, Adam) to reach for
• why gradients vanish or explode, and how Glorot/He initialization plus nonsaturating activations (ReLU, ELU, SELU) fix it
• Batch Normalization — letting each layer normalize its own inputs on the fly
• regularization: L1/L2 penalties, Dropout, MC Dropout, and max-norm
• learning rate scheduling — decaying the learning rate on a schedule instead of leaving it constant

Phase 2 flow

  flowchart TD
  A["Backprop + Optimizers
(SGD, Momentum, Adam)"] --> B["Vanishing & Exploding Gradients
(init, activations, clipping)"]
  B --> C[Batch Normalization]
  C --> D["Regularization & Dropout
(L1/L2, Dropout, MC Dropout)"]
  D --> E["Learning Rate Scheduling
(power, exponential, 1cycle)"]

How the pieces fit together

None of these techniques are used in isolation — a real tf.kerastf.keras training script usually combines several at once: He initialization on every layer, ELU or ReLU activations, a BatchNormalization layer after each Dense layer, a touch of L2 regularization or Dropout, and Adam (or Nadam) as the optimizer with a decaying learning rate. By the end of this phase you’ll understand why each ingredient earns its place in that recipe.