Questions 5–7: a ReLU network that can't learn
4 → 3 (ReLU) → 1 (sigmoid), all weights 1, no biases, squared loss, x = [−1, −1, −1, −1], y = 1.
Q5: 0.125. Pre-activations are −4, ReLU gives h = 0, so ŷ = σ(0) = 0.5 and L = ½(1 − 0.5)².
Q6: 0. ReLU′(−4) = 0, so every gradient into W⁽¹⁾ is zero (dead ReLUs).
Q7: Yes. With a sigmoid hidden layer, σ′(−4) ≈ 0.0177 ≠ 0, so all 12 weights in W⁽¹⁾ get updated.
Q6: 0. ReLU′(−4) = 0, so every gradient into W⁽¹⁾ is zero (dead ReLUs).
Q7: Yes. With a sigmoid hidden layer, σ′(−4) ≈ 0.0177 ≠ 0, so all 12 weights in W⁽¹⁾ get updated.