Connection 1 of 4
Measure each picture
Each 16 × 10 plate picture has two switch slots. For a slot, a is the fraction of the 12 top-half pixels that are bright (level 6 or more). An UP switch lights the whole top half, so a = 1; a DOWN switch lights none of it, so a = 0. The lamp is on when exactly one switch is up.
a = bright top-half pixels / 12Read it as: Count the bright pixels in the top half of the slot and divide by twelve.
Connection 2 of 4
Stack two layers
Two stacked linear layers on the switch features collapse into one layer, 3a − b − 1: still one straight line, and still two plates wrong.
With one number x, the count of switches up, layer one gives 2x + 1 and layer two gives 3u − 2, so the stack is 3(2x + 1) − 2 = 6x + 1. For the plates with 0, 1 and 2 switches up that is 1, 7 and 13 either way. In general the stack collapses into one affine layer.
W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂)Read it as: Applying two layers in turn equals applying one layer with the multiplied weights and the combined constant.
Connection 3 of 4
Add the bend
The full forward pass for all four wall plates: scores, ReLU, sum and the lamp prediction, four plates right.
ReLU(x) = max(0, x). Two hidden units, h₁ = ReLU(a − b) and h₂ = ReLU(b − a), then out = h₁ + h₂, with the lamp on when out is above one half. A up gives (1, 0), B up gives (0, 1), and both-down and both-up give (0, 0).
h = ReLU(W₁ (a, b)), out = h₁ + h₂Read it as: Score the plate two ways, keep only the positive scores, and add them.
Connection 4 of 4
Take the bend away
Without ReLU, h₁ + h₂ = (a − b) + (b − a) = 0 for every plate: W₂W₁ = [0 0] and the constant is 0. The network is one affine map, calls every lamp off, and gets 2 of 4. No single straight line gets more than 3 of 4.
W₂W₁ = [1 1][[1 −1],[−1 1]] = [0 0]Read it as: Multiplying the two layers' weights gives zeros.
Connect it back to the picture
Linear layers transform representations. Nonlinear activations let stacked layers express relationships that linear transformations alone cannot.
Full notation and technical reference
The expressions below collect the operations discussed above. You can return here when you want the compact notation.
W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂)
ReLU(x) = max(0, x)
h = ReLU(W₁(a, b)), out = h₁ + h₂- a and b are measured from the pictures: the fraction of bright pixels in the top half of each switch slot, exactly 0 or 1 here.
- Linear means Wx; affine adds a constant, Wx + b. Stacked layers stay affine.
- The weights are chosen by hand to show what the network can represent. Training, which finds weights, is a separate question.
Numerical reference
With W₁ = [1 −1; −1 1] and W₂ = [1 1], the plate with A up has scores (1, −1), hidden values (1, 0) and output 1 (lamp on); both up has scores (0, 0) and output 0. Without ReLU, W₂W₁ = [0 0], so every output is 0.
Where this explanation stops
ReLU does not learn or detect anything; the weights make the scores and ReLU decides what passes on. A bend is not magic: not every nonlinearity solves every problem. The two-unit network is one valid solution, not the only one.