Connection 1 of 4
Make a positive amount from every score
Each score traced through the exponential: rind 1.2137 becomes 3.3660, flesh −1.5902 becomes 0.2039; every amount is positive and the order is kept.
Exponentiation turns any finite score into a positive amount. Higher scores produce larger amounts. A score difference of one becomes an amount ratio of e, about 2.7183. The rind score 1.2137 becomes 3.3660; the flesh score −1.5902 becomes 0.2039.
That is why we cannot read the original signed scores as percentages.
amount = exp(score)Read it as: exp names the exponential function; its output is always positive.
Connection 2 of 4
Convert amounts into shares
Each amount divided by the same total 4.8244: flesh 4.2%, rind 69.8%, seed 18.9%, background 7.1%, a bar that adds up to one.
The four amounts 0.2039, 3.3660, 0.9138 and 0.3408 add to 4.8244. Rind owns 3.3660 of those 4.8244 units, so its share is about 0.6977.
This is not the raw scores divided by their sum: the scores add to −1.5431, and dividing by that would give negative and above-one results.
share = class amount / total amountRead it as: The same division for every class gives flesh 4.2%, rind 69.8%, seed 18.9% and background 7.1%; the exact shares sum to one.
Connection 3 of 4
Why subtracting the maximum changes nothing important
Adding 2 to every score multiplies every amount and the total by e² = 7.3891; the factor cancels and every share is unchanged.
Subtracting m from every score multiplies every exponential amount by the same factor exp(−m). That factor appears in both the numerator and denominator, so it cancels.
The shares stay the same, while the largest exponential becomes 1 and the total is at least 1. For the watermelon pixel the shifted amounts are 0.0606, 1, 0.2715 and 0.1012, total 1.4333: the same shares, and e to the power 1000 never has to be computed.
exp(zₖ−m) / Σⱼ exp(zⱼ−m) = exp(zₖ) / Σⱼ exp(zⱼ)Read it as: Shifting all the scores together changes their scale as amounts, not their relative shares.
Connection 4 of 4
A large share is still a model output
Equal scores get equal shares, and the largest share need not exceed one half. Raising one score changes every share through the shared total. But softmax only transforms the supplied scores; it does not check whether the model is correct.
With the same hand-set weights, a white inner rind pixel gets a 61.9% background share: a confident-looking share for the wrong part of the picture.
Connect it back to the picture
Softmax makes the competing amounts comparable as shares of one total. It preserves ranking; it does not certify the decision.
Full notation and technical reference
The expressions below collect the operations discussed above. You can return here when you want the compact notation.
pₖ = e^zₖ / Σⱼ e^zⱼ
m = maxⱼ zⱼ
pₖ = exp(zₖ − m) / Σⱼ exp(zⱼ − m)- z is the vector of logits (raw class scores); p is the vector of shares, one per class.
- Adding the same amount to every score multiplies every amount and the total by the same factor, which cancels. Subtracting the maximum uses this to avoid overflow; the result is identical.
Numerical reference
For the watermelon pixel's logits (flesh −1.5902, rind 1.2137, seed −0.0902, background −1.0765), the amounts e^z are 0.2039, 3.3660, 0.9138 and 0.3408, the total is 4.8244, and the shares are 4.2%, 69.8%, 18.9% and 7.1%. Raising seed's score by one moves every share: 3.2%, 52.6%, 38.8%, 5.3%.
Where this explanation stops
Softmax preserves ranking and ties, and equal scores give equal shares; the largest share need not exceed one half. It is not the raw scores divided by their sum. A large share is not a verified chance of being correct: with these hand-set weights a white inner rind pixel gets a 61.9% background share.