A model learns the patterns in its data, including the ones we would rather it didn't.
manwomanAn illustration, not Amazon's real data
Most past hires were men, so "looks like a man" becomes part of the pattern.
Equally qualified women now score lower. Nobody programmed the bias: it was learned.
