AI/ML · ML algorithms

Probabilistic Models

Predict the label that is most probable given what we see.

Contents

  1. History
  2. Inferencing
  3. Unbiased Estimation
  4. MLE
  5. MAP
  6. Brute Force Bayes
  7. Naive Bayes

Checkpoint

  1. History
  2. Inferencing
  3. Unbiased Estimation
  4. MLE
  5. MAP
  6. Brute Force Bayes
  7. Naive Bayes

Brute Force Bayes

Brute Force Bayes

$$ \hat{y} = \underset{y \in \{0,1\}}{\arg\max} \, P(y \mid x) $$

$$ \phantom{\hat{y}} = \underset{y \in \{0,1\}}{\arg\max} \, \frac{P(x \mid y)\,P(y)}{P(x)} $$

$$ \phantom{\hat{y}} = \underset{y \in \{0,1\}}{\arg\max} \, P(x \mid y)\,P(y) $$

\(P(x)\) is the same for both labels, so it cannot change which one wins. Revise priors and Bayes with the Beta distribution deck.

Brute Force Bayes, by hand

\(P(y{=}0) = 1 - P(y{=}1)\)

y = 0
y = 1

Bars: \(P(x \mid y)\,P(y)\). The taller bar is the prediction. A strong likelihood can lose to a small prior.

Checkpoint

  1. History
  2. Inferencing
  3. Unbiased Estimation
  4. MLE
  5. MAP
  6. Brute Force Bayes
  7. Naive Bayes

Naive Bayes

Naive assumption

$$ P(\mathbf{x} \mid y) = \prod_{i} P(x_i \mid y) $$

Treat every feature \(x_i\) as independent of the others once the label is known. Rarely true, often good enough.