Disclaimer: This article is based on my personal lecture notes from my studies at Machine Learning for Data Science 2023/2024 course: Bayesian Learning. The content reflects my understanding and interpretation of the course material.

Objective

Given a data point x, what is the probability of x belonging to some class c?

Our objective is to assess the likelihood of a given data point $x$ belonging to a particular class $c$, rather than precisely determining its label. This approach involves calculating the probability \$P(y=c|x)$ or $P(c|x)$, signifying the probability of $x$ being associated with class $c$.

$$ c = \argmax_c P(c|X) $$

We define the class of each point by choosing the class c with the highest probability

Bayes’ theorem

$$ P(A|B) = \frac{P(B|A)P(A)}{P(B)} $$

Naive Bayes Classifier formula

$$ P(c|x) = \frac{p(x|c)P(c)}{p(x)} = \frac{p(x|c)P(c)}{\sum_{c}p(x|c)p(c)} $$

or

$$ Posterior = \frac{Likelihood \times Prior}{Evidence} $$

where

• $x$ is $j$ features
• $P(c|x)$is Posterior Probability
• $p(x|c)$ is Likelihood of feature $x_j$ given that their class is T
• $P(c)$ is Prior Probability
• $P(x)$ is Marginal Probability or called Evidence

We have