How the Naive Bayes algorithm works

The Naive Bayes algorithm uses Bayes' theorem to calculate the posterior probability of every condition in the dataset and uses these probabilities to calculate the conditional probability of an event given a set of conditions. The Naive Bayes algorithm assumes that each conditional feature is independent of each other. This is an important assumption that helps simplify how the conditional probability is calculated. The independence assumption is the reason why the algorithm gets the name, Naive Bayes. 

In this section, instead of considering one x feature variable, we consider a vector of features, , where n is the number of feature variables used to calculate the class probability. We represent the conditional probability of a class variable for the x vector in formula 3:

Formula 3

As we have assumed that each feature variable is independent of each other, the conditional probability of a class variable can be calculated as follows:

Formula 4

Based on posterior probability calculations shown in the previous sections, this formula can be rewritten as follows:

Formula 5

Formula 5 explains how a probability of event C is calculated based on the  feature variables. An interesting thing to note in this formula is how easy it is to calculate each element from the dataset. Also, since the evidence probability from Bayes' theorem is not dependent on the class variable, it is not used in the Naive Bayes formula.

The Naive Bayes algorithm only requires one pass over the dataset during the training phase to calculate the probability of the value of a feature for each event. During the prediction phase, we calculate the probability of each event given the instance of the features and predict the event with the highest probability. Formula 6 shows how the prediction of a Naïve Bayes classifier is calculated when k events are possible. Argmax in the formula means that the event with maximum probability is selected as the prediction:

 Formula 6

Naïve Bayes classifier is a multiclass classifier that can be used to train on a dataset where two or more class variables need to be predicted. In the next chapters, we will present some examples of binary classifiers that only work with two class variables needs to be predicted. However, we will show you the methodologies of applying binary classifiers to multiclass problems.