10 Probability Concepts for Machine Learning Explained Simply

 

# Introduction to Probability Concepts

 
For a long time I treated probability as the vegetables of machine learning. The boring stuff you choke down before you get to the good part. Later on, I realized that probability is not just a prerequisite for machine learning but it makes much of machine learning work. But nobody tells you up front that a model is almost never sure of anything. Is this image a cat? Probably. Is this transaction fraud, or did someone just buy a lot of socks at 2am? Hard to say. A language model has no clue what word you're about to type next, so it doesn't pretend to. It spreads its confidence across a bunch of options and hands you the most likely one. Once that clicked for me, a lot of the rest stopped feeling like memorization. You don't need a stats PhD for any of this. You need maybe ten ideas. Here they are, the way I wish someone had explained them to me:

 

# 1. Random Variables

 
Let's start with one of the most fundamental ideas in probability.

You don't know if an email is spam before it shows up. You don't know if a visitor will buy anything before they arrive. You don't know what a model will spit out before you run it. Any value that depends on an outcome you haven't seen yet is a random variable, and in machine learning that covers basically everything: the features, the labels, the errors, the outputs.

By convention, random variables are usually written using uppercase letters such as X and Y, while specific observed values are written using lowercase letters such as x and y.

So for a spam classifier you'd write the label as:

\[
Y =
\begin{cases}
1 & \text{if the email is spam} \\
0 & \text{if it's not}
\end{cases}
\]

Y is binary. Before you read the email it could be either. The second it gets labeled, the mystery's gone and you just have a number.

In supervised learning, the usual cast is X for the input features and Y for the target, and the model is chasing this:

\[
P(Y \mid X)
\]

 

Given what I can see, how likely is each label?

 

Feed it the words, the sender, the suspicious links, and it might come back with:

\[
P(Y = 1 \mid X = x) = 0.92
\]

Translation: it's 92% sure this email is spam. Not certain. Just pretty confident.

 

# 2. Probability Distributions

 
Okay, so a variable can take different values. The follow-up question writes itself: which values, and how often? That whole map is a probability distribution.

The bookkeeping rule is simple. For discrete stuff, the probabilities have to add up to one, no more, no less:

\[
\sum_x P(X=x) = 1
\]

For continuous stuff it's the area under the curve that equals one:

\[
\int_{-\infty}^{\infty} p(x)\,dx = 1
\]

Different data wants different distributions. A yes/no thing like spam is a Bernoulli:

\[
Y \sim \text{Bernoulli}(p)
\]

where \(p\) is the probability of a 1. For example, if

\[
P(Y = 1) = 0.3
\]

then

\[
P(Y = 0) = 0.7
\]

and you're done.

Continuous values such as prediction errors, temperatures, heights, or sensor readings are often approximated using a Gaussian distribution:

\[
X \sim \mathcal{N}(\mu, \sigma^2)
\]

where:

• \(\mu\) is the mean
• \(\sigma^2\) is the variance

Probability distributions are important because machine-learning models are often trying to learn one. A regression model tries to estimate likely values for a continuous target, while a classifier tries to estimate a probability distribution over possible classes:

\[
p_\theta(y \mid x)
\]

Here, \(\theta\) represents the parameters learned by the model during training.

 

# 3. Expectation, Variance, and Standard Deviation

 
Suppose you repeat an experiment many times and record the results.

 

What value would you expect to see on average?

 
That average is the expectation, also called the expected value or the mean.

For a discrete variable:

\[
\mathbb{E}[X] = \sum_x xP(X=x)
\]

For a continuous variable:

\[
\mathbb{E}[X] = \int x p(x)\,dx
\]

Where this shows up is average performance. For example, say you have a model that predicts house prices. Its prediction error will vary from one house to another. The expected error tells us the average error we would expect across many predictions.

But, an average can lie to you. Two models, both averaging $10,000 of error. One model gives values near $10,000 almost every time. The other is off by $1,000 on one house and $50,000 on the next. Same average. Completely different behavior, and you'd want to know that before deploying either one.

That's what variance is for. It measures the spread:

\[
\mathrm{Var}(X) = \mathbb{E}\left[(X - \mu)^2\right], \quad \mu = \mathbb{E}[X]
\]

Take the square root and you get the standard deviation:

\[
\sigma = \sqrt{\mathrm{Var}(X)}
\]

I almost always reach for standard deviation over variance because it uses the same units as the original data.

 

# 4. Conditional Probability

 
A model basically never asks a question without context.

A model does not simply ask:

 

What is the probability that an email is spam?

 

Instead, it asks:

 

What is the probability that an email is spam given the information I can see?

 

This little "given" is conditional probability: the chance of one thing once you already know another.

\[
P(A \mid B) = \frac{P(A \cap B)}{P(B)}
\]

Most classifiers are trying to estimate:

\[
P(Y \mid X)
\]

which means:

What is the probability of a label given the observed features?

For example:

\[
P(\text{Spam} \mid \text{Email contains "free"})
\]

represents the probability that an email is spam given that it contains the word "free."

Suppose that among all emails containing the word "free," 80% turn out to be spam.

Then:

\[
P(\text{Spam} \mid \text{contains "free"})=0.8
\]

Notice how the condition changes the probability.

Maybe only 20% of all emails are spam overall. But once we observe a useful clue, such as the word "free," the probability increases significantly. This is how a model makes predictions: it observes features and continuously updates its estimate of each possible outcome.

 

# 5. Bayes' Theorem

 
Conditional probability naturally leads to one of the most famous formulas in statistics: Bayes' theorem.

Bayes' theorem has a reputation for being intimidating, and I think that's mostly a marketing problem. All it really does is tell you how to change your mind when new evidence shows up.

\[
P(A \mid B)=\frac{P(B \mid A)P(A)}{P(B)}
\]

It has four important quantities:

• \(P(A)\): your initial belief (the prior)
• \(P(B \mid A)\): how likely the evidence is if A is true
• \(P(B)\): how common the evidence is in general
• \(P(A \mid B)\): your updated belief after seeing the evidence

Let's return to the spam example. Let A be "the email is spam" and B be "it contains the word free" then Bayes' theorem becomes:

\[
P(\text{Spam} \mid \text{"free"})=
\frac{
P(\text{"free"} \mid \text{Spam})P(\text{Spam})
}{
P(\text{"free"})
}
\]

Suppose that spam is fairly rare, but when it does land, it practically screams "free" at you. So the moment that word appears, your suspicion should shoot up. This is what this formula actually does. This way of thinking is all over the place: Naive Bayes classifiers, Bayesian neural nets, the diagnostic systems that weigh symptoms, anything that has to mix what it already knew with what it just learned.

 

# 6. Joint, Marginal, and Conditional Distributions

 
Up to now it's been one variable at a time. But in machine learning, we often care about how multiple variables relate to each other.

When building that spam detector, you might track two things at once: does the email contain a link, and is it spam. Those aren't strangers. The joint distribution is the probability of both happening together, written as

\[
P(X, Y)
\]

If \(X\) is "has a link" and \(Y\) is "is spam," then

\[
P(X=\text{link},\, Y=\text{spam})
\]

is the probability that both events occur at the same time.

Sometimes, though, you only care about one of them and want the other to vanish. That's a marginal distribution, and you get it by adding up across everything you're ignoring:

\[
P(X) = \sum_y P(X, y)
\]

You sum over every possible value of \(Y\) until only \(X\) is left in the room.

And the conditional distribution is the one you've already seen above, dressed up in the joint:

\[
P(Y \mid X) = \frac{P(X, Y)}{P(X)}
\]

These three are basically the same family. A model often learns the joint story of, say, images and labels, then uses it to estimate the conditional probability: given this image, which label?

Now the fun one. Independence. Two variables are independent when knowing one tells you precisely nothing about the other:

\[
P(X, Y) = P(X)\,P(Y)
\]

When that's true, they ignore each other completely.

However, in the real world, true independence is rare. But pretending it's true anyway can make a model dramatically simpler, and a classic example is Naive Bayes. It assumes that features are conditionally independent once the class label is known:

\[
P(x_1, x_2, \dots, x_d \mid y)
=
\prod_{j=1}^{d} P(x_j \mid y)
\]

This is, to be blunt, a lie. Words in a sentence are deeply entangled. And yet Naive Bayes refuses to fail, especially on text. It's one of those cases where the wrong assumption somehow still gets you to the right place, which used to drive me a little crazy until I made peace with it.

 

# 7. Likelihood and Maximum Likelihood Estimation

 
When training a machine-learning model, we are trying to ask a simple question:

 

How well do the model's parameters explain the data we observed?

 

The number that answers it is the likelihood.

A likelihood measures how probable the observed data is under a particular set of model parameters.

Suppose a model with parameters \(\theta\) assigns probabilities to outcomes:

\[
p_\theta(y_i \mid x_i)
\]

For a dataset containing \(n\) independent examples, the likelihood is:

\[
\mathcal{L}(\theta)
=
\prod_{i=1}^{n}
p_\theta(y_i \mid x_i)
\]

You can think of this as multiplying together the probabilities that the model assigns to all observed training examples. A model that consistently assigns high probability to the correct outcomes will have a high likelihood.

This leads to a training strategy called Maximum Likelihood Estimation (MLE).

The idea is simple: choose the parameter values that make the observed data as likely as possible.

Mathematically:

\[
\hat{\theta}_{\text{MLE}}
=
\arg\max_\theta
\mathcal{L}(\theta)
\]

In practice, multiplying many probabilities together can produce extremely small numbers that are difficult for computers to work with.

To avoid this problem, machine-learning systems usually maximize the log-likelihood instead:

\[
\log \mathcal{L}(\theta)
=
\sum_{i=1}^{n}
\log p_\theta(y_i \mid x_i)
\]

The mathematics becomes easier, and the optimization process becomes more stable.

And if you've ever trained a classifier with binary cross-entropy, surprise, you were doing this the whole time. Maximizing log-likelihood and minimizing cross-entropy are the same act wearing different hats. The intuition behind all of this is simple. A decent model should give high probability to the things that actually happened. If your spam filter keeps seeing spam and keeps shrugging "looks fine to me," it's broken, and the parameters need to be adjusted.

 

# 8. Sampling, the Law of Large Numbers, and the Central Limit Theorem

 
Nobody gets the whole population. You get a chunk of it, and that chunk is your sample. A company might log half a billion clicks and you'll train on three million of them, because that's what fits and that's what's labeled.

The sample mean is:

\[
\bar{X}
=
\frac{1}{n}
\sum_{i=1}^{n}X_i
\]

Naturally, this raises an important question:

 

Can I actually trust a number I computed from a slice instead of the whole pie?

 

The Law of Large Numbers says, mostly, yes. Grow the sample and the average drifts toward the true expected value:

\[
\bar{X}_n
\rightarrow
\mathbb{E}[X]
\]

This is why averages computed from large, representative datasets tend to be more reliable than averages computed from small datasets.

Then there's the Central Limit Theorem, which I still think is genuinely weird in the best way. It tells us that under fairly common conditions, the average of many independent samples becomes approximately normally distributed:

\[
\bar{X}
\approx
\mathcal{N}
\left(
\mu,
\frac{\sigma^2}{n}
\right)
\]

The standard error of the mean is:

\[
\frac{\sigma}{\sqrt{n}}
\]

Notice what happens as \(n\) increases.

The denominator gets larger, which means the uncertainty becomes smaller. In simple terms, larger samples usually produce more stable estimates.

None of this is academic trivia, by the way. Mini-batch gradient descent estimates the gradient from a random handful of examples. A/B tests estimate behavior from samples. Validation sets estimate the future from a held-out slice. A frankly alarming amount of machine learning works only because these results let you trust a part to stand in for the whole.

 

# 9. Entropy, Cross-Entropy, and Kullback-Leibler Divergence

 
Not all probability distributions are equally uncertain.

Consider these two predictions:

\[
[0.99, 0.01]
\]

and

\[
[0.50, 0.50]
\]

The first prediction is highly confident. The model strongly believes one class is correct.

The second prediction is much less certain because both classes seem equally likely.

This idea of uncertainty is captured by entropy.

Entropy measures how much uncertainty exists in a probability distribution.

For a discrete distribution \(p\):

\[
H(p)
=
-\sum_x p(x)\log p(x)
\]

Low entropy means the distribution is very confident.

High entropy means the distribution is much more uncertain.

In machine learning, we often want to compare a model's predictions with the correct answers.

This leads us to cross-entropy.

Cross-entropy measures how well a predicted distribution \(q\) matches a target distribution \(p\):

\[
H(p,q)
=
-\sum_x p(x)\log q(x)
\]

In classification problems, the target is usually represented using one-hot encoding.

Suppose the true class is:

\[
y=[1,0]
\]

and the model predicts:

\[
\hat{y}=[0.9,0.1]
\]

The cross-entropy loss becomes:

\[
-\log(0.9)
\]

which is relatively small.

Now suppose the model predicts:

\[
\hat{y}=[0.01,0.99]
\]

The loss becomes:

\[
-\log(0.01)
\]

which is much larger.

This is exactly what we want. Models should be penalized heavily when they are confidently wrong.

Another related concept is Kullback-Leibler (KL) divergence.

KL divergence measures how different one probability distribution is from another:

\[
D_{\text{KL}}(p \parallel q)
=
\sum_x p(x)
\log
\frac{p(x)}{q(x)}
\]

A KL divergence of zero means the distributions are identical. Larger values indicate greater differences.

KL divergence and cross-entropy are closely related:

\[
H(p,q)
=
H(p)
+
D_{\text{KL}}(p \parallel q)
\]

KL turns up everywhere once you start looking: variational autoencoders, reinforcement learning, knowledge distillation, anywhere a method needs to ask "how far has my learned distribution drifted from the one I want."

 

# 10. Calibration and Predictive Uncertainty

 

When a model announces it's 95% sure, should you actually believe it?

 

That question leads us to the idea of calibration.

Calibration measures whether a model's confidence scores actually match reality.

For a well-calibrated binary classifier:

\[
P(Y=1 \mid \hat{P}=p)
\approx p
\]

Suppose a model makes 100 predictions and assigns each one a confidence score of 80%.

If the model is well calibrated, approximately 80 of those predictions should be correct.

A model can be accurate overall but still poorly calibrated.

For example, a model may claim 99% confidence while only being correct 75% of the time. Such a model is considered overconfident.

Poor calibration can be dangerous in areas such as fraud detection, healthcare, automated decision-making systems, and large language model (LLM) agents that must decide whether they know enough to answer a question.

One common calibration metric is the Brier Score:

\[
\text{Brier Score}
=
\frac{1}{n}
\sum_{i=1}^{n}
(\hat{p}_i-y_i)^2
\]

where \(\hat{p}_i\) is the predicted probability and \(y_i\) is the true label.

 

# Final Thoughts

 
Probability is not just a supporting topic in machine learning. It explains much of what machine-learning models are doing behind the scenes.

You'll keep bumping into these. I still do, constantly, years in. The good news is that once they stop feeling like obstacles and start feeling like tools, the rest of machine learning gets a whole lot less mysterious.
 
 

Kanwal Mehreen is a machine learning engineer and a technical writer with a profound passion for data science and the intersection of AI with medicine. She co-authored the ebook "Maximizing Productivity with ChatGPT". As a Google Generation Scholar 2022 for APAC, she champions diversity and academic excellence. She's also recognized as a Teradata Diversity in Tech Scholar, Mitacs Globalink Research Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having founded FEMCodes to empower women in STEM fields.