Probability • Topic 2 of 3

Total Probability & Bayes’ Theorem

Many experiments happen in two stages: first some hidden "cause" is selected, then an observable event follows. Two results let us move between the forward direction (cause $\to$ effect) and the reverse direction (effect $\to$ likely cause).

Partition of a sample space

A collection of events $E_1,E_2,\dots,E_n$ is a partition of the sample space $S$ if they are pairwise mutually exclusive ($E_i\cap E_j=\varnothing$ for $i\ne j$), collectively exhaustive ($E_1\cup E_2\cup\cdots\cup E_n=S$), and each has positive probability ($P(E_i)>0$). Intuitively the $E_i$ slice the whole space into non-overlapping pieces that together cover everything — the "causes" through which any event $A$ must arise.

Theorem of total probability

If $E_1,\dots,E_n$ partition $S$, then for any event $A$:

$$P(A)=\sum_{i=1}^{n} P(E_i)\,P(A\mid E_i).$$

The logic is a tree diagram: $A$ can happen along any branch, so add up "probability of taking branch $E_i$" times "probability of $A$ along that branch". This turns a hard direct probability into a weighted sum of easy conditional ones.

Bayes' theorem

Bayes' theorem reverses the conditioning — from the known $P(A\mid E_i)$ to the wanted $P(E_k\mid A)$:

$$P(E_k\mid A)=\dfrac{P(E_k)\,P(A\mid E_k)}{\displaystyle\sum_{i=1}^{n} P(E_i)\,P(A\mid E_i)}.$$

The denominator is just $P(A)$ from the total-probability theorem, so Bayes is "one favourable branch divided by all branches that produce $A$". The $P(E_i)$ are the prior probabilities (belief about the cause before observing $A$); $P(E_k\mid A)$ is the posterior (revised belief after $A$ is observed). This is the foundation of quality-control, medical-testing and "which bag was chosen" problems.

A reliable solving recipe. (1) Name the partition $E_i$ — the mutually exclusive causes (machine A/B/C, has-disease / no-disease, Bag I / Bag II). (2) Write each prior $P(E_i)$. (3) Write each likelihood $P(A\mid E_i)$ for the observed event $A$. (4) Compute $P(A)=\sum P(E_i)P(A\mid E_i)$. (5) Divide the relevant term by $P(A)$. Drawing the tree first prevents almost every mistake.

Deeper Insight — the base rate is everything. The famous, counter-intuitive lesson of Bayes' theorem is the base-rate effect: even a very accurate test for a rare condition produces mostly false alarms, because the rare prior $P(E_k)$ multiplies the whole numerator. A test that is $99\%$ accurate for a disease affecting $1$ in $1000$ people will, among everyone who tests positive, leave the majority actually healthy. The mathematics is nothing more than total probability in the denominator and a single branch in the numerator — but it overturns the naive instinct that "positive test" means "almost certainly sick". Internalising why the prior cannot be ignored is the real payoff of this topic.

Probability tree showing a partition into three causes, each branching to event A, with the total probability and Bayes formulas Total Probability & Bayes' Theorem P(E₁) P(E₂) P(E₃) E₁ E₂ E₃ P(A∣E₁) P(A∣E₂) P(A∣E₃) A Total Probability P(A) = ∑ P(Eₙ) P(A∣Eₙ) E₁,E₂,E₃ partition S (disjoint, exhaustive) Bayes' Theorem reverse the arrow P(Eₙ∣A) = P(Eₙ) P(A∣Eₙ) ÷ ∑ P(Eₘ) P(A∣Eₘ)
Solved Examples
1
Worked Example
Bag I has 3 red, 2 black; Bag II has 1 red, 4 black. A bag is chosen at random and a red ball drawn. Find $P(\text{red})$.
Solution

The partition is $E_1=$ "Bag I chosen", $E_2=$ "Bag II chosen", each with prior $\tfrac12$. The likelihoods are $P(\text{red}\mid E_1)=\tfrac35$ and $P(\text{red}\mid E_2)=\tfrac15$. By total probability, $P(\text{red})=\tfrac12\cdot\tfrac35+\tfrac12\cdot\tfrac15=\tfrac{3}{10}+\tfrac{1}{10}=\dfrac{4}{10}=\dfrac{2}{5}.$ Answer: $\dfrac{2}{5}$.

2
Worked Example
For the bags above, given a red ball was drawn, find $P(\text{Bag I}\mid \text{red})$.
Solution

Apply Bayes with the $P(\text{red})=\tfrac25$ just found as the denominator: $P(\text{I}\mid \text{red})=\dfrac{P(E_1)P(\text{red}\mid E_1)}{P(\text{red})}=\dfrac{\tfrac12\cdot\tfrac35}{\tfrac{2}{5}}=\dfrac{3/10}{4/10}=\dfrac{3}{4}.$ The posterior $\tfrac34$ exceeds the prior $\tfrac12$ because Bag I is richer in red. Answer: $\dfrac{3}{4}$.

3
Worked Example
State, with one example, the difference between prior and posterior probability.
Solution

The prior $P(E_i)$ is the probability of a cause before any evidence — e.g. each of two bags is equally likely, $P(\text{Bag I})=\tfrac12$. The posterior $P(E_i\mid A)$ is the revised probability after observing $A$ — e.g. after a red ball is drawn, $P(\text{Bag I}\mid \text{red})=\tfrac34$. Evidence updates belief from prior to posterior. Answer: prior = before evidence; posterior = after observing the event.

4
Worked Example
If $P(E_1)=0.4,\ P(E_2)=0.6,\ P(A\mid E_1)=0.5,\ P(A\mid E_2)=0.5$, find $P(A)$.
Solution

By total probability, $P(A)=P(E_1)P(A\mid E_1)+P(E_2)P(A\mid E_2)=0.4(0.5)+0.6(0.5)=0.2+0.3=0.5.$ Because both likelihoods are equal, $A$ is independent of which cause occurs. Answer: $0.5$.

5
Worked Example
A factory has two machines. Machine A makes $60\%$ of the bolts, Machine B makes $40\%$. Of A's bolts $2\%$ are defective, and of B's $3\%$ are defective. Find the probability that a randomly chosen bolt is defective.
Solution

Partition by machine: $P(A)=0.6,\ P(B)=0.4$ with $P(D\mid A)=0.02,\ P(D\mid B)=0.03$. Total probability gives $P(D)=0.6(0.02)+0.4(0.03)=0.012+0.012=0.024.$ Answer: $0.024$ (i.e. $2.4\%$).

6
Worked Example
For the factory in the previous example, a bolt is found defective. Find the probability it was made by Machine A.
Solution

Bayes with denominator $P(D)=0.024$: $P(A\mid D)=\dfrac{P(A)P(D\mid A)}{P(D)}=\dfrac{0.6\times0.02}{0.024}=\dfrac{0.012}{0.024}=\dfrac12.$ Even though A makes more bolts, its lower defect rate balances out, giving an even chance. Answer: $\dfrac{1}{2}$.

7
Worked Example
Three machines A, B, C produce $25\%,\ 35\%,\ 40\%$ of the output, with defective rates $5\%,\ 4\%,\ 2\%$ respectively. An item is found defective. Find the probability it came from machine C.
Solution

Priors $0.25,0.35,0.40$; likelihoods $0.05,0.04,0.02$. Total probability: $P(D)=0.25(0.05)+0.35(0.04)+0.40(0.02)=0.0125+0.014+0.008=0.0345.$ Then $P(C\mid D)=\dfrac{0.40\times0.02}{0.0345}=\dfrac{0.008}{0.0345}=\dfrac{16}{69}\approx0.232.$ Answer: $\dfrac{16}{69}\approx0.232$.

8
Worked Example
A rare disease affects $1\%$ of a population. A test detects it correctly $99\%$ of the time (sensitivity) and gives a false positive in $2\%$ of healthy people. If a person tests positive, find the probability they actually have the disease.
Solution

Let $E_1=$ "has disease" ($P=0.01$) and $E_2=$ "healthy" ($P=0.99$). Likelihoods of a positive test: $P(+\mid E_1)=0.99,\ P(+\mid E_2)=0.02$. Total probability: $P(+)=0.01(0.99)+0.99(0.02)=0.0099+0.0198=0.0297.$ Bayes: $P(E_1\mid +)=\dfrac{0.0099}{0.0297}=\dfrac13\approx0.333.$ Despite the accurate test, most positives are false alarms because the disease is rare — the base-rate effect. Answer: $\dfrac13\approx0.333$.

9
Worked Example
In a class, $60\%$ of students are boys and $40\%$ are girls. $30\%$ of the boys and $20\%$ of the girls passed a competitive exam with distinction. A student chosen at random has a distinction. Find the probability the student is a girl.
Solution

Priors $P(B)=0.6,\ P(G)=0.4$; likelihoods $P(D\mid B)=0.30,\ P(D\mid G)=0.20$. Total probability: $P(D)=0.6(0.30)+0.4(0.20)=0.18+0.08=0.26.$ Bayes: $P(G\mid D)=\dfrac{0.4\times0.20}{0.26}=\dfrac{0.08}{0.26}=\dfrac{4}{13}\approx0.308.$ Answer: $\dfrac{4}{13}\approx0.308$.

10
Worked Example
A man is known to speak the truth $3$ out of $4$ times. He throws a die and reports that it shows a six. Find the probability that it actually shows a six.
Solution

Let $E_1=$ "die shows six" ($P=\tfrac16$) and $E_2=$ "not six" ($P=\tfrac56$). He reports "six" truthfully if it is six ($P(R\mid E_1)=\tfrac34$) or by lying about a non-six ($P(R\mid E_2)=\tfrac14$). Total probability: $P(R)=\tfrac16\cdot\tfrac34+\tfrac56\cdot\tfrac14=\tfrac{3}{24}+\tfrac{5}{24}=\tfrac{8}{24}=\tfrac13.$ Bayes: $P(E_1\mid R)=\dfrac{\tfrac16\cdot\tfrac34}{\tfrac13}=\dfrac{3/24}{8/24}=\dfrac{3}{8}.$ Answer: $\dfrac{3}{8}$.

11
Worked Example
If $P(E_1)=0.5,\ P(E_2)=0.3,\ P(E_3)=0.2$ and $P(A\mid E_1)=0.1,\ P(A\mid E_2)=0.2,\ P(A\mid E_3)=0.3$, find $P(A)$ and $P(E_3\mid A)$.
Solution

Total probability: $P(A)=0.5(0.1)+0.3(0.2)+0.2(0.3)=0.05+0.06+0.06=0.17.$ Bayes for the third cause: $P(E_3\mid A)=\dfrac{0.2\times0.3}{0.17}=\dfrac{0.06}{0.17}=\dfrac{6}{17}\approx0.353.$ Answer: $P(A)=0.17$, $P(E_3\mid A)=\dfrac{6}{17}\approx0.353$.

Key Points

  • A partition $E_1,\dots,E_n$ is mutually exclusive, exhaustive, and each has $P(E_i)>0$.
  • Total probability: $P(A)=\sum_i P(E_i)\,P(A\mid E_i)$ — a weighted sum over all causes (a tree diagram).
  • Bayes' theorem: $P(E_k\mid A)=\dfrac{P(E_k)P(A\mid E_k)}{\sum_i P(E_i)P(A\mid E_i)}$, i.e. one branch over $P(A)$.
  • Prior $P(E_i)$ = belief before evidence; posterior $P(E_k\mid A)$ = belief after observing $A$.
  • Solving recipe: name causes, list priors, list likelihoods, sum for $P(A)$, then divide.
  • Base-rate effect: for a rare cause, even an accurate test yields mostly false positives — never ignore the prior.
Quick Check — 4 MCQs
Tap an option to check your answer0 / 4
Q1.The law of total probability gives $P(A)=$
Explanation: Sum over the partition.
Q2.Bayes' theorem computes:
Explanation: It reverses the conditioning.
Q3.The posterior probability is the one computed:
Explanation: Posterior = updated after evidence.
Q4.In the two-bag example, $P(\text{red})=$
Explanation: $\tfrac12\cdot\tfrac35+\tfrac12\cdot\tfrac15=\tfrac25$.
Practice more MCQs → 📝 Assignment · 35 marks