More Probability

The Axioms of Probability

The axioms of probability say what mathematical rules must be followed in assigning probabilities to events. Let P(A) denote the probability of the event A. The axioms are rules the function P must follow: 

The Axioms of Probability

  1. The probability of every event is at least zero.

    2. (For every event A, P(A) >= 0. There is no such thing as a negative probability.)
  2. The probability of the entire outcome space is 100%. 

    3. (P(S) = 100%. The chance that something in the outcome space occurs is 100%, because the outcome space contains every possible outcome.)
  3. If two events are disjoint, the probability that either happens is the sum of the probabilities that each happens. 

    4. (If AB = {}, P(AUB) = P(A) + P(B).)
Everything else that is mathematically true of probability is a consequence of these axioms, and of further definitions.

For example, we have the complement rule: 

The Complement Rule 

the probability that an event A does not happen is 100% minus the probability that A happens:

P(Ac) = 100% - P(A).
The complement rule can be derived from the axioms: the union of A and its complement is S (either A happens or it does not, and there is no other possibility), so P(AUAc) = P(S) = 100%, by axiom 2. The event A and its complement are disjoint (if "A does not happen" happens, A does not happen; if A happens, "A does not happen" does not happen), so P(AUAc) = P(A) + P(Ac) by axiom 3. Putting these together, we get P(A) + P(Ac) = 100%. If we subtract P(A) from both sides of this equation, we get what we sought: P(Ac) = 100%-P(A).

A special case of the complement rule is that

P({}) = 0%,

because P(S) = 100%, and Sc = {}.

An event A that has probability one is said to be certain or sure. S is certain.

The union of two events, A UB, can be broken up into three disjoint sets:

elements of A that are not in B (ABc)
elements of B that are not in A (AcB)
elements of both A and B (AB)

Together, these three sets contain every element of AUB. Therefore, the chance that either A or B occurs is

P(AUB) = P(ABc U AcB U AB ).

The three sets on the right are disjoint, so the third axiom implies that

P(AUB) = P(ABc) + P(AcB) + P(AB).

On the other hand,

P(A) = P(ABc U AB) = P(ABc) + P(AB),

because ABc and AB are disjoint. Similarly,

P(B) = P(AcB U AB) = P(AcB) + P(AB),

because AcB and AB are disjoint. Adding, we find

P(A) + P(B) = P(ABc)+ P(AcB) +2×P(AB).

This would be P(AUB), but for the fact that P(AB) is counted twice, not once. It follows that, in general,

P(AUB) = P(A) + P(B) - P(AB).

Again, while this is a true statement, it is not one of the axioms of probability. In the special case that AB = {}, this reduces to one of the axioms, because, as we saw in the preceding paragraph, P({}) = 0%. It follows that

P(AUB) <= P(A) + P(B),

because, by axiom 1, P(AB) >= 0.

Moreover, because taking a union can only include additional outcomes,

P(AUB) >= P(A), and

P(AUB) >= P(B).

Probability is analogous to area or volume or mass. Consider the unit square, which has length unity on each side. Its total area is 1 (= 100%). Let's call the square S, just like outcome space. Now consider regions inside the square S (subsets of S). The area of any such region is at least zero, the area of S is 100%, and the area of two regions is the sum of their areas, if they do not overlap (i.e., if their intersection is empty). These facts are direct analogues of the axioms of probability, and we shall often use this model to get intuition about probability.

A further analogy that I find useful is to consider the square S to be a dartboard. A trial or experiment consists of throwing a dart at the board once. The event A occurs if the dart sticks in the set A. The event AB occurs if the dart sticks in both A and B on that one toss. Clearly, AB cannot occur unless A and B overlap--the dart cannot stick in two places at once. AUB occurs if the dart sticks in either A or B (or both) on that one throw. A and B need not overlap for AUB to occur.

This analogy is also useful for thinking about logical implication. If A is a subset of B, the occurrence of A implies the occurrence of B; we shall sometimes say that A implies B. In the dartboard model, the dart cannot stick in A without sticking in B as well, so if A occurs, B must occur also. If A implies B, AB=A, so P(AB)=P(A). If AB = {}, A implies Bc and B implies Ac: if the dart sticks in A it did not stick in B, and vice versa. If A implies B, then if B does not occur A cannot occur either: Bc implies  Ac, so Bc is a subset of Ac.
 
 

The options in the next questions change only if you hold down the Shift key while you reload the page. If you reload the page without holding down the Shift key, the questions can be out of synch with the answers.


 

Conditional Probability

Conditioning means updating probabilities to incorporate new information. The conditional probability of A given B is the probability of the event A, updated on the basis of the knowledge that the event B occurred. Suppose that AB = {} (A and B are disjoint). Then if we learn that B occurred, we know A did not occur, so we should revise the probability of A to be zero (the conditional probability of A given B is zero). On the other hand, suppose that AB = B (B is a subset of A, so B implies A). Then if we learn that B occurred, we know A must have occurred as well, so we should revise the probability of A to be 100% (the conditional probability of A given B is 100%).

For in-between cases, the conditional probability of A given B is defined to be 

P(AB)
P(A|B) = ------------ ,
P(B)
provided P(B) is not zero (division by zero is undefined). "P(A|B)" is pronounced "the (conditional) probability of A given B." Why does this formula make sense? First of all, note that it does give back the intuitive answers we arrived at above: if AB = {}, then P(AB) = 0, so P(A|B) = 0/P(B) = 0; and if AB = B, P(A|B) = P(B)/P(B) = 100%. Similarly, if we learned that S occurred, this is not really new information (by definition, S always occurs, because it contains all possible outcomes), so we would like P(A|S) = P(A). This is how it works out: AS = A, so P(A|S) = P(A)/P(S) = P(A)/100% = P(A).

Now suppose that A and B are not disjoint. Then if we learn that B occurred, we can restrict attention to just those outcomes that are in B, and disregard the rest of S, so we have a new outcome space that is just B. We need P(B) = 100% to consider B an outcome space; we can make this happen by dividing all probabilities by P(B). For A to have occurred in addition to B requires that AB occurred, so the conditional probability of A given B is P(AB)/P(B), just as we defined it above.

Example. We deal two cards from a well shuffled deck. What is the conditional probability that the second card is an Ace (event A), given that the first card is an Ace (event B)? This is P(AB)/P(B) by definition. The (unconditional) chance that the first card is an Ace is 100%/13 = 7.7%, because there are 13 possible faces for the first card, and all are equally likely. The chance that both cards are Aces is as follows: from the four suits, we need to pick two; there are 4C2 = 6 ways that can happen. The total number of ways of picking two cards from the deck is 52C2 = 52×51/2 = 1326, so the chance that the two cards are both Aces is (6/1326)×100% = 0.5%. The conditional probability that the second card is an Ace given that the first card is an Ace is thus 0.5%/7.7% = 5.9%. As we might expect, it is somewhat lower than the chance that the first card is an Ace, because we know one of the Aces is gone. We could approach this more intuitively as well: given that the first card is an Ace, the second card is an Ace too if it is one of the three remaining Aces among the 51 remaining cards. These possibilities are equally likely if the deck was shuffled well, so the chance is 3/51 × 100% = 5.9%.

Independence

Two events are independent if learning that one occurred does not affect the chance that the other occurred. That is, if P(A|B) = P(A), and vice versa. A slightly more general way to write this is that A and B are independent if P(AB) = P(A) × P(B). (This covers the case that either P(A), P(B), or both, are equal to zero, while the definition in terms of conditional probability requires the probability in the denominator to be positive.) To reiterate: two events are independent if and only if the probability that both events happen simultaneously is the product of their unconditional probabilities. If two events are not independent, they are dependent.

Independence and Mutual Exclusivity are Different!

In fact, the only way two events can be both mutually exclusive and independent is if at least one of them has probability zero. If two events are mutually exclusive, learning that one of them happened tells us that the other did not happen. This is clearly informative: the conditional probability of the second event given the first is zero! This changes the (conditional) probability of the second event unless its (unconditional) probability was already zero.

Independent events bear a special relationship to each other. Independence is a very precise point between being disjoint (so that one event implies that the other did not occur), and one event being a subset of the other (so that one event implies the other).

Recap:

The following figure represents two events, A and B, as subsets of a rectangle. The probabilities of the events are proportional to their areas. Try dragging the events in the figure around to make them independent (that is, so that the area of their intersection is the product of their areas). Notice that it is not easy to do: to get the probability of the intersection equal to the product of the probabilities requires just the right amount of overlap.

You need Java to see this. 

If A and B are independent, so are

What kinds of events are independent? The outcomes of successive tosses of a fair coin, the outcomes of random draws from a box with replacement, etc. Draws without replacement are dependent, because what can happen on a given draw depends on what happens on previous draws.

Example: Suppose I have a box with four tickets in it, labeled 1, 2, 3, and 4. I stir the tickets and then pick one, stir them again without replacing the ticket I got, and pick another. Consider the event A = {I get the ticket labeled 1 on the first draw} and the event B = {I get the ticket labeled 2 on the second draw}. Are these events dependent or independent?

Solution: The chance that I get the 1 on the first draw is 25%. The chance that I get the 2 on the second draw is 25%. The chance that I get the 2 on the second draw given that I get the 1 on the first draw is 33%, which is much larger than the unconditional chance that I draw the 2 the second time. Thus A and B are dependent.

Now suppose that I replace the ticket I got on the first draw and stir the tickets again before drawing the second time. Then the chance that I get the 1 on the first draw is 25%, the chance that I get the 2 on the second draw is 25%, and the conditional chance that I get the 2 on the second draw given that I drew the 1 the first time is also 25%. A and B are thus independent if I draw with replacement.

Example: Two fair dice are rolled independently; one is blue, the other is red. What is the chance that the number of spots that show on the red die is less than the number of spots that show on the blue die?

Solution: The event that the number of spots that show on the red die is less than the number that show on the blue die can be broken up into mutually exclusive events, according to the number of spots that show on the blue die. The chance that the number of spots that show on the red die is less than the number that show on the blue die is the sum of the chances of those simpler events. If only one spot shows on the blue die, the number that show on the red die cannot be smaller, so the probability is zero. If two spots show on the blue die, the number that show on the red die is smaller if the red die shows exactly one spot. Because the number of spots that show on the blue and red dice are independent, the chance that the blue die shows two spots and the red die shows one spot is (1/6)(1/6) = 1/36. If three spots show on the blue die, the number that show on the red die is smaller if the red die shows one or two spots. The chance that the blue die shows three spots and the red die shows one or two spots is (1/6)(2/6) = 2/36. If four spots show on the blue die, the number that show on the red die is smaller if the red die shows one, two, or three spots; the chance that the blue die shows four spots and the red die shows one, two, or three spots is (1/6)(3/6) = 3/36. Proceeding similarly for the cases that the blue die shows five or six spots gives the ultimate result:

P(red die shows fewer spots than the blue die) = 1/36 + 2/36 + 3/36 + 4/36 + 5/36 = 15/36.

Alternatively, one could just count the ways: there are 36 possibilities, which can be written in a square table: 

  Blue Die
R
e
d

D
i
e

 1,1 1,2 1,3 1,4 1,5 1,6
2,1 2,2 2,3 2,4 2,5 2,6
3,1 3,2 3,3 3,4 3,5 3,6
4,1 4,2 4,3 4,4 4,5 4,6
5,1 5,2 5,3 5,4 5,5 5,6
6,1 6,2 6,3 6,4 6,5 6,6
The outcomes above the diagonal comprise the event whose probability we seek. There are 36 outcomes in all, of which 6 are on the diagonal. Half of the remaining 36-6=30 are above the diagonal; half of 30 is 15. The 36 outcomes are equally likely, so the chance is 15/36. The outcomes highlighted in yellow are one of the mutually exclusive pieces used in the computation just above: the three ways the red die can show a smaller number of spots than the blue die, when the blue die shows exactly 4 spots.
 
 

Hint: to solve this problem, you need to evaluate an expression of the form

1 - (1-x)n,

where x is nearly zero and n is very large. You can find the answer approximately using the following result:

(1-x)n = 1 + n×(-x) + (n×(n-1)/2)×(-x)2 +  . . .  + nCk×(-x)k +  . . .  + (-x)n.

The function (1-x)n is called a binomial; the fact that the coefficient of xk in the expansion of (1-x)n is nCk is the reason that nCk is sometimes called a binomial coefficient. When x is very small, x2, x3,  . . .  are much smaller still (and they get smaller faster than nCk   grows), so the terms involving higher powers of x than x1 are effectively negligable. That is, when x is nearly zero,

(1-x)n is approximately 1-n×x, so
1 - (1-x)n is approximately n×x.

Using that approximation is equivalent to ignoring the possibility that the sentence is typed more than once. The probability that the sentence is typed more than once is tiny compared to the chance that the sentence is typed exactly once, which is already quite small.
 

The Multiplication Rule

We can rearrange the definition of conditional probability to solve for the probability that both A and B occur (that AB occurs) in terms of the probability that B occurs and the conditional probability of A given B:

P(AB) = P(A|B)×P(B).

This is called the multiplication rule.

Example: A deck of cards is shuffled well, then two cards are drawn. What is the chance that both cards are aces?

P(card 1 is an Ace and card 2 is an Ace) = P(card 2 is an Ace | card 1 is an Ace)×P(card 1 is an Ace)
= 3/51 × 4/52 = 0.5%.

You can see that the multiplication rule can save you a lot of time!

Example: Suppose there is a 50% chance that you catch the 8:00am bus. If you catch the bus, you will be on time. If you miss the bus, there is a 70% chance that you will be late. What is the chance that you will be late?

P(late) = P(miss the bus and late)

= P(late|miss the bus) × P(miss the bus)

= 0.5 × 0.7 = 35%.

Example: Suppose that 10% of a given population has benign chronic flatulence. Suppose that there is a standard screening test for benign chronic flatulence that has a 90% chance of correctly detecting that one has the disease, and a 10% chance of a "false positive" (erroneously reporting that one has the disease when one does not). We pick a person at random from the population (so that everyone has the same chance of being picked) and test him/her. The test is positive. What is the chance that the person has the disease?

Solution: We shall combine several things we have learned. Let D be the event that the person has the disease, and T be the event that the person tests positive for the disease. The problem statement told us that:

The problem asks us to find P(D|T) = P(DT)/P(T). We shall find P(T) by breaking T into two mutually exclusive pieces, DT and DcT, corresponding to testing positive and having the disease (DT) and testing positive falsely (DcT). Then P(T) is the sum of P(DT) and P(DcT). We will find those two probabilities using the multiplication rule. We need P(DT) for the numerator, and it will be one of the terms in the denominator as well. The probability of DT is, by the multiplication rule,

P(DT) = P(T|D) × P(D) = 90% × 10% = 9%.

The probability of DcT is, by the multiplication rule and the complement rule,

P(DcT) = P(T|Dc) × P(Dc) = P(T|Dc) × (100%- P(D) ) = 10% × 90% = 9%.

By one of the axioms,

P(T) = P(DT) + P(DcT) = 9% + 9% = 18%,

because DT and DcT are mutually exclusive.Finally, plugging in the definition of P(D|T) gives

P(D|T) = P(DT)/P(T) = 9%/18% = 50%.

Because only a small fraction of the population actually have benign chronic flatulence, the chance that a positive test result for someone selected at random from the population is a false positive is 50%, even though the test is 90% accurate.

This problem illustrates Bayes' Rule:

P(A|B) = P(B|A) × P(A) / ( P(B|A)×P(A) + P(B|Ac) × P(Ac) ).

The numerator on the right is just P(AB), computed using the multiplication rule. The denominator is just P(B), computed by partitioning B into the mutually exclusive sets AB and AcB, and finding the probability of each of those pieces using the multiplication rule.

Bayes' Rule is useful to find the conditional probability of A given B in terms of the conditional probability of B given A, which is the more natural thing to measure in some problems. For example, in the disease-screening problem just above, the natural way to calibrate a test is to see how well it does at detecting a certain thing (e.g., a disease) when the thing is present, and to see how poorly it does at raising false alarms when the thing is not really present. These are, respectively, the conditional probability of detecting the thing given that the condition is present, and the conditional probability of incorrectly raising an alarm given that the thing is not present. However, the interesting quantity for an individual is the conditional chance that he or she has the disease, for example, given that the test raised an alarm.