Jointly distributed random variables

So far we have been only dealing with the probability distributions of single random variables.
However, we are often interested in probability statements concerning two or random variables. In
order to do this, we dene the joint (cumulative) distribution functions of these random variables.
Denition 1 Suppose that X and Y are two random variables. The joint (cumulative) distribution
function of X and Y is the function on R
2
dened by
F(x, y) = P(X x, Y y), (x, y) R
2
.
We can get the distribution function of X easily from the joint distribution function of X and Y :
F
X
(x) = P(X x) = P(X x, Y < ) = P( lim
y
{X x, Y y})
= lim
y
P(X x, Y y) = lim
y
F(x, y) = F(x, ).
Similarly we can get the distribution function of Y easily from the joint distribution function of X
and Y :
F
Y
(y) = lim
x
F(x, y) = F(, y).
The distribution functions F
X
and F
Y
are sometimes called the marginal distribution functions of
X and Y respectively.
The joint distribution function F of X and Y contains all the statistical information about X
and Y . In particular, given the joint distribution function F of X and Y , we can calculate the
probability of any event dened in terms of X and Y . For instance, for any real numbers a b
and c d, we have
P(a < X b, c < Y d) = F(b, d) F(a, d) F(b, c) +F(a, c).
In the case when X and Y are both discrete random variables, it is more convenient to use the
joint probability mass function of X and Y .
Denition 2 Suppose that X and Y are two discrete random variables. The joint probability mass
function of X and Y is the function on R
2
dened by
p(x, y) = P(X = x, Y = y), (x, y) R
2
.
The mass function p
X
of X can be easily obtained from the joint mass function of X and Y :
p
X
(x) = P(X = x) =

y:p(x,y)>0
p(x, y), x R.
1
Similarly,
p
Y
(y) = P(Y = y) =

x:p(x,y)>0
p(x, y), y R.
The mass functions p
X
and p
Y
are sometimes called the marginal mass functions of X and Y
respectively.
Example 3 A box contains 3 balls labeled 1, 2 and 3. 2 balls are randomly drawn from the box
without replacement. Let X be the number on the rst ball and Y the number on the second ball.
Then the joint mass function of X and Y is given by p(1, 2) = p(1, 3) = p(2, 1) = p(2, 3) = p(3, 1) =
p(3, 2) =
1
6
and p(x, y) = 0 elsewhere.
This joint mass function can be expressed by the following table
y
x
1 2 3
1 0 1/6 1/6
2 1/6 0 1/6
3 1/6 1/6 1/6
Example 4 Suppose that X and Y are two discrete random variables with joint mass function
given by the following table
y
x
1 2 3 4
1 1/4 1/8 1/16 1/16
2 1/16 1/16 1/4 1/8
Then p
X
(1) =
1
4
+
1
8
+
1
16
+
1
16
=
1
2
, and p
X
(2) = 1p
X
(1) =
1
2
. Similarly, p
Y
(1) =
1
4
+
1
16
=
5
16
,
p
Y
(2) =
3
16
, p
Y
(3) =
5
16
, p
Y
(4) =
3
16
.
Example 5 Suppose that X and Y are two discrete random variables with joint mass function
given by the following table
y
x
-1 0 2 6
-2 1/9 1/27 1/27 1/9
1 2/9 0 1/9 1/9
3 0 0 1/9 4/27
Find (a) P(Y is even); (b) P(XY is odd).
Solution (a) P(Y is even) = P(Y = 2) =
8
27
. (b) P(XY is odd) = P(X = 1, Y = 1) +
P(X = 1, Y = 3) =
2
9
.
Suppose that X and Y are two random variables. We say that X and Y are jointly absolutely
continuous if there is a nonnegative function f(x, y) on R
2
such that for any region C in R
2
,
P((X, Y ) C) =
__
(x,y)C
f(x, y)dxdy.
2
The function f(x, y) is called the joint probability density function of X and Y .
If A and B are subset of R, then as a special case of the display above, we have
P(X A, Y B) =
_
B
_
A
f(x, y)dxdy.
Because
F(x, y) = P(X x, Y y) =
_
y

_
x

f(u, v)dudv,
it follows, upon dierentiation, that
f(x, y) =

2
F
xy
(x, y).
If X and Y are jointly absolutely continuous, then both X and Y are absolutely continuous
and their density functions can be obtained as follows:
f
X
(x) =
_

f(x, y)dy, f
Y
(y) =
_

f(x, y)dx.
The proofs of these two formulae are exactly the same, we give give the proof of the rst one: for
any subset A of R,
P(X A) = P(X A, Y (, )) =
_
A
_

f(x, y)dydx.
The densities f
X
and f
Y
are sometimes called the marginal densities of X and Y respectively.
Example 6 Suppose that X and Y are jointly absolutely continuous with density function given
by
f(x, y) =
_
2e
x
e
2y
, 0 < x < , 0 < y <
0, otherwise.
Find (a) P(X > 1, Y < 1) (b) P(X < Y ).
Solution. (a)
P(X > 1, Y < 1) =
_

1
(
_
1
0
2e
x
e
2y
dy)dx =
_

1
e
x
dx
_
1
0
2e
2y
dy
= e
1
(1 e
2
).
(b)
P(X < Y ) =
_

0
(
_

x
2e
x
e
2y
dy)dx =
_

0
e
x
(
_

x
2e
2y
dy)dx
=
_

0
e
x
e
2x
dx =
_

0
e
3x
dx =
1
3
.
Example 7 A point (X, Y ) is randomly chosen from a disk of radius R centered at the origin so
that each point in the disk is equally likely. Then X and Y are jointly absolutely continuous with
joint density given by
f(x, y) =
_
1
R
2
x
2
+y
2
< R
2
0 otherwise.
Find the marginal densities of X and Y . Find the density of Z =

X
2
+Y
2
.
Solution.
3
We can also dene the joint distributions for n random variables in exactly the same way as
for we did for n = 2. For instance, the joint distribution function F of the n random variables
X
1
, . . . , X
n
is dened by
F(x
1
, . . . , x
n
) = P(X
1
x
1
, . . . , X
n
x
n
), (x
1
, . . . , x
n
) R
n
.
Further, the n random variables are said to be jointly absolutely continuous if there is a nonnegative
function f on R
n
such that for any region C is R
n
,
P((X
1
, . . . , X
n
) C) =
_
. . .
_
{x
1
,...,x
n
)C}
f(x
1
, . . . , x
n
)dx
1
. . . dx
n
.
In particular, for any n subsets A
1
, . . . , A
n
of R,
P(X
1
A
1
, . . . , X
n
A
n
) =
_
A
n
. . .
_
A
1
f(x
1
, . . . , x
n
)dx
1
. . . dx
n
.
One can similarly dene the joint mass function of n discrete random variables.
Independent random variables
Two random variables X and Y are said to be independent if for any any subsets A and B of
R,
P(X A, Y B) = P(X A)P(Y B).
It can be shown that the following results are true.
Theorem 8 Two random variables X and Y are independent if and only if
F(x, y) = F
X
(x)F
Y
(y), (x, y) R
2
.
Theorem 9 Two discrete random variables X and Y are independent if and only if
p(x, y) = p
X
(x)p
Y
(y), (x, y) R
2
.
Theorem 10 Suppose that X and Y are jointly absolutely continuous. Then X and Y are inde-
pendent if and only if
f(x, y) = f
X
(x)f
Y
(y), (x, y) R
2
.
It is easy to see that the random variables X and Y in Example 3, in Example 4 and in Example
5 are dependent, the random variables in Example 6 are independent, and the random variables in
Example 7 are dependent.
One can dene the independence of n > 2 random variables similar to the case n = 2. Similarly
we have the following
n random variables X
1
, . . . , X
n
are independent if
F
X
1
,...,X
n
(x
1
, . . . , x
n
) = F
X
1
(x
1
) F
X
n
(x
n
), (x
1
, . . . , x
n
) R
n
).
n discrete random variables X
1
, . . . , X
n
are independent if
p
X
1
,...,X
n
(x
1
, . . . , x
n
) = p
X
1
(x
1
) p
X
n
(x
n
), (x
1
, . . . , x
n
) R
n
).
n jointly absolutely continuous random variables X
1
, . . . , X
n
are independent if
f
X
1
,...,X
n
(x
1
, . . . , x
n
) = f
X
1
(x
1
) f
X
n
(x
n
), (x
1
, . . . , x
n
) R
n
).
4
Example 11 Suppose that n + m independent trials, having a common success probability p, are
performed. If X is the number of successes in the rst n trials, Y is the number of successes in the
last m trials and Z is the total number of successes, then X and Y are independent, but X and Z
are dependent.
Example 12 Suppose that the number of people entering a post oce on a given day is a Poisson
random variables with parameter . Suppose that each person entering the post oce is a male
with probability p and a female with probability 1 p. Let X be the number of males entering the
post oce on that day and Y the number of females entering the post oce on that day. Find the
joint mass function of X and Y .
Example 13 A man and a woman decides to meet at a certain location. If each person arrives
independently at a time uniformly distributed between noon and 1 pm, nd the probability that the
rst to arrive has to wait longer than 10 minutes.
Solution Using geometric considerations, we can easily nd the answer is
25
36
.
Example 14 Suppose that X and Y are independent geometric random variables with parameter
p. (a) Find the distribution of min(X, Y ). (b) Find P(Y X). (c) Find the distribution of X+Y .
(d) Find P(Y = y|X +Y = z) for z 2 and y = 1, . . . , z 1.
Solution. (a) For any positive integer z, we have
P(min(X, Y ) z) = P(X z, Y z) = P(X z)P(Y z) = (1 p)
2(z1)
.
Thus min(X, Y ) is a geometric random variable with parameter
1 (1 p)
2
= 2p p
2
.
(b)
P(Y X) =

x=1
P(X = x, Y X) =

x=1
P(X = x, Y x)
=

x=1
P(X = x)P(Y x) =

x=1
p(1 p)
x1
(1 p)
x1
= p

x=1
(1 p)
2(x1)
=
p
2p p
2
.
(c) Let z 2 be an integer. Then
P(X +Y = z) =
z1

x=1
P(X = x, X +Y = z) =
z1

x=1
P(X = x, Y = z x)
=
z1

x=1
P(X = x)P(Y = z x) =
z1

x=1
p(1 p)
x1
p(1 p)
zx1
= (z 1)p
2
(1 p)
z2
.
(d)
5
P(Y = y|X +Y = z) =
P(Y = y, X +Y = z)
P(X +Y = z)
=
P(X = z y, Y = y)
P(X +Y = z)
=
P(X = z y)P(Y = y)
P(X +Y = z)
=
p(1 p)
zy1
p(1 p)
y
1
(z 1)p
2
(1 p)
z2
=
1
z 1
.
Sums of Independent Random Variables
We often have to compute the distribution of X + Y from teh distributions of X and Y when
X and Y are independent. We start with the case when both X and Y are absolutely continuous.
Suppose that X and Y are independent absolutely continuous random variables with densities
f
X
and f
Y
respectively. Then the distribution of X +Y is
F
X+Y
(z) = P(X +Y z) =
_

_
zy

f
X
(x)f
Y
(y)dxdy
=
_

_
z

f
X
(u y)f
Y
(y)dudy
=
_
z

f
X
(u y)f
Y
(y)dydu.
Thus X +Y is also an absolutely continuous random variable and its density is given by
f
X+Y
(z) =
_

f
X
(z y)f
Y
(y)dy =
_

f
X
(x)f
Y
(z x)dx.
If, furthermore, X and Y are nonnegative, then
f
X+Y
(z) =
_
_
z
0
f
X
(x)f
Y
(z x)dx, z > 0
0, otherwise.
Example 15 Suppose that X and Y are independent random variables and both are uniformly
distributed on (0, 1). Find the density of X +Y .
Solution We could solve this problem by the formula above. Here I give a direct solution. By
geometric considerations, we can easily nd that for z (0, 1],
P(X +Y z) =
z
2
2
,
and for z (1, 2),
P(X +Y z) = 1
(2 z)
2
2
.
Thus
f
X+Y
(z) =
_

_
z, 0 < z 1,
2 z, 1 < z < 2,
0, otherwise.
6
Theorem 16 Suppose that X and Y are independent random variables, X is a Gamma random
variable with parameters (
1
, ) and Y is a Gamma random variable with parameters (
2
, ). Then
X +Y is a Gamma random variable with parameters (
1
+
2
, ).
Proof X and Y are both positive random variables and
f
X
(x) =
_

()
x

1
1
e
x
x > 0,
0 x 0,
f
Y
(y) =
_

()
y

1
1
e
y
y > 0,
0 y 0.
Thus, f
X+Y
(z) = 0 for z 0 and for z > 0,
f
X+Y
(z) =

1
+
2
e
z
(
1
)(
2
)
_
z
0
x

1
1
(z x)

2
1
dx.
Make the change of variable x = zu in the above integral we get
f
X+Y
(z) = c

1
+
2
z

1
+
2
1
e
z
, z > 0,
where
c =
_
1
0
u

1
1
(1 u)

2
1
du
(
1
)(
2
)
.
The proof is now nished.
Theorem 17 Suppose that X and Y are independent random variables, X is a normal random
variable with parameters (
1
,
2
1
) and Y is a normal random variable with parameters (
2
,
2
2
).
Then X +Y is a normal random variable with parameters (
1
+
2
,
2
1
+
2
2
).
Example 18 Suppose that X and Y are independent random variables and both of them are normal
random variables with parameters (0,
2
). Find the density of X
2
+Y
2
.
Solution X
2
and Y
2
are independent random variables and each is a Gamma random variable
with parameters (
1
2
,
1
2
2
). Thus by Theorem 16, we know that X
2
+Y
2
is a Gamma random variable
with parameters (1,
1
2
2
), which is the same as an exponential random variable with parameter
1
2
2
.
Now we deal with the case when X and Y are both discrete random variables.
Suppose that X and Y are independent discrete random variables with mass functions p
X
and
p
Y
respectively. Let x
1
, x
2
, . . . be the possible values of X. For any real number z,
P(X +Y = z) =

I
P(X = x
i
, X +Y = z) =

i
P(X = x
i
, Y = z x
i
)
=

i
P(X = x
i
)P(Y = z x
i
) =

i
p
X
(x
i
)p
Y
(z x
i
).
Thus the mass function of X +Y is given by
p
X+Y
(z) =

x
p
X
(x)p
Y
(z x).
7
If, furthermore, X and Y are integer valued random variables, then the mass function of X +Y is
given by
p
X+Y
(z) =
_

x=
p
X
(x)p
Y
(z x), when z is an integer
0, otherwise.
And if X and Y are nonnegative integer valued random variables, then the mass function of X+Y
is given by
p
X+Y
(z) =
_

z
x=0
p
X
(x)p
Y
(z x), z = 0, 1, . . .
0, otherwise.
Theorem 19 Suppose that X and Y are independent random variables, X is a binomial random
variable with parameters (n, p) and Y is a binomial random variable with parameters (m, p). Then
X +Y is a binomial random variable with parameters (n +m, p).
Theorem 20 Suppose that X and Y are independent random variables, X is a Poisson random
variable with parameter
1
and Y is a Poisson random variable with parameter
2
. Then X + Y
is a Poisson random variable with parameter
1
+
2
.
Expectations of random variables
Given the joint distribution of X and Y , we want to nd the expectation of a function of X
and Y . We will deal with this in two separate cases: when X and Y are both discrete or when X
and Y are jointly absolutely continuous.
Theorem 21 If X and Y are discrete random variables with joint mass function p(x, y), then for
any function on R
2
,
E[(X, Y )] =

y
(x, y)p(x, y).
If X and Y are jointly absolutely continuous with joint density f(x, y), then then for any function
on R
2
,
E[(X, Y )] =
_

(x, y)f(x, y)dxdy.

By taking the function to be (x, y) = x +y, we get the following result:
E[X +Y ] = E[X] +E[Y ].
Using induction we immediately get the following
E[
n

i=1
X
i
] =
n

i=1
E[X
i
].
By taking the function to be (x, y) = xy, we get the following result:
Theorem 22 If X and Y are independent random variables, then
E[XY ] = E[X] E[Y ].
8
More generally, we have the following:
Theorem 23 If X and Y are independent random variables, then for any functions and on
R,
E[(X)(Y )] = E[(X)] E[(Y )].
Just as the expectation and variance of a single random variable give us information about the
random variable, so does the covariance of two random variables give us information about the
relationship between the two random variables.
Denition 24 The covariance of two random variables X and Y , denoted by Cov(X, Y ), is dened
as
Cov(X, Y ) = E[(X E[X])(Y E[Y ])].
Expanding the right hand side of the above denition, we see that
Cov(X, Y ) = E [XY E[X]Y XE[Y ] +E[X]E[Y ]]
= E[XY ] E[X]E[Y ] E[X]E[Y ] +E[X]E[Y ]
= E[XY ] E[X]E[Y ].
Note that if X and Y are independent, then Cov(X, Y ) = 0. However, the converse is not true.
A simple example is as follows. Let X be a random variable such that
P(X = 0) = P(X = 1) = P(X = 1) =
1
3
and let Y be a random variable dened by
Y =
_
0, if X = 0
1, if X = 0.
X and Y are obviously dependent, and yet Cov(X, Y ) = 0.
One measure of degree of dependence between the random variables X and Y is the correlation
coecient (X, Y ) dened by
(X, Y ) =
Cov(X, Y )
_
Var(X)Var(Y )
.
The correlation coecient (X, Y ) is always between 1 and 1. (X, Y ) = 1 if and only if
P(X = aY +b) = 1 for some a > 0 and b R, and (X, Y ) = 1 if and only if P(X = aY +b) = 1
for some a < 0 and b R.
The following result list some properties of covariance.
Theorem 25 (i) Cov(X, Y ) = Cov(Y, X);
(ii) Cov(X, X) = Var(X);
(iii) Cov(aX, Y ) = aCov(X, Y );
(iv) Cov(

n
i=1
X
i
,

m
j=1
Y
j
) =

n
i=1

m
j=1
Cov(X
i
, Y
j
).
9
As a particular case of the last theorem we get
Var(
n

i=1
X
i
) =
n

i=1
n

j=1
Cov(X
i
, X
j
) =
n

i=1
Var(X
i
) +

i=j
Cov(X
i
, X
j
).
In particular, if X and Y are independent, then
Var(
n

i=1
X
i
) =
n

i=1
Var(X
i
).
Example 26 An accident occurs at a point X that is uniformly distributed on a road of length
L. At the time of the accident, an ambulance is at a location Y that is also uniformly distributed
on that road. Assuming that X and Y are independent, nd the expected distance between the
ambulance and the location of the accident.
Example 27 A box contains 3 balls labeled 1, 2, and 3. 2 balls are randomly selected from the box
without replacement. Let X be the number on the rst ball and let Y be the number on the second
ball. Find Cov(X, Y ).
Example 28 The joint density of X and Y is given by
f(x, y) =
_
1
4
e
y/2
, if 0 < x < y < ,
0, otherwise .
Find Cov(X, Y ).
Example 29 Suppose that X is a binomial random variable with parameters (n, p).
For i = 1, . . . , n, let X
i
= 1 if the i-th trial is a success and X
0
= 0 otherwise. Then X
1
, . . . , X
n
are independent Bernoulli random variables with parameter p, and X = X
1
+ + X
n
. Thus
E[X] = np and Var(X) = np(1 p).
Example 30 Independent trials, each results in a success with probability p, are performed. Let
X be the number trials needed to get a total of n successes.
Let X
1
be the number of trials needed to get the rst success. For i > 1, let X
i
be the number
of additional trials, after the (i 1)-th success, needed to get the i-th success. Then X
1
, . . . , X
n
are independent geometric random variables with parameter p. It follows from the denition that
X = X
1
+ +X
n
. Thus E[X] =
n
p
and Var(X) =
n(1p)
p
2
.
Example 31 Suppose that N people throw their hats into the center of the room. The hats are
mixed up, and each person randomly selects. Let X be the number of people that get their own hat.
Find E[X] and Var(X).
Solution For i = 1, . . . , N, let X
i
= 1 if the i-th person gets his own hat and X
i
= 0 otherwise.
Then X = X
1
+ + X
N
. For each i = 1, . . . , N, P(X
i
= 1) =
1
N
. Thus E[X] = 1. For i = j,
P(X
i
= 1, X
j
= 1) =
1
N(N1)
. Thus
Cov(X
i
, X
j
) =
1
N(N 1)

1
N
2
=
1
N
1
N(N 1)
.
10
Hence
Var(X) =
N

i=1
Var(X
i
) +

i=j
Cov(X
i
, X
j
)
= N
1
N
(1
1
N
) + (N
2
N)
1
N
1
N(N 1)
= 1
Example 32 If n balls are randomly selected (without replacement) from a box containing N balls
of which m are white. Let X be the number of white balls selected. Find E[X] and Var(X).
Solution For i = 1, . . . , m, let X
i
= 1 if the i-th white ball is among the selected and X
i
= 0
otherwise. Then X = X
1
+ +X
m
.
Now E[X
i
] = P(X
i
= 1) =
n
N
. Thus E[X] =
mn
N
. For i = j,
E[X
i
X
j
] = P(X
i
= 1, X
j
= 1) =
_
N 2
n 2
_
_
N
n
_ =
n(n 1)
N(N 1)
and
Cov(X
i
, X
j
) =
n(n 1)
N(N 1)

n
2
N
2
.
Therefore
Var(X) =
m

i=1
Var(X
i
) +

i=j
Cov(X
i
, X
j
)
= m
n
N
(1
n
N
) + (m
2
m)
_
n(n 1)
N(N 1)

n
2
N
2
_
= n
_
m
N
__
1
m
N
_
_
1
n 1
N 1
_
.
Example 33 Suppose that there are N dierent types of coupons, and each time one obtains a
coupon, it is equally likely to be any one of the N types. Find the expected number of coupons one
need to amass in order to get a complete set of at least one of each type.
Solution Let X be the number of coupons one need to amass in order to get a complete set of
at least one of each type. For i = 0, 1, . . . , N 1, let X
i
be the number of additional coupons that
need to be obtained after i distinct types have been collected in order to obtain another distinct type.
Then
X = X
0
+X
1
+ +X
N1
.
Obviously X
0
= 1, and for i = 1, . . . , N
1
, X
i
is a geometric random variable with parameter
(N i)/N and thus
E[X
i
] =
N
N i
.
Therefore
E[X] = 1 +
N
N 1
+ +
N
1
= N
_
1 + +
1
N 1
+
1
N
_
.
Remark on Notations
11