Mathematics 376 – Probability and Statistics II Problem Set 8 Solutions April 26, 2004

A)

1) From the linear model, ϵ_{i} = Y_{i} − β_{0} − β_{1}x_{i} is normally distributed (N(0, σ^{2})). Hence the pdf for ϵ_{i} is

f(ϵ) = \frac{1}{\sqrt{2πσ^{2}}} e^{−ϵ^{2}/2σ^{2}}

From observations (x_{1}, y_{1}), …, (x_{n}, y_{n}) of Y, the likelihood of these observations (as a function of β_{0} and β_{1}) is given by

\begin{aligned}L(β_{0}, β_{1}) & = (\frac{1}{\sqrt{2πσ^{2}}} )^{n} exp ((− ∑_{i=1}^{n} ϵ_{i}^{2}) \frac{1}{2σ^{2}} ) \\ & = (\frac{1}{\sqrt{2πσ^{2}}} )^{n} exp ((− ∑_{i=1}^{n}(y_{i} − β_{0} − β_{1}x_{i})^{2}) \frac{1}{2σ^{2}} ) \end{aligned}

2) Continuing from the last line, we have

ln(L) = \frac{−n}{2} ln(2πσ^{2}) − \frac{1}{2σ^{2}} ∑_{i=1}^{n}(y_{i} − β_{0} − β_{1}x_{i})^{2}

Hence

\frac{∂ ln(L)}{∂β_{0}} = \frac{−1}{2σ^{2}} ∑_{i=1}^{n} 2(y_{i} − β_{0} − β_{1}x)(−1)

= \frac{1}{σ^{2}} ((Σy_{i}) − nβ_{0} − (Σx_{i})β_{1})

\frac{∂ ln(L)}{∂β_{1}} = \frac{−1}{2σ^{2}} ∑_{i=1}^{n} 2(y_{i} − β_{0} − β_{1}x)(−x_{i})

= \frac{1}{σ^{2}} ((Σx_{i}y_{i}) − (Σx_{i})β_{0} − (Σx_{i}^{2})β_{1})

Setting \frac{∂ ln(L)}{∂β_{0}} = \frac{∂ ln(L)}{∂β_{1}} = 0 and multiplying by σ^{2}, we get exactly the same normal equations as from the least-squares approach:

\begin{aligned}nβ_{0} + (Σx_{i})β_{1} & = Σy_{i} \\ (Σx_{i})β_{0} + (Σx_{i}^{2})β_{1} & = Σx_{i}y_{i} \end{aligned}

Thus, the maximum likelihood estimators for β_{0}, β_{1} are the same as the least-squares estimators.

3) To check that (β̂_{0}, β̂_{1}) gives a maximum of the likelihood, we continue on to the second derivative test:

\frac{∂^{2} ln(L)}{∂β_{0}^{2}} = \frac{−n}{σ^{2}}
\frac{∂^{2} ln(L)}{∂β_{1}^{2}} = \frac{−(Σx_{i}^{2})}{σ^{2}}
\frac{∂^{2} ln(L)}{∂β_{1}∂β_{0}} = \frac{−(Σx_{i})}{σ^{2}}

The Hessian determinant is

H = \frac{∂^{2} ln(L)}{∂β_{0}^{2}} \frac{∂^{2} ln(L)}{∂β_{1}^{2}} − (\frac{∂^{2} ln(L)}{∂β_{1}∂β_{0}})^{2} = \frac{n(Σx_{i}^{2}) − (Σx_{i})^{2}}{σ^{4}}

The numerator here is nS_{xx} > 0 (see B below). Hence H > 0 at all points (including the critical point found in part 2). Furthermore, \frac{∂^{2} ln(L)}{∂β_{0}^{2}} = \frac{−n}{σ^{2}} < 0, so (β̂_{0}, β̂_{1}) is a local (and global) maximum.

Comment: If you missed some points on this problem, be sure to review the second derivative test for functions of two variables again and be ready to use it correctly on the final exam(!)

B)

1) We have

S_{xx} = Σ(x_{i} − \overline{x})^{2}
= (Σx_{i}^{2}) − 2\overline{x}(Σx_{i}) + n\overline{x}^{2}
= (Σx_{i}^{2}) − 2n\overline{x}^{2} + n\overline{x}^{2}
= (Σx_{i}^{2}) − n\overline{x}^{2}

Similarly,

S_{yy} = Σ(x_{i} − \overline{x})(y_{i} − \overline{y})
= (Σx_{i}y_{i}) − \overline{x}(Σy_{i}) − \overline{y}(Σx_{i}) + n\overline{x}\overline{y}
= (Σx_{i}y_{i}) − 2n\overline{x}\overline{y} + n\overline{x}\overline{y}
= (Σx_{i}y_{i}) − n\overline{x}\overline{y}

Hence

\frac{S_{xy}}{S_{xx}} = \frac{(Σx_{i}y_{i}) − n\overline{x}\overline{y}}{(Σx_{i}^{2}) − n\overline{x}^{2}}

= \frac{n(Σx_{i}y_{i}) − n^{2}\overline{x}\overline{y}}{n(Σx_{i}^{2}) − n^{2}\overline{x}^{2}}

= \frac{n(Σx_{i}y_{i}) − (Σx_{i})(Σy_{i})}{n(Σx_{i}^{2}) − (Σx_{i})^{2}}

= β̂_{1} 2) Then we have

\overline{y} − \overline{x}β̂_{1} = \overline{y} − \overline{x}\frac{n(Σx_{i}y_{i}) − (Σx_{i})(Σy_{i})}{n(Σx_{i}^{2}) − (Σx_{i})^{2}}

= \frac{\overline{y}(n(Σx_{i}^{2}) − (Σx_{i})^{2}) − \overline{x}(n(Σx_{i}y_{i}) − (Σx_{i})(Σy_{i}))}{n(Σx_{i}^{2}) − (Σx_{i})^{2}}

= \frac{\frac{1}{n}(Σy_{i})(n(Σx_{i}^{2}) − (Σx_{i})^{2}) − \frac{1}{n}(Σx_{i})(n(Σx_{i}y_{i}) − (Σx_{i})(Σy_{i}))}{n(Σx_{i}^{2}) − (Σx_{i})^{2}}

= \frac{(Σy_{i})(Σx_{i}^{2}) − (Σx_{i})(Σx_{i}y_{i})}{n(Σx_{i}^{2}) − (Σx_{i})^{2}}

= β̂_{0}

11/6. We have SSE = Σ(y_{i} − β_{1}x_{i})^{2}, so to minimize, we compute \frac{∂SSE}{∂β_{1}} and set equal to zero:

0 = 2Σ(y_{i} − β_{1}x_{i})(−x_{i})
= 2(Σx_{i}y_{i}) − 2β_{1}(Σx_{i}^{2})
⇒ β_{1}(Σx_{i}^{2}) = Σx_{i}y_{i}

Solving for β_{1}, we obtain the least-squares estimator:

β̂_{1} = \frac{Σx_{i}y_{i}}{Σx_{i}^{2}}

Note that this is different from β̂_{1} in the case that the regression line is not constrained to pass through (0,0).

11/11. We substitute β̂_{0} = \overline{y} − β̂_{1}\overline{x} into the formula for SSE:

SSE = Σ(y_{i} − β̂_{0} − β̂_{1}x_{i})^{2}
= Σ ((y_{i} − \overline{y}) − β̂_{1}(x_{i} − \overline{x}))^{2}
= S_{yy} − 2β̂_{1}S_{xy} + β̂_{1}^{2}S_{xx}

But,

β̂_{1}^{2}S_{xx} = β̂_{1} (\frac{S_{xy}}{S_{xx}} ) S_{xx} = β̂_{1}S_{xy}

So the last two terms in the above can be combined to yield:

SSE = S_{yy} − β̂_{1}S_{xy}

which is what we wanted to show.

For the other problems, see the following Maple session.