\chapter{Descriptive Statistics}

\specialsection{Introduction}
In the previous chapters, we have concentrated on modeling.  In this section of the text, 
we will introduce a number of additional techniques for analyzing the information contained
in datasets.  We will use these even more extensively in later chapters in learning how to make inferences from data.  
Some of the basic questions involved in analyzing the distribution of values in a dataset are:
\begin{itemize}
\item Where is the \emph{center, or middle} of the data?
\item How \emph{spread out} is the data?
\end{itemize}
Some important types of inferences we will learn about making are:
\begin{itemize}
\item Is a particular data value an \emph{outlier}, and is it so unrepresentative that we might
\index{outlier}
be justified in treating it separately or even omitting it from consideration?
\item If we select a sample
\index{sample}
 from some population of organisms and make some
measurement on each individual (for example a body length or weight), can
we estimate the average over the whole population, or perhaps a likely interval of values  
for the average of that measurement on the whole population?
\item If two populations are compared by drawing samples and computing some statistic
from measured values, is there a \emph{demonstrable difference} between the populations
(that is, a difference that is unlikely to come just from random variations stemming from 
the process of selecting the samples)?
\end{itemize}
In this chapter, we will present some first \emph{descriptive statistics}
\index{statistics!descriptive|(}
\index{descriptive statistics|(}
 that can 
be used to understand the distribution of a set of data values and that are the 
basis for the sorts of inference procedures we will address later.   In general, by
\index{statistic (defined)}
a \emph{statistic}, we will mean a numerical value computed in some definite way 
from the data values.   We have already seen some first steps in 
this kind of thinking in our discussion of least squares regression from Chapter 4,
\index{linear regression}
and also in our treatment of fitting power law distributions from Chapter 6.
\index{distribution!power law}
\index{power law distributions}

Recall that in Chapter 4 we discussed how the straight line best fitting a bivariate dataset 
$(x_i,y_i)$ for $i = 1, \ldots, N$ could be found by solving the \emph{normal equations}
\index{normal equations}
\index{linear regression}
from \eqref{eq:NormalEqs}:
\begin{align*}
\left(\sum_{i=1}^N x_i^2\right)\cdot m + \left(\sum_{i=1}^N x_i\right)\cdot b & =  \sum_{i=1}^N x_i y_i \\ 
\left(\sum_{i=1}^N x_i\right)\cdot m + N\cdot b & =  \sum_{i=1}^N y_i .
\end{align*}
If we divide through by $N$ in the second equation, we obtain:
$$\left(\frac{1}{N} \sum_{i=1}^N x_i\right) \cdot m + b = \frac{1}{N}\sum_{i=1}^N y_i.$$
You probably recognize the 
$$\frac{1}{N} \sum_{i=1}^N x_i, \quad {\rm and} \quad  \frac{1}{N}\sum_{i=1}^N y_i$$
as the \emph{numerical averages} of the $x_i$ and the $y_i$.  In this chapter we will 
introduce the notation
$$\overline{x} =  \frac{1}{N}\sum_{i=1}^N x_i$$
\index{mean}
\index{descriptive statistics!mean}
for the average, or \emph{mean}, of the $x_i$ and we will see how it is used as one
sort of way to estimate the \emph{center, or middle} of a distribution of data values.  
If we do this for the $y_i$ as well, we get a mean $\overline{y}$ of the $y_i$, 
and then the second normal equation just says $m \cdot \overline{x} + b = \overline{y}$.
In particular, the \emph{point of averages} $(\overline{x},\overline{y})$ lies on the 
best-fit regression line $y = \hat{m}\cdot x + \hat{b}$ when we solve the normal 
equations to find the slope and intercept $\hat{m}, \hat{b}$.  

In earlier chapters, we also discussed (in a quite informal way) how the $R^2$ statistic could be used to 
evaluate the goodness of fit of a regression line.  We did not discuss how the $R^2$
\index{R@$R^2$-statistic}
statistic was computed before, but we can now with the concepts introduced in the
previous paragraph.  In the case that we are modeling $y$ as a linear function of an
exactly known \emph{explanatory} variable $x$, then the $R^2$ statistic is computed with this formula
\begin{equation}
\label{eq:CC}
R^2 = \frac{\displaystyle{\left(\sum_{i=1}^N (x_i - \overline{x}) \cdot (y_i - \overline{y})\right)^2}}{\displaystyle{\sum_{i=1}^N (x_i - \overline{x})^2 \cdot \sum_{i=1}^N (y_i - \overline{y})^2}},
\end{equation}
or by one of several possible rearranged forms.  Taking square roots gives the so-called \emph{correlation coefficient}, $R$.
\index{correlation coefficient}
\index{descriptive statistics!correlation coefficient}

In this chapter we will  introduce a number of other techniques for analyzing data sets with different properties
and build up to understanding the rationale behind the correlation coefficient (presented in a final, optional section).

\specialsection{Measures of the Center of a Data Set}
We will begin with the question of how to estimate what we would think of intuitively as 
the middle, or center, of a collection of data values.  
There are two commonly-used statistics for this. 

The first statistic we will discuss is 
\index{mean|(}
\index{descriptive statistics!mean|(}
the \emph{mean}, or numerical average
of the values $x_i$, $i = 1, ..., N$, defined as follows:
\begin{equation}
\label{eq:Mean}
\framebox{$\displaystyle{\overline{x} = \frac{1}{N}\sum_{i=1}^N x_i = \frac{1}{N}(x_1 + \cdots + x_N).}$}
\end{equation}
The mean is often used, for instance, to compute things like class averages on tests or other assignments.

Another way to say where the middle of a distribution of data
\index{descriptive statistics!median|(}
\index{median|(}
values is by means of another statistic you have probably encountered, called the \emph{median}.
Roughly speaking, the median is a number such that half the data values are greater than or equal
to that number and half the data values are less than or equal to that number.
To simplify computing the median by hand, we sort the data values 
into non-decreasing order:
\begin{equation}
\label{eq:NonDec}
x_1 \le x_2 \le x_3 \le \cdots \le x_{N-1} \le x_N.
\end{equation}
Then we can find the median just by looking for the ``middle value.''

 If $N = 2K + 1$ is an \emph{odd
number} then there is a value right in the middle of the list, which is then the median because
of the way the data values have been listed.   
This is the value $x_{K+1}$ since there are $K$ values $x_1, \ldots, x_K$ 
less than or equal to $x_{K+1}$ and also $K$ values $x_{K+2}, \ldots, x_{2K+1}$ values
greater than or equal to $x_{K+1}$.   

On the other hand, if $N = 2K$ is an \emph{even number}, then there is no value 
exactly in the middle of the list, and the median is typically taken to be the 
average or mean of the two data values closest to the middle of the list:
$\frac{x_{K} + x_{K+1}}{2}$.  

Putting together these two cases, if the data is arranged in non-decreasing order
as in \eqref{eq:NonDec}, then 
\begin{equation}
\label{eq:Median}
\framebox{
${\rm median} = \begin{cases} 
                         x_{K+1} & {\rm if\ } N = 2K + 1 {\rm\ is\ odd}\\
                         & {\rm and}\\
                         \displaystyle{\frac{x_K + x_{K+1}}{2}} & {\rm if\ } N = 2K {\rm\ is\ even.}
                         \end{cases}$
                         }
\end{equation} 

(You can remember this as a formula if you 
think: take the middle of the list when $N$ is odd and the average of the end of the first half of the list -- the $x_K$ -- and the beginning
of the second half of the list -- the $x_{K+1}$ -- when $N$ is even.)

\begin{example}
\label{MeanMedian}
To keep things simple, let us consider two hypothetical data sets.  

(a)  First, consider these $x_i$:
$$5.2  \quad 8.9 \quad 2.1 \quad 7.5 \quad 9.3 \quad 4.3 \quad 4.7,$$
where $N = 7$, an odd number.   The mean is given by 
$$\overline{x} = \frac{5.2 + 8.9 + 2.1 + 7.5 + 9.3 + 4.3 + 4.7}{7} = \frac{42.0}{7} = 6.0.$$
If we rearrange to non-decreasing order as in \eqref{eq:NonDec}, we have
$$2.1 \le 4.3 \le 4.7 \le 5.2 \le 7.5 \le 8.9 \le 9.3$$
and we see the middle number -- the median -- is $5.2$.  

(b)  Second, consider these $x_i$:
 $$5.2  \quad 8.9 \quad 2.1 \quad 7.5 \quad 9.3 \quad 4.3.$$
 This is the same as the previous list, but omitting the $4.7$, so now $N = 6$.  We have
 $$\overline{x} = \frac{5.2 + 8.9 + 2.1 + 7.5 + 9.3 + 4.3}{7} = \frac{37.3}{6} \doteq 6.2.$$
 To find the median, we omit the 4.7 from the sorted list:
 $$2.1 \le 4.3 \le 5.2 \le 7.5 \le 8.9 \le 9.3$$
 and we average the number at the end of the first half and the number at the beginning
 of the second half:
 $${\rm median} = \frac{5.2 + 7.5}{2} = 6.35.$$
 We will discuss some patterns visible in these examples next.
$\triangle$
\end{example}

A first observation is that in both cases the mean and the median \emph{are different}.
This is to be expected because these two statistics give a measure of the middle of a 
collection of data values in two different ways.  

A physical analogy may help to make what the \emph{mean} does more vivid.  
Suppose we have a collection of \emph{equal}, say 1 kg, masses located along
a beam.  We introduce a reference point and measure distances from that
point in either direction using some distance unit, say meters.  Then the location
of each mass is described by some real number $x_i$ and the reference point is 
located at the point corresponding to the number $0$.  Reversing the process, given any data set, we
can also visualize the numbers involved as a collection of positions of a collection of 
equal masses in this way.  

Then, by (a special case of) the \emph{law of the lever} introduced by 
\index{mean!and law of the lever}
Archimedes,\footnote{In his \emph{Equilibrium of Planes}, Propositions 6 and 7.
\index{Archimedes (Greek mathematician and physicist)}
This is also the background for the much-discussed boasting quotation traditionally ascribed
to Archimedes:  ``Give me (a lever and) a place to stand and I will move the Earth!''}
the configurations will \emph{balance exactly} if the beam is placed on a 
fulcrum at the point corresponding to the mean, $\overline{x}$.  
For instance, in Example~\ref{MeanMedian} (a), the collection of equal masses
would balance with a fulcrum at $\overline{x} = 6.0$, and that balance
point shifts to $\overline{x} = 6.2$ when the mass at location $x_i = 4.7$ removed.
In effect, the balance point, \emph{the mean, shifts to the right} (i.e. to a larger numerical
value) because we have taken away
one of the masses lying to the left of (i.e. smaller than) the original balance point.   The same kind of 
shifting of the mean to the right would occur if we moved any one of the numbers
in (a) farther to the right.  For instance, increasing the largest number from $9.3$ to $10.3$
shifts the mean to $\frac{43.0}{7} \doteq 6.15$.    On the other hand, removing a value
lying to the right of (i.e. larger than) the mean, or shifting one or more of the values
to the left (i.e. to a smaller value) would make the mean smaller.  

The description of the median leading up to the recipe for computing it in 
\eqref{eq:Median} shows that the median estimates the middle 
in a quite different way.  In particular, notice that moving a data value to the
left or right can often have no effect at all on the median of a data set.  
For instance, if we shifted the value $9.3$ to $10.3$ in the data set from part (a)
of the example, the position of the median would not change at all; we would
still have median $= 5.2$.  Similarly, if we shifted any of the four numbers 
not involved in the computation of the median in part (b), but we left the
ordering unchanged, there would be no effect on the median. 
In this sense, the median is more \emph{robust} than the mean -- it is less affected 
by small changes in the data values.  

Putting these observations together, we can see that the mean takes the individual
values in the data set more into account than the median does, while the median
has more to do with the relative ordering of the data values.  There are situations
where each of these ways of finding the middle is more appropriate, and we might
even want to use both of them in combination(!)

\begin{example}
\label{MeanMedianCont}
In part (a) of Example~\ref{MeanMedian}, we might say that the distribution is \emph{skewed}
\index{skewness} towards larger 
values, meaning that the larger values are farther from the median (on the right side) than the smaller 
values are from the median (on the left side).  We can see this, for instance by 
comparing the difference $9.3 - 5.2 = 4.1$ (from the largest value) with the difference 
$5.2 - 2.1 = 3.1$ (from the smallest value) and so on.   And in fact the median is \emph{smaller than the mean}
because those larger values pull the mean in that direction.

  On the other hand in (b) of the example,  the median is \emph{larger 
than the mean}.  When that happens it is sometimes claimed that the data values
should be \emph{skewed} towards smaller values.   This is partly true in this case if we 
compare the difference $9.3 - 6.35 = 2.95$ (from the largest value) with the
difference $6.35 - 2.1 = 4.25$ (from the smallest value).  However the pattern
is not quite as consistent as in part (a).  For instance, the second largest value $8.9$ is actually
farther from the median than the second smallest value $4.3$ is.   $\triangle$
\end{example}

We will discuss a way to quantify the notion of skewness of a distribution later in the
chapter and we will return to these examples to illustrate that the connection between skewness
and the relative positions of the median and mean is more complicated than is sometimes claimed(!)
\index{mean|)}
\index{descriptive statistics!mean|)}
\index{median|)}
\index{descriptive statistics!median|)}

\specialsection{Measures of Spread and Distribution}
\subsection*{The Range} One of the most basic notions of how spread-out a data set is simply the 
numerical range of values represented in the data.  This can be measured
by computing the largest and smallest values, say $max$ and $min$, and
taking the difference
\index{range}
\index{descriptive statistics!range}
\begin{equation}
\label{eq:Range}
\framebox{${\rm range} = {\it max} - {\it min}$}
\end{equation}

\begin{example}
\label{Range}
The range of the datasets in both parts of Example~\ref{MeanMedian} is
$${\rm range} = 9.3 - 2.1 = 7.2.$$
Note that the other five or six data values do not enter here at all.  $\triangle$
\end{example}

\subsection*{The Standard Deviation.}
To capture the difference between a data set like the one from part (a)
of Example~\ref{MeanMedian}
$$5.2  \quad 8.9 \quad 2.1 \quad 7.5 \quad 9.3 \quad 4.3 \quad 4.7$$
with $\overline{x} = 6.0$, and another like 
$$6.2  \quad 7.9 \quad 2.1 \quad 6.5 \quad 9.3 \quad 4.8 \quad 5.2,$$
which also has $\overline{x} = 6.0$ and range $9.3 - 2.1 = 7.2,$
\index{standard deviation (SD)|(}
\index{descriptive statistics!standard deviation (SD)|(}
we can use another statistic known as the \emph{standard deviation}, or SD for short, 
usually denoted by $s$.
By definition, the SD is the \emph{root mean square deviation from the mean}.  
To see what this means (and see how to compute $s$ by hand), we
will break this down.  This will also show why $s$ is a reasonable measure of spread.
\begin{itemize}
\item The \emph{deviation} of a data value $x_i$ from the mean $\overline{x}$ is just $x_i - \overline{x}$.
\item The \emph{square deviation} is $(x_i - \overline{x})^2$.  (The effect of squaring is to make larger deviations
count more and smaller ones count less, while disregarding whether the deviation is positive  $x_i > \overline{x}$) or negative
($x_i < \overline{x}$).)
\item The \emph{mean square deviation} is essentially the mean of the square deviations.  And in fact some
(mostly older) treatments of statistics would use exactly the mean of the square deviations, in the form
$$\frac{1}{N} \sum_{i=1}^N (x_i - \overline{x})^2.$$
However, for technical reasons,\footnote{Technical note:  This is done to obtain an \emph{unbiased estimator}
for the population standard deviation when we sample from a population with a known distribution.} we will use a slightly different version with the $N$ in the 
denominator replaced by $N - 1$.  
$$\frac{1}{N-1} \sum_{i=1}^N (x_i - \overline{x})^2.$$
\item Finally, the \emph{root mean square deviation} means to take the square root of the result of the previous step.
\end{itemize}
Putting everything together, we get the following formula for the standard deviation:
\begin{equation}
\label{eq:SD}
\framebox{$s = \displaystyle{\sqrt{\frac{1}{N-1}\sum_{i=1}^N (x_i - \overline{x})^2}}$.}
\end{equation}

One reason for taking the square root is so that we obtain a quantity that is measured \emph{in the same 
units} as the original data values $x_i$.  Note, for instance, that if $x_i$ were distances in units of meters ($m$), then
the square deviations $(x_i - \overline{x})^2$ would have units of $m^2$.  To compare our measure
of spread with the values of the data, it is more convenient to have something in the same units, and
because of the square root, $s$ would have units of $m$ again.

There is also a rearranged form of this equation that is somewhat more efficient for hand computation.
We will be relying on software such as Excel for these computations, though, so we will address this
\index{spreadsheet}
only in the exercises.  

\begin{example}
\label{SDEx}
Consider the hypothetical data sets
$$5.2  \quad 8.9 \quad 2.1 \quad 7.5 \quad 9.3 \quad 4.3 \quad 4.7$$
and
$$6.2  \quad 7.9 \quad 2.1 \quad 6.5 \quad 9.3 \quad 4.8 \quad 5.2$$
given above.  Both have $\overline{x} = 6.0$ and both have the same range. 
We now compute the standard deviations.   For the first one we compute the 
square deviations:
\begin{align*}
(5.2 - 6.0)^2 &= 0.64, (8.9 - 6.0)^2 = 8.41, (2.1 - 6.0)^2 = 15.21, (7.5 - 6.0)^2 = 2.25\\
(9.3 - 6.0)^2 &= 10.89, (4.3 - 6.0)^2 = 2.89, (4.7 - 6.0)^2 = 1.69.
\end{align*}
Hence the mean square deviation (using a denominator $N - 1 = 6$) is 
$$\frac{0.64 + 8.41 + 15.21 + 2.25 + 10.89 + 2.89 + 1.69}{6} \doteq 6.997$$
and hence
$$s \doteq \sqrt{6.20} \doteq 2.645.$$

Similarly, for the second data set we compute
\begin{align*}
(6.2 - 6.0)^2 &= 0.04, (7.9 - 6.0)^2 = 3.61, (2.1 - 6.0)^2 = 15.21, (6.5 - 6.0)^2 = 0.25\\
(9.3 - 6.0)^2 &= 10.89, (4.8 - 6.0)^2 = 1.44, (5.2 - 6.0)^2 = 0.64.
\end{align*}
Hence the mean square deviation (using a denominator $N - 1 = 6$) is 
$$\frac{0.04 + 3.61 + 15.21 + 0.25 + 10.89 + 1.44 + 0.64}{6} \doteq 5.347$$
and hence
$$s \doteq \sqrt{5.347} \doteq 2.312.$$
The values in the second data set  (other than the maximum and the minimum, which are the same
in both data sets) are somewhat more closely ``clumped'' together around
the mean.  This can be seen, for instance, by locating the points on a number line.
The smaller value of $s$ for the second data set is an indication of 
this feature of the two distributions.  $\triangle$
\end{example}

Once we have the mean $\overline{x}$ and the standard deviation $s$, we can use
those numbers to define a collection of standard intervals that are often used
to describe the distribution of the data values.  For instance, the interval 
defined by $\overline{x} - s \le x \le \overline{x} + s$ is called the interval
of $x$ values \emph{within one standard deviation of the mean}.  Similarly,
the interval $\overline{x} - 2\cdot s \le x \le \overline{x} + 2\cdot s$ consists
of $x$-values \emph{within two standard deviations of the mean}.  

\begin{example}
\label{SDExCont}
For instance, with the first data set from Example~\ref{SDEx}, we have
$\overline{x} = 6.0$ and $s \doteq 2.645$.  So the interval of values
within two standard deviations of the mean is the interval of $x$ with 
$$0.71 = 6.0 - 2\cdot 2.645 \le x \le 6.0 + 2\cdot 2.645 = 11.29.$$
Note that \emph{all} of the data values are in this interval in this case.  $\triangle$
\end{example}
\index{standard deviation (SD)|)}
\index{descriptive statistics!standard deviation (SD)|)}

\subsection*{Chebyshev's Theorem and One Criterion for Outliers}
There is a theoretical result known as Chebyshev's theorem\footnote{Named after the 
Russian mathematician Pafnuty Chebyshev, 1821-1894.  Because this is a Russian 
\index{Chebyshev, Pafnuty}
name, it was originally written in the Cyrillic alphabet.  It has been transliterated to the Roman alphabet
to give an English equivalent.  There are several different systems for these
transliterations and you will find many different-looking forms of the name
if you look at different sources:  Chebysheff, Tchebyshev, etc.} that states that \emph{in every data set}, 
\emph{at least} 
$$\left(1 - \frac{1}{n^2}\right) \cdot 100\%$$
of the data values will be in the interval
\index{Chebysev's theorem}
within $n$ standard deviations of the mean.  With $n = 2$, for instance, this gives 
at least $\left(1 - \frac{1}{4}\right) \cdot 100\% = 75\%$
within two standard deviations of the mean.   Thus it is perhaps not surprising that 
a large fraction of the data values ended up in this interval for our example data set.  
Similarly, with $n = 3$, the interval of values within \emph{three} standard deviations of the 
mean will always contain at least $\left(1 - \frac{1}{9}\right) \cdot 100\% \doteq 89\%$
of the data values.   The statement of Chebyshev's theorem also gives a sort of indication why the 
\emph{standard deviation} has that name:  there are uniform statements one can
make about the distribution of the values in all data sets using the values of $s$ 
(and $\overline{x}$)!

These estimates from Chebyshev's theorem are, in a sense, ``worst case''
lower bounds.  Under some fairly general conditions, it can be shown that
this fraction of data values within two or three standard deviations is actually much larger -- 
often more like $95\%$ within 2 SDs and $99\%$ within 3 SDs.\footnote{Technical note:  This is true for instance
if the data is \emph{normally distributed}, as we will see later.}  This is the basis for one rule of thumb that is often used
to identify \emph{outliers} in a data set.  This criterion
\index{outlier!$3\sigma$ rule for} 
is to consider any value that lies more than three standard deviations from the mean (in 
either direction) to be an outlier.   This criterion is sometimes called
the ``$3\sigma$ rule'' since the Greek letter $\sigma$ (``sigma'') is often used to represent a parameter
of theoretical distributions analogous to the data set SD, which we have denoted by $s$. 
The question then becomes, what should be done with outliers?   Simply removing
them and proceeding as if those values were never included in the data is generally considered to be a
``shady'' practice because it is essentially hiding information that might in fact
be valid and might lead to different conclusions.  That sort of ``cherry-picking'' is 
not an honest use of data but it is done from time to time; statistics can definitely be \emph{misused!}
Labeling the questionable data values as 
outliers but still considering their effects in the analysis is considered
better statistical practice in most cases.  

\subsection*{The Quartiles}
The basic idea here is to extend the way the median of a data set sorted in non-decreasing
order is defined and identify two additional values:
\index{quartiles}
\index{descriptive statistics!quartiles}
\begin{itemize}
\item $Q_1$, the \emph{first quartile}, at the middle of the first half of the 
data set (so, roughly speaking, one quarter of the data values lie at or below $Q_1$), and 
\item $Q_3$, the \emph{third quartile}, at the middle of the second half of the 
data set (so, roughly speaking, three quarters of the data values lie at or below $Q_3$).
\end{itemize}
The median then serves as the \emph{second quartile}, with one half of the data values at
or below the median.  Perhaps unfortunately, and certainly inconveniently, there
is no standard way to handle the different special cases that come up depending
on whether the first and second halves of a data set have even or odd numbers
of entries.  In this text we will use the following method to find the first, second, and third
quartiles.  The guiding principle is that the median is \emph{included} in the two halves
used to compute the first and third quartiles when it is one of the values in the data set.\footnote{We will show later how to find
the quartiles following this convention using Excel.  But you should be aware that other books
and other software packages sometimes use different procedures to find the first and third quartiles.
For example, sometimes when $N$ is odd, the median \emph{is not included in either half} to 
compute $Q_1$ and $Q_3$.  To
check numerical results, you will need to know what convention is being used.}
\begin{enumerate}
\item[(1)]  First compute the median as before, giving the second quartile.  
\item[(2)]  If the original data set had an odd number  $N = 2K + 1$ of elements, then include the 
element $x_{K+1}$ in both halves.  The first half will be $x_1,\dots,x_{K+1}$ and the second half
will be $x_{K+1},\ldots, x_{2K+1}$, each with $K+1$ elements.
\item[(3)] If the original data set had an even number $N = 2K$ of elements, then make
the first half $x_1,\ldots,x_K$ and the second half $x_{K+1},\ldots,x_{2K}$.
\item[(4)]  Then $Q_1$ is the median of the first half and $Q_3$ is the median of the second half, 
computed as we described before.
\end{enumerate}

Here are examples illustrating some of the possible cases.

\begin{example}
\label{QuartilesEx}
(a)  First consider the sorted (hypothetical) data set 
$$1.1 \le 2.3 \le 2.4 \le 4.5 \le 5.6 \le 7.8 \le 9.5 \le 10.3$$
Here $N = 8$, which is even.  The median (second quartile) is $\frac{4.5 + 5.6}{2} = 5.05$.  The 
first quartile is then computed as the median of the first half:
$$1.1 \le 2.3 \le 2.4 \le 4.5,$$
so $Q_1 = \frac{2.3 + 2.4}{2} = 2.35$, since the first half also has an even number of elements.
Similarly, $Q_3 = \frac{7.8 + 9.5}{2} = 8.65$.   Something similar would happen for data sets
of length $N = 4, 12, 16$, etc. as well.

(b)  Now consider another hypothetical sorted data set with $N = 7$ elements
$$3.8 \le 3.9 \le 4.3 \le 5.4 \le 6.7 \le 8.9 \le 9.0.$$
Here the median (second quartile) is $x_4 = 5.4$.  The quartiles are then the 
medians of the first and second halves (both including $x_4$).  Both halves also have an even
number of elements this time.  So $Q_1 = \frac{3.9 + 4.3}{2} = 4.1$ and $Q_3 = 
\frac{6.7 + 8.9}{2} = 7.8$ are just the medians of the first and second halves.   Something similar would happen for $N = 3,11,15$, etc.  as well.  

It would also be possible, of course, for $N$ to be even, while the halves are odd
in length -- this happens for instance for $N = 6, 10, 14$, etc.  In yet other cases, $N$ might be odd but 
the halves (including the overall median) could be odd in length -- this happens for $N = 9, 13, 17$, etc.\footnote{Technical note:  Readers
familiar with the notion of integer congruences might recognize that the four cases identified here 
correspond exactly with the four different possible values $N \equiv 0,1,2,\ {\rm or}\ 3 \bmod 4$.}
$\triangle$
\end{example}

The \emph{inter-quartile range}, sometimes abbreviated to IQR, is defined as the difference $Q_3 - Q_1$.  
\index{inter-quartile range (IQR)}
\index{descriptive statistics!inter-quartile range (IQR)}
The IQR provides another sort of measure
of how spread-out a data set is.  

\begin{example}
\label{IQRange}
Let us consider the two data sets from Example~\ref{SDEx} again:
$$5.2  \quad 8.9 \quad 2.1 \quad 7.5 \quad 9.3 \quad 4.3 \quad 4.7$$
and
$$6.2  \quad 7.9 \quad 2.1 \quad 6.5 \quad 9.3 \quad 4.8 \quad 5.2.$$
In sorted order, the first of these is 
$$2.1 \le 4.3 \le 4.7 \le 5.2 \le 7.5 \le 8.9 \le 9.3$$
so the median is $5.2$, and $Q_1 = \frac{4.3+4.7}{2} = 4.5$, $Q_3 = \frac{7.5 + 8.9}{2} = 8.2$.  The IQR
is $Q_3 - Q_1 = 8.2 - 4.5 = 3.7$.  

Doing the same computations with
the second data set, 
$$2.1 \le 4.8 \le 5.2 \le 6.2 \le 6.5 \le 7.9 \le 9.3,$$
so the median is $6.2$, and $Q_1 = \frac{4.8 + 5.2}{2} = 5.0$, $Q_3 = \frac{6.5+7.9}{2} = 7.2$.  The IQR
 is $Q_3 - Q_1 = 7.2 - 5.0 = 2.2$.  Recall that we used the standard
deviation before and saw that the SD of the first was larger while the SD
of the second is smaller.  The inter-quartile range indicates that the 
second data set is less spread-out around its middle in a different way.
$\triangle$
\end{example}

Just as the median is less sensitive to small changes in data values than the mean,
the inter-quartile range is less sensitive to such changes than the standard deviation.  It therefore can be 
thought of as a \emph{more robust} alternative to the standard deviation that might be 
more useful in situations where there is uncertainty or variability in the data values
being analyzed.  However, it should be kept in mind that the SD and the inter-quartile
range values are not directly comparable, given what we have discussed so far.  For 
instance, the values of the SD and 
the IQR values from Examples~\ref{SDEx} and \ref{IQRange} are
not especially close to each other.   

\subsection*{A Measure of Skewness}
The quartiles and the IQR of a dataset can be used to provide a quantitative measure of skewness of a 
\index{skewness}
\index{descriptive statistics!skewness}
data set.  The idea is that \emph{positive skewness} should mean that $(Q_3 - {\rm median})$
is greater than $({\rm median} - Q_1)$, while \emph{negative skewness} should mean
that the reverse is true.  In other words, a dataset is positively skewed if the larger
values are (\emph{consistently}) farther away from the median (to the right along a number line) than
the smaller values are from the median (to the left).  A measure that is commonly
used is 
\begin{equation}
\label{eq:Skewness}
\framebox{${\rm skewness} = \displaystyle{ \frac{(Q_3 - {\rm median}) - ({\rm median} - Q_1)}{Q_3 - Q_1} = \frac{Q_3 - 2\cdot{\rm median} + Q_1}{Q_3 - Q_1}.}$}
\end{equation}

\begin{example}
\label{SkewnessEx}
Let's return to one of the data sets considered in Examples~\ref{MeanMedian} and \ref{MeanMedianCont}:
$$2.1 \le 4.3 \le 4.7 \le 5.2 \le 7.5 \le 8.9 \le 9.3,$$
has mean $\overline{x} = 6.0$, ${\rm median} = 5.2$, $Q_1 = 4.5$ and $Q_3 = 8.2$.  So we compute
$${\rm skewness} = \frac{8.2 - 2\cdot 5.2 + 4.5}{8.2 - 4.5} \doteq 0.62.$$
The fact that the skewness is positive agrees with the observation that the 
large values are farther away from the median than the smaller ones are.  

We also looked at what happens when we remove the value $4.7$:  The shortened data set
 $$2.1 \le 4.3 \le 5.2 \le 7.5 \le 8.9 \le 9.3$$
 has mean $\overline{x} = 6.2$, ${\rm median} = 6.35$, $Q_1 = 4.3$ and $Q_3 = 8.9$.  It
 is often claimed that we should expect a negative skewness when the mean is less than the 
 median.  However, note that in this case
$${\rm skewness} = \frac{8.9 - 2 \cdot 6.35 + 4.3}{8.9 - 4.3} \doteq 0.11$$
is also positive.  The point of this example is that the notion of skewness of a distribution of data values is not
completely intuitive.  Moreover, the relationship
between skewness and the relative positions of the mean and median is not always
as direct as saying: ``skewed towards larger values (``positive skew'') when the mean is greater than median and
skewed towards smaller values (``negative skew'') when the reverse is true.''\footnote{Despite what many 
elementary statistics textbooks claim!}  $\triangle$
\end{example}


\subsection*{Another Criterion for Outliers}
\index{outlier!Tukey criterion for}
The inter-quartile range is used in another criterion for detecting outliers in a data set
originally proposed by John Tukey. 
\index{Tukey, John}
Tukey's criterion is that data values \emph{outside
the range}: 
\begin{equation}
\label{eq:Tukey}
Q_1 - (1.5)\cdot {\rm IQR} \le x \le Q_3 + (1.5)\cdot {\rm IQR}
\end{equation}
are considered as outliers, where ${\rm IQR} = Q_3 - Q_1$ as before.  This gives results comparable in many cases to the 
$3\sigma$ rule discussed above.\footnote{Technical note: The multiplier value 
$1.5$ gives roughly the same $99\%$ fraction of data values within the interval
when the data is \emph{normally distributed}, and that is where this choice came
from historically.}  However,  there is a certain amount of arbitrariness in
the choice of the multiplier $1.5$ here and other choices would be possible.

\subsection*{The ``Five Number Summary'' and Box Plots}
\begin{figure}
\begin{center}
\includegraphics[width=1.0in,height=3.0in]{TroutLengthBoxPlot.png}
\caption{``Box and whisker plot'' of trout length data from Table 3, Chapter 5.}
\label{fig:TroutBox}
\end{center}
\end{figure}
When a statistician wants to begin to understand what information might be present in a set of data values, 
he or she might begin by computing the mean, the SD, and the five statistics 
\begin{equation}
\label{eq:FiveNumberSummary}
{\rm min}, Q_1, {\rm median}, Q_3, {\rm max}.
\end{equation}
Because there is so much information contained in this collection of statistics, it is often called
the \emph{five number summary} of a data set.  

The five number summary is also often presented in a visual way using what is called a
\emph{box plot}, or \emph{``box and whisker'' plot}.  
\index{box plot}

\begin{example}
\label{BoxEx}
Let us consider the data on trout \emph{lengths} contained in Table 3 in Chapter 5 and used in Exercise (12)
in that chapter.  (You will deal with the weight data in a similar way in Exercise~\ref{ex:TroutWeights}.)  The ``box and whisker plot''
is shown in Figure~\ref{fig:TroutBox}.

To interpret this, this data set has $\overline{x} \doteq 352.9$, $s \doteq 57.0$, and five number summary
$${\rm min} = 247, Q_1 = 323.5, {\rm median} = 345.5, Q_3 =  387.25, {\rm max} = 502.$$
Using the scale on the vertical axis, we can see that 
\begin{itemize}
\item The horizontal lines in the ``box'' part of the plot are located at $Q_1$, the median, and $Q_3$.
So the height of the box is the ${\rm IQR} = 63.75$.  
\item The location of the mean is indicated by the small diamond above the horizontal line at
the level of the median.  Note that the mean is slightly larger than the median in this case.  The location of 
the mean is not always included in a box plot, but it adds information.  
\item The ``whiskers'' extend to the maximum and the minimum values in the data set.  So the range
is just the total height of the plot:  $502 - 247 = 255$.  
\end{itemize}
Thus we get a rich visual representation of the distribution of the data by considering this box plot.  
For instance, we can tell that the skewness will be \emph{positive} just by noticing the top portion
of the box (the part between the median and $Q_3$) is taller than the lower part of the box
(the part between $Q_1$ and the median).   
$\triangle$
\end{example}

One question that might arise in considering this data is whether the maximum and minimum values
should be considered outliers, since they are so far outside the range between $Q_1$ and $Q_3$.  
However, note that 
$$502 = {\rm max} < \overline{x} + 3\cdot s = 352.9 + 3\cdot 63.57 \doteq 542.6,$$ so 
this value is still within 3 standard deviations of the mean.  By the $3\sigma$ rule, we would not
label it as an outlier.  Similarly, the minimum is also within 3 standard deviations of the mean:
$$247 = {\rm min} > \overline{x} - 3\cdot s = 352.9 - 3\cdot 63.57 \doteq 162.2.$$
You will carry out the analysis using Tukey's criterion from \eqref{eq:Tukey} in 
Exercise~\ref{ex:Outliers} below.


\specialsection{Exploratory Data Analysis and Excel}
In this section we will consider some larger, real-world data sets and perform some initial 
\emph{exploratory data analysis}.   We will also indicate how the statistics introduced
previously in this chapter can be computed in Excel spreadsheets.  
\index{spreadsheet}

\begin{example}
\label{PitcherPlantData}
\index{pitcher plant (\emph{Darlingtonia californica})|(}
{\it Darlingtonia californica} is a carnivorous pitcher plant that lives in wetlands in mountains in
Oregon and California.  It feeds mostly on ants, flies, and wasps that enter a structure on the
top of the plant called the ``pitcher,'' and that are then trapped.  See the structures at the 
tops of the tubes making up the plants in Figure~\ref{fig:Darlingtonia}.\footnote{Downloaded from
{\tt https://en.wikipedia.org/wiki/Darlingtonia$\_$californica}, August 22, 2017.} The unlucky
insects are eventually digested by secretions generated in the plant's body.  

\begin{figure}
\begin{center}
\includegraphics[width=3.0in,height=3.0in]{Darlingtoniacalifornica.jpg}
\caption{ \emph{Darlingtonia californica} plants.}
\label{fig:Darlingtonia}
\end{center}
\end{figure}

The data set we will use is a collection of measurements made from 25 individual 
{\it Darlingtonia} plants, 
given in Table~\ref{Darlingtonia}.\footnote{This is borrowed from Gotelli and Ellison,
{\it A Primer of Ecological Statistics}, p. 217.}  The second column gives the overall height of the plant, the third gives
the pitcher ``mouth'' diameter, and the fourth gives the pitcher tube diameter.
All measurements are in units of millimeters ($mm$).  

\begin{table}[ht]
\renewcommand\arraystretch{1.5}
\caption{Measurements from individual pitcher plants, {\it Darlingtonia californica}}
\label{Darlingtonia}
\noindent\[
\begin{array}{|c|c|c|c|}
\hline
{\rm Plant\ number} & {\rm Height}  & {\rm Mouth} & {\rm Tube} \\ 
\hline
1 & 744 & 34.3& 18.6\\ \hline
2 & 700 & 34.4 & 20.9\\  \hline
3 & 714 & 28.9 & 19.7\\  \hline
4 & 667 & 32.4 & 19.5\\  \hline
5 & 600 & 29.1 & 17.5\\  \hline
6 & 777 & 33.4 & 21.1\\  \hline
7 & 640 & 34.5 & 18.6\\  \hline
8 & 440 & 29.4 & 18.4\\  \hline
9 & 715 & 39.5 & 19.7\\  \hline
10 & 573 & 33.0 & 15.8\\  \hline
11 & 1500 & 33.8 & 19.1\\  \hline
12 & 650 & 36.3 & 20.2\\  \hline
13 & 480 & 27.0 & 18.1\\  \hline
14 & 545 & 30.3 & 17.3\\  \hline
15 & 845 & 37.3 & 19.3\\  \hline
16 & 560 & 42.1 & 14.6\\  \hline
17 & 450 & 31.2 & 20.6\\  \hline
18 & 600 & 34.6 & 17.1\\  \hline
19 & 607 & 33.5 & 14.8\\  \hline
20 & 675 & 31.4 & 16.3\\  \hline
21 & 550 & 29.4 & 17.6\\  \hline
22 & 5.1 & 0.3 & 0.1\\  \hline
23 & 534 &30.2  & 16.5\\  \hline
24 & 655 & 35.8 & 15.7\\  \hline
25 & 65.5 & 3.52 & 1.77\\ 
\hline
\end{array}
\]
\end{table}

As a first step in analyzing a collection of data like this, we might compute means, SDs, the five number
summaries, box plots, etc. for each of the three measurements.  The purposes of this might be to 
try to identify any issues with the data such as obvious outliers, strange values, etc. and to
start to get a feel for the distribution of values of each of the measurements.

After entering the data into an Excel spreadsheet in four consecutive columns A through D in rows
2 through 26, we can compute these statistics for the data in column B, for instance, as follows.
\begin{itemize}
\item To compute the \emph{mean}  for the data in column B by enter a command {\tt = AVERAGE(B2:B26)} in an 
unoccupied cell.
\item To compute the \emph{standard deviation}, use a command  {\tt =STDEV(B2:B26)} in an unoccupied cell, etc.
\item To compute the \emph{medians}, use a command {\tt =MEDIAN(B2:B26)} in an unoccupied cell.
\item To compute the \emph{quartiles},  use commands like {\tt =QUARTILE.INC(B2:B26,1)} for the first quartile; replace the 
{\tt 1} by {\tt 3} for the third quartile.\footnote{The {\tt QUARTILE.INC} command in Excel computes
the quartiles by the \emph{inclusive} method we described above.  There is another version called {\tt QUARTILE.EXC} that 
computes the quartiles \emph{excluding} the median value in the two halves of the list of data when $N$ is odd.}
\item To compute the \emph{maximum and minimum}, use commands like {\tt =MAX(B2:B26)} and {\tt =MIN(B2:B26)}.  
\item Unfortunately, the basic functions of Excel do not include a way to generate box plots directly.  There are a number
of sets of detailed step-by-step directions for ``tricking'' Excel into creating box plots that are available online, however.  Google
``Excel box plot" if you are interested in trying to do this.  A warning:  however you do it, the process is rather involved(!) 
Fortunately, it is also not difficult to create accurate box plots by hand once you have the five number summary.
\end{itemize}

\begin{figure}
\begin{center}
\includegraphics[width=1.0in,height=2.0in]{DarlingtoniaBox.png}
\caption{Box plot of Height data from Table~\ref{Darlingtonia}}
\label{fig:DarlingtoniaHeightBox}
\end{center}
\end{figure}

For the Height measurements, we have $\overline{x} \doteq 611.7$, $s = 265.1$, and the five number
summary is ${\rm min} = 5.1$, $Q_1 = 545$, ${\rm median} = 607$, $Q_3 = 700$, and 
${\rm max} = 1500$.  From this we see that the IQR is ${\rm IQR} = 700 - 545 = 155$.  
A box plot of the Height data created with a different software package is given in 
Figure~\ref{fig:DarlingtoniaHeightBox}.

Looking at these results and the values in the Height column from the table, 
it is pretty clear that there are at least potential issues with the measurements for plants 11, 22, and 25.
The Tukey criterion for outliers from \eqref{eq:Tukey}, for instance, says that we should consider values outside the interval
$$312.5  = 545 - (1.5)\cdot(155) \le {\rm Height} \le 700 + (1.5)\cdot (155) = 932.5$$
as outliers.  The values $1500, 5.1, 65.5$ \emph{all lie far outside that interval}.   So by the Tukey 
criterion we would label all three of those measurements as outliers.   \index{outlier!Tukey criterion for}

On the other hand, the $3\sigma$ rule yields \index{outlier!$3\sigma$ rule for}
more ambiguous conclusions because the interval of values within three SDs of the mean is 
$$-183.6 = 611.7 - (3)\cdot (265.1) \le {\rm Height} \le 611.7 + (3)\cdot (265.1) = 1407.$$
The small values $5.1, 65.5$ are included in this interval, but the maximum value $1500$ is excluded.
This is actually a very good illustration of a limitation of the $3\sigma$ rule.  Namely, when there
are values in the data set that are very far from the mean and the other data values, the 
computed $s$ can also be inflated and this makes the interval of values within 3 SDs of the 
mean very large.  Look back at the formula \eqref{eq:SD} to see how a few very large $(x_i - \overline{x})$ terms
could produce a large SD value.  In particular, in this case, that interval includes a whole range of \emph{negative values}
that are \emph{impossible} as Height measurements.   So the $3\sigma$ rule would flag only
the one very large value as an outlier in this data set.  

As we said before, though, either way, identifying a data point as an outlier \emph{does not} mean that
we will simply delete or ignore it.  In the context of the pitcher plants studied here, a height of 1500 mm
is not entirely out of the question because some long-lived individuals can definitely grow to more than
a meter in height.  Similarly, immature plants can be very small, so a height of 66.5 or even 5.1 mm is not impossible
either.   It is also good to \emph{resist the temptation} to say whoever recorded the data for plants 22 and 25 ``clearly''
just misplaced the decimal points or forgot the measurements were in millimeters, not centimeters or decimeters.
We should also resist the temptation to say there's a typo in the 1500 (an extra 1).   Those are plausible explanations
for the data, but 
we have no definitive reason to assert either of those things happened.  Editing questionable data to ``fix it'' according
to a theory of how it got that way, however plausible that theory might be,
 is also considered a very ``shady'' statistical practice unless you know for certain that an 
 error was introduced in that exact fashion!

You will carry out a similar analysis for the Mouth diameter and Tube diameter data sets in Exercise~\ref{ex:Exploratory}.

Continuing with the exploratory analysis of this data, let's return to the kinds of calculations we did in 
Chapter 4 on fitting linear models.  We can ask, for instance, whether the Mouth diameter of the pitcher 
plants is linearly dependent on the Height--or, as we said before, whether a linear model is a good fit
for this relationship.   Figure~\ref{fig:DarlingtoniaMouthHeight} shows the scatter plot
of the data points with the Height on the horizontal axis and the Mouth on the vertical axis, and the
best-fit regression line is plotted dashed and in black. 

\begin{figure}
\begin{center}
\includegraphics[width=3.0in,height=3.0in]{DarlingtoniaHeightMouth.png}
\caption{Scatter plot of Mouth diameter vs. Height data from Table~\ref{Darlingtonia}}
\label{fig:DarlingtoniaMouthHeight}
\end{center}
\end{figure}

The computed $R^2$ value is $R^2 \doteq .4618$, indicating a rather poor fit for the linear model.  
\index{R@$R^2$-statistic}
The points
corresponding to the outliers we identified above are clearly visible here -- indeed, the two 
points very close the origin come from plants 22 and 25, while the point corresponding 
to plant 11 is the one far to the right of, and below the regression line.  We might
expect from this picture that the linear relationship would be at least somewhat stronger if the 
outliers were omitted.  But in fact, this is something of an optical illusion because the large
ranges of Height and Mouth diameter values in the original data set tend to make the 
non-linearity of the remaining points less visible.   If those three
points are omitted, the best-fit regression line changes slightly, but $R^2 \doteq .2011$
is even smaller(!)  You will check this in Exercise~\ref{ex:Truncate}.  This sort of omission
of some outliers in a data set is called \emph{truncation} in statistics.  
\index{truncation of outliers}
\index{outlier!truncation}

Another exploratory data analysis technique that can be revealing is to plot 
box plots for two different data sets side by side, or stacked horizontally.
This allows one to compare the distributions in a visual fashion.  For instance,
Figure~\ref{fig:MouthAndTube} shows the box plots for the Mouth diameter and the Tube
diameter data from Table~\ref{Darlingtonia}, side by side.
\begin{figure}
\begin{center}
\includegraphics[width=2.0in,height=2.0in]{MouthsTubes.png}
\caption{Parallel box plots for the Mouth diameter and Tube diameter data.}
\label{fig:MouthAndTube}
\end{center}
\end{figure}
From this we can see that the Tube diameters are generally smaller, that they also
show less variablility (for instance, the IQR is smaller for the Tube diameter data), 
and both data sets are negatively skewed (i.e. skewed toward smaller values).  
\index{pitcher plant (\emph{Darlingtonia californica})|)}
$\triangle$
\end{example}

\specialsection{The $R^2$ Statistic and the Correlation Coefficient (Optional)}
We will conclude this chapter by returning to the computation of the 
the $R^2$ statistic that we have used in an informal
\index{R@$R^2$-statistic}
\index{correlation coefficient}
\index{linear regression}
way to assess the goodness of fit when using least-squares regression,
and a related statistic called the \emph{correlation coefficient}.  

As we said in \eqref{eq:CC} in the Introduction, this formula:
$$R^2 = \frac{\displaystyle{\left(\sum_{i=1}^N (x_i - \overline{x}) \cdot (y_i - \overline{y})\right)^2}}{\displaystyle{\sum_{i=1}^N (x_i - \overline{x})^2 \cdot \sum_{i=1}^N (y_i - \overline{y})^2}}$$
(or an equivalent, rearranged form) is used to compute the $R^2$ statistic.  The correlation
coefficient is a square root of this, computed by 
$$R =  \frac{\displaystyle{\sum_{i=1}^N (x_i - \overline{x}) \cdot (y_i - \overline{y})}}
{\sqrt{\displaystyle{\sum_{i=1}^N (x_i - \overline{x})^2 \cdot \sum_{i=1}^N (y_i - \overline{y})^2}}}$$
We recognize now that the two factors in the denominator of the formula for $R$, namely
$$\sqrt{\sum_{i=1}^N (x_i - \overline{x})^2} \quad {\rm and} \quad \sqrt{\sum_{i=1}^N (y_i - \overline{y})^2}$$
multiply to give $(N-1)$ times the product of the SDs of the $x_i$ and the $y_i$.  In the formula for $R^2$, 
we have the squares of these.  The square $s^2$ of the SD for a data set is called the \emph{variance}, 
so the denominator in the formula for $R^2$ is $(N-1)^2$ times the product 
$$({\rm variance\ of\ the\ } x_i's)\cdot ({\rm variance\ of\ the\ } y_i's).$$

The expression
$$\frac{1}{N-1} \sum_{i=1}^N (x_i - \overline{x})\cdot (y_i - \overline{y}),$$
related to the numerator in the formula for $R^2$, is known as the \emph{covariance} of the
the $x_i$ and the $y_i$.  If the $x_i$ and the $y_i$ were equal for all $i$, then 
the covariance would reduce to the variance of that single data set.  When the 
two data sets are different, the covariance, and the correlation coefficient
are designed to measure the degree to which the $y_i$ \emph{depend linearly on}
the $x_i$ (or vice versa).  We can see the connection if we look at the case where
$y_i = mx_i + b$ for some constant $m,b$ and all $i$ -- or equivalently, where the 
data points $(x_i,y_i)$ all lie exactly on some straight line.  If that is true, then 
$$(y_i - \overline{y}) = (m x_i + b - (m \overline{x} + b) = m (x_i - \overline{x}).$$
Substituting this into the formula for $R$ and simplifying, we have (when
there is an exact linear relation between the $y_i$ and the $x_i$):
\begin{equation}
\label{eq:Rlinear}
R = \frac{m\cdot \sum_{i=1}^N (x_i - \overline{x})^2}{\sqrt{m^2} \cdot \sum_{i=1}^N (x_i - \overline{x})^2} = \frac{m}{|m|} = \begin{cases} +1 & {\rm if\ } m > 0\\  -1 & {\rm if\ } m < 0\end{cases}.
\end{equation}
Hence $R = \pm 1$ in these cases and it can be shown that $-1 \le R \le 1$ in all cases.  Moreover, 
the closer $R$ is to the endpoints of the interval from $-1$ to $+1$, the closer the data points $(x_i,y_i)$
are to lying along a straight line.  

We can see this in intuitive terms
by considering several data sets as in Figure~\ref{fig:CorrCoeff}.
\begin{figure}
\begin{tabular}{cc}
\includegraphics[width=60mm]{RPoint969.png} &   \includegraphics[width=60mm]{RPoint471.png}\\
(a) $R\doteq .969$ & (b) $R \doteq .471$\\
 \includegraphics[width=60mm]{RPoint104.png} &  \includegraphics[width=60mm]{RNegPoint249.png}\\
(c) $R \doteq .104$ & (d) $R\doteq -.249$\\
   \includegraphics[width=60mm]{RNegPoint307.png} &  \includegraphics[width=60mm]{RNegPoint880.png}\\
   (e) $R\doteq -.307$ & (f) $R \doteq -.890$\\
\end{tabular}
\caption{The meaning of the correlation coefficient}
\label{fig:CorrCoeff}
\end{figure}
The figure shows six synthetic data sets with $N = 1000$, with $R$-values ranging from something close
to $+1$ in (a) to something close to $-1$ in (f).\footnote{Technical note:  These were generated from 
\emph{bivariate normal} distributions where the correlation coefficient can be adjusted as a parameter.}  
We note the following observations:
\begin{itemize}
\item As the $R$ values decrease from $.969$ to $.104$ in subfigures (a), (b), (c), the ``point cloud''
gets progressively less linear.  In subfigure (c), in fact, the cloud is almost circular in shape
with essentially no linear relationship between the $x_i$ and $y_i$.  
\item The regression line would still have positive slope in all of the first three cases, however.
\item The slope of the regression line turns negative in subfigures (d), (e), (f).
\item As the $R$ values decrease further from $-.249$ to $-.890$, the ``point cloud''
gets progressively more linear again.
\end{itemize}
This is a somewhat more precise indication of the background behind our use of the $R^2$ statistic in regression analysis.
We will discuss more of the geometry of regression and making inferences about
regression coefficients in a later chapter.  

\specialsection{Chapter Project}
\index{sampling|(}
\index{spreadsheet|(}
The descriptive statistics we have introduced in this chapter are important tools, but they are not the end of the 
story.  The basic goal of statistics is to analyze the information that is contained in a data set and
to make inferences from that information.  The data sets used in environmental questions might
come from sampling some larger population of organisms.  Even when that is not the case, we can 
use the idea of sampling from a larger population of all possible measurements as a way to conceptualize
how the data was generated.  

To understand some of the statistical topics we will look at starting in the next chapter, it will be good to have some more 
intuitive understanding of the process of taking samples (random or otherwise) from a population, and the variability that is 
built into the process of computing statistics that are produced by that sampling process.  This may seem counterintuitive
at first -- after all, there is just one mean or SD (for instance) associated to any particular sample from a larger population. 
There is no variation or uncertainty in dealing with any one data set.  
But the idea is that those sample means and SDs and other statistics produced from \emph{different} sample data sets 
 \emph{can vary a lot} and we need to have a good working understanding of several aspects of that sort of variation.  

We will also practice with some 
additional features of Excel that may be useful for your work on larger assignments for this and other courses.  
Unlike some of the other chapter projects, this project will not focus on an 
environmental data set, but instead will use synthetic data produced in different ways.   Most of your work for this will 
be contained in a spreadsheet file.

\subsection*{Constructing a First ``Population'' and Sampling From It}
Open a new Excel spreadsheet and enter the formula  {\tt =RAND()}  in cell A1.  This function computes a random value from a uniform 
distribution on the interval $0$ to $1$.\footnote{Technical note: The outputs of {\tt RAND} are called \emph{random numbers}, but they are produced using
a definite algorithm, as is true for essentially everything done with computer software.   
So a better name would be \emph{pseudo-random numbers}.  They are only random in the sense that they have
no apparent patterns and if you compute a large collection of them, their distribution will be approximately uniform over the 
interval from $0$ to $1$.}   Copy and paste that into all the cells in the block extending from column A through column J and row 
1 through row 20 (200 cells in all).  Show 3 decimal places in all of the numbers.  You now have a random sample of size 200 from 
that uniform distribution.   

Because of a ``quirk'' of the way Excel deals with spreadsheets containing this {\tt RAND()} function in formulas, you will see that if you now 
enter a formula in another cell of the spreadsheet and calculate that other value, all of the numbers in this block will be recomputed too.  
To turn that feature off (it's annoying and it will mean your results are constantly changing!)  highlight the whole block and from the tool 
bar select Edit/Copy.  Then Edit/PasteSpecial, highlight only the option for Values  (or Numbers in Libre Office), then OK.  If you examine 
the contents of the cells in the block now, you should see just numbers, not the formulas that generated them. 

\subsection*{First Calculations}
Now perform the following calculations:
\begin{enumerate}
\item[(A)]  Compute the average (mean) and the SD of the 200 numbers.  Compute the five-number summary and construct a box plot (by hand or with Excel, 
if you are feeling adventurous and confident).  
\item[(B)]  Next compute the data needed for a \emph{frequency histogram} showing the distribution of this population of 200 numbers (look back at Chapter 6
if you need a refresher on this).  Here's a ``shortcut'' Excel method for doing this.   Since all the numbers are between $0$ and $1$ and we have 
a relatively large data set with $N = 200$,  we can use 
$10$ equal bins with upper boundaries at $0.1, 0.2, \ldots, 1.0$.  
Enter those numbers into cells in one column in the spreadsheet at least three rows below the block of 200 cells above
 (say in column {\tt A} and rows 31 to 40).   Then in the parallel column {\tt B}, highlight the squares in all 10 rows next to the bin boundaries 
 (cells {\tt B31:B40}). In the last cell at the bottom enter the command: 
 {\tt =FREQUENCY(A1:J20,A31:A41)}, and press {\tt SHIFT-CTRL-ENTER} (all together) (or {\tt F2} then {\tt SHIFT-CTRL-RETURN} on a Mac)
 all together.  The whole frequency table should be computed and displayed in the selected cells.
\item[(C)]  Generate a frequency histogram using the computed frequencies with the bins as above.
\item[(D)]  Now, we consider sampling from this ``population.''  Notice that the block {\tt A1:J20} can be thought of as either: 
 \begin{itemize}
\item	$20$ rows (horizontal) with $10$ cells in each, or as 
\item $10$ columns (vertical)  of $20$ cells each.
 \end{itemize}  
We will think of those as two different groups of samples from the population consisting of all $200$ of the numbers--one group of 
$10$ samples of size $20$ from the columns, and as second group of $20$ samples of size $10$ from the rows.  (These are not 
really random samples, of course, since they are constructed in this systematic way.  However the whole block was computed by 
processes that produce random-looking results, so it will do no harm to think of them as random.)
\begin{enumerate} 
\item[(1)] Compute the averages of each of the $20$ rows and put those averages in the cells {\tt L1:L20} (i.e. in the cells in column {\tt L} 
parallel to the block)
\item [(2)] Find the mean and SD of the 20 row averages from part 1.  Generate a frequency histogram for those $20$ values using $5$ bins.  
(You will need to decide on appropriate bin boundaries by examining the values you get.)
\item[(3)] Compute the averages of each of the $10$ columns and put those averages in the cells {\tt A22:J22} 
(i.e. in the cells in row 22 under the block).
\item[(4)]  Find the mean and SD of the $10$ column averages from step 3.  Since we only have 10 of these, making a frequency 
histogram is not too meaningful, so you do not need to do that.  
\end{enumerate}
\end{enumerate}

\subsection*{Constructing a Second Population and Sampling From It}
Open a second tab (sheet) in your spreadsheet and enter the formula 
\vskip 5pt
\centerline{\tt =SQRT(-2*LN(1-RAND()))*COS(RAND()*2*PI())}
\vskip 5pt
\noindent
in cell {\tt A1}.  (This might be tricky -- if you get an error message, check the parentheses carefully and make sure they are exactly as above.)  
Copy and paste that into all the cells in the block extending from column {\tt A} through column {\tt J} and row 1 through row 20 (200 cells in all).  
Show 3 decimal places in all of the numbers.    Repeat parts A -- D above for this new data set, except that you will need to decide on appropriate 
bin boundaries for the histograms.     (Don't worry about exactly what the formula means.  The only thing you need to know for now is that 
it's designed to produce different random values from the ones you used in the first section.)

\subsection*{Further Questions}
\begin{enumerate}
\item[(E)]  Note that {\tt RAND()} appears in both formulas you used to generate both of the ``population data sets?  above.     
Again, this is a built-in Excel formula that generates random (more precisely ``pseudo-random'') numbers in the interval from $0$ to $1$.  
Looking at your population histogram from the first data set, what can you say about the distribution of those numbers?  Do they seem to be 
reasonably uniformly distributed?    What about the second data set where you used the ?black magic? formula?  How is that one different?
How do the two population box plots compare?
\item[(F)]  What about the distributions of the row and column averages in both cases: what did the distributions look like there?   
Did it depend which data set they came from?
\item[(G)]  What can you say about the SD's of the original population, the SD's of the averages of the samples of size 20, 
and the SD's of the averages of the samples of size 10?  In particular, was one of these consistently the largest and one consistently 
the smallest?  If the pattern was not entirely consistent, was there a general trend?
\item[(H)]  If we redid these computations with any fixed rule for computing the ``population'' values, would it always be true that the average 
of the row means is the same as the population mean?  Similarly for the average of the column means?  (Hint: algebra!)
\item[(I)]  What would truly random samples of sizes 10 or 20 from these populations look like?  Suggest a way to generate such samples.  
\item[(J)]  If we had 20 truly random samples of size 10 from either of these populations, would the average of the sample means be the same 
as the population mean?  (This is the same question as H above, but for a different way of producing the samples.)  Explain your answer.
\end{enumerate}

\subsection*{Assignment} Submit your spreadsheet and your answers to questions E - J in a separate document.  
\index{sampling|)}
\index{spreadsheet|)}

\begin{xcb}{Chapter Exercises}
\vskip 5pt
\begin{enumerate}
\item Compute (by hand) the mean, median, range, and standard deviation of the following 
hypothetical data set:
$$4.8 \quad 2.9 \quad 5.4 \quad 4.3 \quad 5.1 \quad 3.2 \quad 39 \quad 4.7.$$

\item 
\label{ex:TroutWeights}
Using Excel, compute the mean, standard deviation, and five number summary
for the fish {\it weight} data from Table 3 in Chapter 5 used in Exercise (12)
in that chapter. (The \emph{length} data is analyzed in the text.)

\vskip 5pt
\item
\index{tanker oil spills}
Using Excel, compute the mean, standard deviation, and the five number summary
for the tanker oil spill amounts from Table 5 at the end of Chapter 6.  \vskip 5pt
\begin{enumerate}
\item[(a)] How many of the spills are within one standard deviation of the mean?\vskip 5pt
\item[(b)] Is this data set positively or negatively skewed?  Draw a box plot to 
verify your answer.  
\end{enumerate}

\item
\label{ex:Outliers}  Are the maximum and minimum values in Example~\ref{BoxEx}
outliers according to Tukey's Criterion \eqref{eq:Tukey}?  Why or why not?

\vskip 5pt
\item  This problem gives an alternate formula for the standard deviation. \vskip 5pt
\index{standard deviation (SD)}
\begin{enumerate} 
\item[(a)] Show that the formula given in \eqref{eq:SD} in 
the text can be rearranged by means of algebra to yield 
$$s = \sqrt{\frac{1}{N-1} \left(\sum_{i=1}^N x_i^2 - N\cdot \overline{x}^2\right)}.$$
(Hint:  Expand out the square in each term in the form given in the text.  Then simplify
using the formula for $\overline{x}$.)
\item[(b)] Use this alternate form to compute the standard deviations of
the data sets from Example~\ref{SDEx}.  You should find that the computations
are somewhat simpler because you do not need to compute the deviations.  You can work 
just with the $x_i$ and with $\overline{x}$.  
\end{enumerate}

\vskip 5pt
\item
Show that the formula for the $R^2$ statistic can be rearranged to each of the following
\index{R@$R^2$-statistic}
forms:
$$R^2 = \frac{N\cdot \sum_{i=1}^N x_i y_i  - \sum_{i=1}^N x_i \cdot \sum_{i=1}^N y_i}{\sqrt{N \cdot \sum_{i=1}^N x_i^2 - (\sum_{i=1}^N x_i)^2}\cdot 
\sqrt{N \cdot \sum_{i=1}^N y_i^2 - (\sum_{i=1}^N y_i)^2}},$$
which also simplifies to 
$$R^2 =\frac{\sum_{i=1}^N x_i y_i - N\cdot \overline{x}\cdot \overline{y}}{\sqrt{\sum_{i=1}^N x_i^2 - N \cdot \overline{x}^2} \cdot \sqrt{\sum_{i=1}^N y_i^2 - N \cdot \overline{y}^2}}.$$

\item
\label{ex:Exploratory}
Carry out the sort of analysis given in Example~\ref{PitcherPlantData} for 
the Mouth diameter and the Tube diameter data sets in Table~\ref{Darlingtonia}.  \vskip 5pt
\begin{enumerate}
\item[(a)]  Compute the mean, SD, and five number summary of each data set.  Construct
a box plot for each by hand, or using Excel if you are feeling adventurous(!) \vskip 5pt
\item[(b)]  Are there any outliers in those data sets?  \vskip 5pt
\item[(c)]  Does there appear to be a linear relation between the Tube diameter and the Height?
\end{enumerate}

\vskip 5pt
\item 
Show that the three datasets in Table~\ref{Darlingtonia} (the Height, the Mouth diameter,
and the Tube diameter) all satisfy the statement of Chebyshev's theorem with $n = 2$.


\vskip 5pt
\item
\index{truncation of outliers}
\index{outlier!truncation}
\label{ex:Truncate}
Verify the claims made at the end of Example~\ref{PitcherPlantData} regarding the 
effects of truncation of outliers in the Mouth diameter versus Height bivariate
dataset.  In particular, generate the point plot of the truncated dataset, plot
the regression line, and show that the $R^2$ value decreases to $R^2 \doteq .2011$.

\vskip 5pt
\item  Investigate the relation between the Mouth diameter and the Tube diameter
in Table~\ref{Darlingtonia}.  \vskip 5pt
\begin{enumerate}  
\item[(a)]  Does there seem to be a linear relationship there? \vskip 5pt
\item[(b)]  What happens if the data is truncated to remove outliers?  Does the 
$R^2$ statistic move closer to $1$?
\end{enumerate}

\vskip 5pt
\item
The data in Table~\ref{Galapagos} gives the areas and the numbers
of species of plants inhabiting each of the small islands making up the Galapagos
chain off the coast of South America.\footnote{This was one of the stops of the H.M.S. \emph{Beagle} where
Charles Darwin collected samples that he eventually used as examples to support his theory of 
evolution by natural selection.  Data from F.W. Preston, The canonical distribution
of commonness and rarity: Part I, Ecology {\bf 43} (1962), 185--215.}
\begin{table}[ht]
\renewcommand\arraystretch{1.5}
\caption{Galapagos Island Plant Species Diversity}
\label{Galapagos}
\noindent\[
\begin{array}{|c|c|c|}
\hline
{\rm Island} & {\rm Area\ (square\ km)} & {\rm Number\ of\ Species}  \\ 
\hline
{\rm Isabela} & 5824.9 & 325\\ \hline
{\rm Floreana} & 165.8 & 319 \\  \hline
{\rm San\ Cristobal} & 505.1 & 306\\  \hline
{\rm Santiago} & 525.8 & 224\\  \hline
{\rm Santa\ Cruz} & 1007.5 &193 \\  \hline
{\rm Pinta} & 51.8 & 119 \\  \hline
{\rm Pinzon} & 18.4 & 103\\  \hline
{\rm Fernandina} & 634.6 & 80\\  \hline
{\rm Espanola} & 46.6 & 79\\  \hline
{\rm Seymour} & 2.6 & 52 \\  \hline
{\rm Santa\ Fe} & 19.4 & 48\\  \hline
{\rm Gardner} & 0.5 & 48 \\  \hline
{\rm Marchena} & 116.6 & 47\\  \hline
{\rm Rabida} & 4.8 & 42\\  \hline
{\rm Genovesa} & 11.4 & 22\\  \hline
{\rm Wolf} & 4.7 & 14\\  \hline
{\rm Darwin}& 2.3 & 7 \\  \hline
\end{array}
\]
\end{table}
\begin{enumerate}
\item Compute the mean, SD, and five number summary for both the Area and Number of Species datasets. \vskip 5pt
\item Are there outliers according the to the criteria discussed in the text?  Would it be reasonable to omit those data items
if there are?  Explain.\vskip 5pt
\item Investigate the goodness of fit of linear, exponential, and power law models for the Number of Species
as a function of the Area of the islands.  Which type of model seems to fit best?
\end{enumerate}
\index{statistics!descriptive|)}
\index{descriptive statistics|)}
\end{enumerate}
\end{xcb}

\endinput
