\chapter{Linear Functions as Models}

\begin{quotation}
\emph{Mathematicians are like Frenchmen -- whatever you say to them, they translate it into their own language and it is 
immediately something completely different }-- Johann Wolfgang von Goethe
\end{quotation}

\specialsection{Introduction}
In Table~\ref{HybridMarketShare}, we show the percentage of the 
U.S. new car sales accounted for by hybrid-electric vehicles (such as
the Toyota Prius).\footnote{From Alternative Fuels and Advanced
Vehicle Data Center, U.S. Department of Energy,
{\tt https://www.afdc.energy.gov/data/10301}, accessed June 26, 2017.}   
Hybrids incorporate internal combustion engines and electric engines
powered by storage batteries.  This design
increases fuel efficiency (typically to levels over
40 mpg) and reduces tailpipe emissions.  These vehicles became widely available
in the early 2000's.
\begin{table}[ht]
\renewcommand\arraystretch{1.5}
\caption{Hybrid Vehicle Market Share in the U.S.}
\label{HybridMarketShare}
\noindent\[
\begin{array}{|c|c|c|c|c|c|c|c|}
\hline
{\rm Year} & 2002 & 2003 & 2004 & 2005 & 2006 & 2007 & 2008\\ 
\hline
\% {\rm\ of\ sales}& .24 & .32 & .56 & 1.40 & 1.77 & 2.55 & 2.37\\ 
\hline
\hline
{\rm Year} & 2009 & 2010 & 2011 & 2012 & 2013 & 2014 & 2015 \\
\hline
\% {\rm\ of\ sales} & 2.79 & 2.37 & 2.11 & 3.01 & 3.19 & 2.75 & 2.21\\
\hline
\end{array}
\]
\end{table}
Mathematically, we can consider the hybrid vehicle market share as a 
\emph{function} of the years in this range.  This means that for each year shown, the
market share of hybrid vehicles during that year is a well-defined number
depending on the year.   The value 3.19 for the year 2013, for instance, means
that $3.19\%$ of all new vehicle sales in the U.S. were hybrids that year.

This information can be represented graphically in several ways.  In Figure~\ref{fig:HybridPointPlot},
we show a point plot with the year on the horizontal axis and the market share on the vertical axis.
\begin{figure}
\begin{center}
\includegraphics[width=2.5in,width=2.5in]{HybridPointPlot.png}
\caption{Point Plot of Hybrid Car Market Share Data}
\label{fig:HybridPointPlot}
\end{center}
\end{figure}
Another way information such as this might be presented is as a ``bar chart,'' as in Figure~\ref{fig:HybridBarChart}.
There we include one bar for each year, whose height gives the market share in that year.
Both of these plot styles are designed to let us visualize the change in the hybrid
market share over time.  And they both show a rather irregular up-and-down pattern
with peaks in 2009 and 2013, followed by declines.  Evidently, the appeal of hybrid vehicles
is different in years when gasoline prices are trending upward; one of the attractive
features of hybrids is their greater fuel efficiency.  But this also comes at a higher cost
in the sticker price of hybrid vehicles. As a result, people considering purchasing a hybrid often take their expected fuel consumption
into account in making the decision and hybrids tend to be less attractive when fuel prices are low.

Nevertheless, we can see 
what is apparently an upward trend in the hybrid market share over time.  
In this chapter we will focus on modeling situations like this using what are called
\emph{linear} models, where the graph of the corresponding mathematical function
would be a \emph{straight line}.  We will also see a way, called \emph{linear regression},
to estimate the straight line that fits a data set like our hybrid market share data as 
closely as possible.  

\begin{figure}
\begin{center}
\includegraphics[width=2.5in,width=2.5in]{HybridBarChart.png}
\caption{Bar Chart of Hybrid Car Market Share Data}
\label{fig:HybridBarChart}
\end{center}
\end{figure}

\specialsection{Functions}
We will begin by discussing the general mathematical concept of a \emph{function}.  This idea may
be familiar if you have taken a precalculus or calculus course, but we will review the basics and
use that language here as well.  A function
can be thought of as a rule, or a process, or perhaps even as a sort of machine, that assigns to each element
in some set (called the \emph{domain}) a \emph{unique} element of some other set (called the \emph{range}).  
We often think of the domain $D$ as the set of \emph{input values} for the function; the corresponding \emph{output values}
make up the range.  The most important part of this definition is that there is \emph{exactly one
corresponding output for each possible input} -- there is no ambiguity or mulitplicity involved
in the process of going from the input to the output.  

For us, the domain
and range will always be sets of numbers.  Mathematicians often give functions single-letter
names such as $f$ and write $f : D \to R$ to mean that $D$ is the domain of the function and $R$ is
the range.  If $x$ is an element of the domain $D$, then the notation $f(x)$ represents the element
of $R$ that the function produces as output when the input value is $x$. 

\begin{example}
\label{FormulaFunction}
Probably the most common way to specify a function is to give a \emph{formula} for 
producing $f(x)$ from $x$. 

(a) For instance the formula
$$f(x) = 17.9\cdot x + 23.7$$
defines a function with domain the set of all real numbers because we can compute $f(x)$ for any real
number by taking $x$, multiplying it by $17.9$, then adding $23.7$ to the result.   For instance
$$f(1.1) = 17.9 \cdot 1.1 + 23.7 = 43.39.$$
Similarly
$$f(-2) = 17.9 \cdot (-2) + 23.7 = -12.1.$$
This function is especially simple because we can substitute any real value for $x$ in the formula.

(b) This may not be true for other more complicated function such as the one defined by 
$$g(x) = \frac{1}{x^2 - 16}.$$
Here notice that $g(4)$ and $g(-4)$ are not defined because substituting $x = 4$ or $x = -4$ makes the denominator
in the formula equal to zero; division by zero is not defined.  In cases like this, the 
usual ``rule of thumb'' is to take the domain to be the set of all real $x$ that can be 
substituted into the formula to yield a well-defined value (unless some other domain is explicitly specified).

(c) The base-$a$ logarithms that we discussed in Chapter 1 are values of function 
$f(x) = \log_a(x)$ with domain equal to the set of all strictly positive real numbers.   The range of the base-$a$ 
logarithm is the set of all real numbers.  These statements reflect the facts that the logarithm
of a negative number is not defined in the real numbers, but the value of a logarithm can be negative
if the input is $< 1$ and $a > 1$. $\triangle$
\end{example}

Other ways of specifying functions will also be important for us in this course.  

\begin{example}
\label{TableFunction}
The hybrid vehicle market share values in Table~\ref{HybridMarketShare} in the Introduction to this chapter
can be thought of as the definition of function with domain 
$$D = \{2002,2003,\ldots,2015\}$$
and range given by the set of market share values.  The other tables of values we
have considered in previous chapters could be thought of in the same way.  Especially
when a table of values represents the data collected in a scientific experiment or
measurement process, we may want to think of those values as a sort of \emph{incomplete
description} of a function that might be defined for other input values as well.   For example,
the tabulated data in Tables~\ref{ArithLogTable} and \ref{GeomLogTable} from our discussion
of the Weber-Fechner law in Chapter 1 show subsets of the ranges of the $\log_{10}$ function,
but that function is defined for all strictly positive $x$, not just for the input values
given in those tables. $\triangle$
\end{example}

\begin{example}
A function could also be defined by giving a graph in the $xy$-plane where each
vertical line $x = c$ meets the graph \emph{at most once}, as in Figure~\ref{fig:Graph}.  If $(c,d)$ is the 
point on the graph with $x = c$, then we would define $f(c) = d$.   For instance, from this plot
we can see $f(0) = 0$ and $f(2) \doteq 2.7$. We say such a graph
``passes the vertical line test.''  Even if we do not have an explicit formula for such
a function, the graph is a complete specification.  $\triangle$
\end{example}

\begin{figure}
\begin{center}
\includegraphics[width=2.5in,width=2.5in]{FunctionGraph.png}
\caption{Function Defined by a Graph}
\label{fig:Graph}
\end{center}
\end{figure}

\specialsection{Linear Functions}
In this chapter, we focus on one particular class of functions--the ones known as \emph{linear functions}--and their applications.
There are several ways to say which functions are linear and to understand what makes a function 
linear.  

First, linear functions are exactly the functions whose formulas can be rearranged to the form
\begin{equation}
\label{eq:LinearFunction}
\framebox{$f(x) = m x + b$}
\end{equation}
where $m, b$ are constants.  As you probably recognize, the graph 
$$y = f(x) = m x + b$$
is just a \emph{straight line} and that gives a second characterization of the linear functions 
in terms of their graphs.  The constants $m,b$ have a nice interpretation
in terms of that line.  The constant $m$ is called the \emph{slope} of the line
and it has the following interpretation.  Suppose we start from $x = a$ and the 
corresponding point $(a,f(a)) = (a,m a + b)$ on the graph of the linear function.
If we change the $x$-coordinate to $a + \Delta x$, then the $y$-coordinate
changes to 
$$f(a + \Delta x) = m(a + \Delta x) + b = m a + m\Delta x + b.$$
Hence the corresponding change in the $y$-coordinate is just
$$\Delta y = f(a + \Delta x) - f(a) = m a + m \Delta x + b - (m a + b) = m\Delta x.$$
Assuming $\Delta x \ne 0$,\footnote{If $\Delta x = 0$ for any two distinct points on 
the line, then the line is vertical, and the equation has a different form, namely
$x = c$ for some constant $c$.  We do not have the graph of a function in that case because 
a vertical line does not pass the vertical line test.} we can divide through to yield
\begin{equation}
\label{eq:Slope}
\frac{\Delta y}{\Delta x} = m.
\end{equation}
\begin{figure}
\begin{center}
\includegraphics[width=2.5in,width=2.5in]{SlopeMeaning.png}
\caption{Slope of a Line}
\label{fig:LinearSlope}
\end{center}
\end{figure}
Notice that this equation is true for all $a$ and all $\Delta x$.  In other words, according to the 
language introduced in Chapter 2, for all $a$ and all $\Delta x$, the ratios $\frac{\Delta y}{\Delta x}$
are always the same, equal to the slope $m$.  Another way to say this is that $\Delta y$ and
$\Delta x$ are always \emph{proportional}, with ratio equal to $m$.  
This gives an important additional way to understand the property of linearity, which is illustrated
by the diagram in Figure~\ref{fig:LinearSlope}.
\emph{If we take any two points on the graph of a linear function and draw in a small right triangle
with legs parallel to the coordinate axes and hypotenuse along the graph, as in Figure~\ref{fig:LinearSlope},
then ratio of the legs (which is the tangent of the angle between the horizontal leg and the line)
is always the same, and equal to the constant $m$ in the equation of the line as in 
Equation~\eqref{eq:LinearFunction}.}
A slope value also comes with associated units, as is true for any ratio.  These are
$\frac{{\rm units\ of\ } y}{{\rm units\ of\ } x}$.  

There is also an easy interpretation for the constant $b$ in Equation~\eqref{eq:LinearFunction}.  The
value $b = f(0) = a\cdot 0 + b$.  So the point $(0,b)$ on the $y$-axis is also on the line defined
by $y = m x + b$.  This point is called the \emph{$y$-axis intercept}.  Hence $b$ is sometimes
also called the \emph{intercept} and Equation~\eqref{eq:LinearFunction} is called the 
\emph{slope-intercept form} of the equation of a line.   Because the $b$ is a value of $y$, its
units are the same as those of $y$.

\begin{example}
\label{LineExample}
Hence, for instance, the 
line defined by $f(x) = -3 x - 1$ has slope $m = -3$ and $y$-axis intercept $(0,-1)$.
The negative value of the slope means that the line is ``sloping down'' left to right -- 
in other words, the $y$-coordinates of points on the line are decreasing as the 
$x$-coordinates increase.   $\triangle$
\end{example}

An alternative way to specify a linear function is to give the slope $m$ and
one point $(x_0,y_0)$ on the line (i.e. the graph of the function).  If $(x,y)$
is any other point on the line, then using those two points we have
$$\Delta x = x - x_0 \quad {\rm and} \quad \Delta y = y - y_0$$
so from Equation~\eqref{eq:Slope}, we obtain
$$\frac{y - y_0}{x - x_0} = m,$$
and hence
\begin{equation}
\label{eq:PointSlope}
\framebox{$y - y_0 = m(x - x_0)$.}
\end{equation}
This is called the \emph{point-slope form} of the equation of the line.  Note that this 
form can be rearranged to give
$$y = m x + (y_0 - mx_0)$$
by distributing the product of $m$ and $x - x_0$ and adding $y_0$ to both sides.
So we have a linear function $f(x) = m x + (y_0 - mx_0)$ as in Equation~\eqref{eq:LinearFunction}, where 
$$b = y_0 - m x_0.$$
There is no need to memorize this formula for the intercept; you can always rearrange an equation
given in point-slope form to slope-intercept form as above if you need to find the intercept.  

\begin{example}
\label{PointSlopeEx}
The line with slope $m = 4.2$ containing the point $(1.1,2.8)$ has equation 
$$y - 2.8 = 4.2(x - 1.1) \quad {\rm or}\quad y = 4.2 x - 1.82. \quad \triangle$$
\end{example}

It is a basic fact from geometry that there is exactly one line passing through any two
distinct points in the plane.  GIven the two points, $(x_1,y_1)$ and 
$(x_2,y_2)$, if we need to determine
the line, the fastest way is to compute the slope using Equation~\eqref{eq:Slope} first, 
and then use Equation~\eqref{eq:PointSlope}.  The known point $(x_0,y_0)$
can be either of the two given points; the resulting equation will be the 
same after simplification.

\begin{example}
\label{TwoPointsEx}
Say we have the two points $(x_1,y_1) = (2.2,8.7)$ and $(x_2,y_2) = (4.3,5.0)$.  We ask, what
is the equation of the line through those points?  The slope must be
$$m = \frac{y_2 - y_1}{x_2 - x_1} = \frac{5.0 - 8.7}{4.3 - 2.2} \doteq -1.76.$$
So, using $(2.2,8.7)$ as the known point in Equation~\eqref{eq:PointSlope}, the line is (approximately)
$$y - 8.7 = -1.76(x - 2.2) \quad {\rm or} \quad y = -1.76 x + 12.57.$$
Note that if we substitute $x = 4.3$ and compute $y$ we find
$$(-1.76)(4.3) + 12.57 = 5.002, $$
which is not exactly $5.0$ as we expect for $x = 4.3$.  The reason for this is that our slope $\frac{-3.7}{2.1} = -\frac{37}{21}$ is 
a rational number with an infinite repeating decimal expansion.  Therefore the 
rounded value $m \doteq -1.76$ is only approximate. $\triangle$
\end{example}

To conclude this section, we will consider how to tell whether a table of values
comes from a linear function and determine the equation if it does.  This basically
uses all the ideas we have introduced so the idea should be clear if we just
consider the following example.

\begin{example}
\label{LinearTable}
The population of the state of California, measured at 10-year intervals in
the period between 1870 and 1920 is given Table~\ref{CalPop}.\footnote{According to U.S. Census data.}
\begin{table}[ht]
\renewcommand\arraystretch{1.5}
\caption{Population of California 1870 to 1920}
\label{CalPop}
\noindent\[
\begin{array}{|c|c|c|c|c|c|c|}
\hline
{\rm Year} & 1870 & 1880 & 1890 & 1900 & 1910 & 1920\\ 
\hline
 {\rm Population}& 560,247 & 864,694 & 1,213,398 &  1,485,053 & 2,377,549 & 3,426,861\\ 
\hline
\end{array}
\]
\end{table}
\begin{figure}
\begin{center}
\includegraphics[width=2.5in,width=2.5in]{CalPopPlot.png}
\caption{California Population Data}
\label{fig:CalPops}
\end{center}
\end{figure}
We ask whether these are values of some linear function of time in years.  According to 
Equation~\eqref{eq:Slope}, that would be true
exactly when the slope between any two of the points was the same.  However, using the 
populations for 1870 and 1880 gives
$$\frac{864,694 - 560,247}{1880-1870} = 30,444.7\ {\rm people/year},$$
while the same computation for the populations in 1880 and 1890 gives
$$\frac{1,213,398 - 864,694}{1890 - 1880} = 34,870.4\ {\rm people/year}.$$
This alone is enough to show that the tabulated values \emph{do not} come from any single
linear function.  

Nevertheless if we make a point plot of these population values as in Figure~\ref{fig:CalPops} then we see some 
interesting things.
First notice that the points here are not actually all that far from lying on a straight line.
We could ask -- Is there a line that ``fits'' the data as well as possible.  Looking more
deeply, we can also see that there was apparently also some sort of ``change point''
in 1900.  The first four points from the census values for 1870, 1880, 1890 and 1900
are actually even closer to lying on one line than the data points as a whole.  The
same is true for the final three points from the census values for 1900, 1910, and 1920.
We might try to ``fit'' two lines separately and compare their slopes to understand
the apparent difference between the first part of the time period and the second part.
We will return to this example later in this chapter. $\triangle$
\end{example}

\specialsection{Mathematical Models}
Many of the mathematical topics we will study this year will be applied to construct
\emph{mathematical models} of various real-world and environmental situations.
The idea of mathematical modeling is shown in the diagram given in 
Figure~\ref{fig:MathModels}.\footnote{Source: {\tt http://mathforteaching.com}, downloaded on 
June 27, 2017.}
\begin{figure}
\begin{center}
\includegraphics[width=3.0in,width=3.0in]{modeling.jpg}
\caption{The Mathematical Modeling Process}
\label{fig:MathModels}
\end{center}
\end{figure}
This sort of mathematical work is (almost) always motivated by the desire to 
understand and/or make predictions about the behavior of some real-world system.\footnote{Some 
mathematicians also study mathematics ``purely'' for its own sake, for the beauty they see 
in its structures, or for other reasons, without
any applications in mind.  One of the surprising things we can see from the history
of the subject is that even though many mathematical techniques  were originally developed  without
any applications in mind, they have later turned out to be useful for applications even so.  A 
famous essay by the physicist Eugene Wigner called \emph{The unreasonable effectiveness
of mathematics in the natural sciences} discusses some of this.}  

The modeling process can be viewed as a sort of ``loop'' or iteration as shown by the 
arrows in Figure~\ref{fig:MathModels}.  The loop should ideally start with a deep understanding 
of the real-world problem, and for this reason mathematicians doing this sort of work often collaborate 
with biologists, demographers, climate scientists, and other experts in the applications areas they 
want to study.  Moving into the mathematical world, the underlying real-world problem is often reformulated 
and abstracted as a mathematical question.  This often involves making simplifying assumptions or 
leaving out or ignoring aspects of the real world system that are thought to be irrelevant
(or at least less important) for the problem at hand.  

A mathematical model can be an equation, or a graph, or some other mathematical structure
that captures some of the features of the real-world system under study.  The mathematical
structures make it possible to compute solutions of equations or produce other mathematical
results.  Is that the end of the story?  {\it Definitely not.}  These results
from the realm of mathematics must be \emph{tested against}, or compared with, data generated from the real-world
system.  
\begin{itemize}
\item If there \emph{is} sufficiently close agreement or the mathematical results yield good
insights about that  system, then the model has produced useful information.   Even in
this case, though, we must always be ready to acknowledge that there may be other important
processes in play that our models are not not taking into account.  We may consider refining
and/or completely scrapping our models because of that.
\item If there is \emph{not} the sort of agreement we are seeking, 
and in many cases, even if there is, the modeling loop goes through additional iterations
to refine the models and improve their predictive power.  In particular, it might be the case
that some features ignored previously are seen to be essential to the development of 
a realistic model and useful predictions.  
\end{itemize}
Unlike the case in pure mathematics where solving an equation, or 
developing a proof of a statement can be the end of the story, mathematical modeling 
can be a much more incremental process--much more like the sort of experimental work
that goes on in many scientific disciplines, in fact.

Some of the simplest mathematical models that are used in practice make use of 
the linear functions we introduced earlier.  A linear model says that one quantity, the \emph{dependent
variable} corresponding to the $y$, depends on another quantity, the \emph{independent
variable} corresponding to $x$, in a linear fashion as in $y = mx + b$, at least \emph{approximately}.

\begin{example}
\label{CalPopCont}
We continue the discussion of the California population data from Example~\ref{CalPop} above.
Recall that we said that, in the point plot from Figure~\ref{fig:CalPops}, the population values 
for 1870, 1880, 1890, and 1900 were visually very close to being collinear.  We will see in the 
next section how to derive an equation of the line that ``best fits'' this data, and using the 
techniques to be presented there we will be able to derive the equation 
\begin{equation}
\label{eq:CalPopRegLine}
{\rm Population} \doteq (31231.22)\cdot ({\rm Year} )- 5.784 \times 10^7
\end{equation}
for this best fit line.  The closeness of the line defined by this equation and our data points
can be seen if we plot both together on the same axes as in Figure~\ref{fig:PopPlusLine}.
\begin{figure}
\begin{center}
\includegraphics[width=2.5in,width=2.5in]{CalPopModel1.png}
\caption{A First ``Best Fit'' Line}
\label{fig:PopPlusLine}
\end{center}
\end{figure}

Our main point is that we can view this linear function as a \emph{mathematical model} for the 
California population, at least during the period between 1870 and 1900.   A portion of the
meaning of this would be that the population was growing at a constant rate of 
about $31,231$ people per year, from the numerical value of the slope coefficient
in the linear function from Equation~\eqref{eq:CalPopRegLine}.
Having this model would, for instance, 
allow us to generate estimates for the population in years where there was no Federal
census.  For instance with ${\rm Year} = 1882$, we obtain 
$${\rm Population} \doteq (31,231.22)(1882) - 5.784 \times 10^7 \doteq 937,154.$$
Looking back at Table~\ref{CalPop}, this is clearly reasonable as an estimate for the population
that year.

On the other hand, as we noted before, something seemingly changed in the way the California population
was growing around 1900.  The value obtained from the linear function in Equation~\eqref{eq:CalPopRegLine}
for ${\rm Year} = 1920$, 
$$(31,231.22)\cdot (1920) - 5.784 \times 10^7 \doteq 2,123,941$$
is much smaller than the actual population that year, namely $3,426,861$.  Similarly, extrapolating
back to values like ${\rm Year} = 0$ yields \emph{negative} population estimates(!)  Hence predictions
from a mathematical model must be used with care.  Here the operative restriction to keep in mind
is that the formula in Equation~\eqref{eq:CalPopRegLine} was derived using only the actual population
values in the period from 1870 to 1900 so for it to produce reliable estimates, it should essentially only be applied for
Year values in or close to that time range. $\triangle$
\end{example}

\specialsection{Linear Regression}
In this section we will present an overview of the technique of \emph{least squares regression} which
is the standard method used to compute the equation of a line, or \emph{linear model}, best fitting a collection of data
points as in Equation~\eqref{eq:CalPopRegLine} in Example~\ref{CalPopCont}.  The computations 
involved are somewhat tedious to perform by hand, so after giving a simple example, 
we will also discuss how they can be 
done in an Excel spreadsheet.  It is good to understand what is really going on ``under the hood'' there, 
though, so we will not treat least squares \emph{only} as a ``black box.'' 

The basic problem is this:  Given a collection of points $(x_i,y_i)$, for $i = 1,\ldots, N$, find
the equation of the line that ``comes the closest'' to passing through the points in a suitable sense.  
For the moment, we consider a general line $y = m x + b$, where we think of the slope
$m$ and the intercept $b$ as variables whose values determine which line we are talking
about.  In most cases we think of the $x_i$ as \emph{known exactly} since the independent variable
$x$ is something we, or whoever collected the data we are using, had control over in the data 
collection.  For instance, if $x$ represented a quantity like a temperature and $y$ represented
the proportion of insect larvae that survive to adulthood as a function of the temperature, then
measurements were probably made at several different temperatures $x_i$ and the proportions
$y_i$ were determined for each one.  Similarly, in the California population data set, there is 
no uncertainly in the $x_i$ -- they are just the years when the Federal census is taken.  On the
other hand, the $y_i$ typically do involve some randomness or uncertainty such as measurement
error.  

Hence it makes sense to compare each $y_i$ with the $y$-value predicted by the linear
function $m x + b$ with the input value $x = x_i$.  The comparison is made by means
of the difference or \emph{deviation} of $y_i$ from the value $m x_i + b$ predicted by the model:
$$y_i - (m x_i + b).$$
In graphical terms, we can think of this as the vertical distance between the point $(x_i,y_i)$ and
the point on the line $y = m x + b$ with $x = x_i$.
Least squares regression determines the line--equivalently the pair of values of $m$ and $b$--that \emph{minimizes the sum of the
squares of the deviations}:
\begin{equation}
\label{eq:SSD}
S = \sum_{i=1}^N (y_i - (mx_i + b))^2 = (y_1 - m(x_1 + b))^2 + \cdots + (y_N - m(x_N + b))^2.
\end{equation}
Squaring the deviations makes small deviations\footnote{i.e. those less than 1 in absolute value.}  
``count less'' and large ones\footnote{i.e. those greater than 1 in absolute value.} ``count more.''  Minimizing $S$--that is, essentially, varying
$m,b$ to make $S$ as small as possible--means we are eliminating large deviations as much as possible.  Squaring also means that
underestimates, where the deviation is positive, are \emph{not canceled out by overestimates}, where
the deviation is negative.   This is reasonable if we are looking for a line that is as close as possible
to passing through all of the points $(x_i,y_i)$ but we are willing to allow the line to pass over or under 
any one of those points.

The one technical point that we will not try to explain here is the following.   Techniques from calculus
(or linear algebra) show that the $m,b$ values that achieve the minimum for $S$ are always given by the  solutions
of the two so-called \emph{normal equations}:
\begin{align}
\label{eq:NormalEqs}
\left(\sum_{i=1}^N x_i^2\right)\cdot m + \left(\sum_{i=1}^N x_i\right)\cdot b & =  \sum_{i=1}^N x_i y_i \\ \nonumber
\left(\sum_{i=1}^N x_i\right)\cdot m + N\cdot b & =  \sum_{i=1}^N y_i .
\end{align}
This gives a system of two simultaneous linear equations for $m$ and $b$, and we can solve them with techniques
from high school algebra to find the $m$ and $b$ values in the equation of the least squares regression line.

\begin{example}
\label{RegEx}
In order to be able to say one really understands least squares regression, everyone should do one of these
calculations at least once \emph{by hand}.  Let us consider the simple ``made-up'' data set from Table~\ref{HypData}
\begin{table}[ht]
\renewcommand\arraystretch{1.5}
\caption{A hypothetical data set}
\label{HypData}
\noindent\[
\begin{array}{|c|c|c|c|c|c|}
\hline
i & 1 & 2 & 3 & 4 & 5\\
\hline
x_i & 1.2 & 1.9 & 2.1 & 2.5 & 3.7\\ 
\hline
 y_i & 5.4 & 6.6 & 6.4 & 7.1 & 6.9\\ 
\hline
\end{array}
\]
\end{table}
The idea is that we can simply write down the normal equations from Equation~\eqref{eq:NormalEqs} without going through
the whole computation of the total squared deviation.  We have 5 data points, so $N = 5$.  Then
\begin{align*}
\sum_{i=1}^N x_i^2 &= (1.2)^2 + (1.9)^2 + (2.1)^2 + (2.5)^2 + (3.7)^2 = 29.40\\
\sum_{i=1}^N x_i y_i &= (1.2)(5.4) + (1.9)(6.6) + (2.1)(6.4) + (2.5)(7.1) + (3.7)(6.9) = 75.74\\
\sum_{i=1}^N x_i &= 1.2 + 1.9 + 2.1+ 2.5 + 3.7 =  11.4\\
\sum_{i=1}^N y_i &= 5.4 + 6.6 + 6.4 + 7.1 + 6.9 = 32.4.
\end{align*}
Hence the normal equations are
\begin{align*}
29.4 m + 11.4 b &= 75.74\\
11.4 m + 5 b &= 32.4.
\end{align*}
By elimination or Cramer's Rule, the solution is 
$$\hat{m} \doteq .548 \quad {\rm and} \quad \hat{b} \doteq 5.23.$$
We will consistently write $\hat{m}$ and $\hat{b}$ for the solution of the normal equations in a linear
regression.\footnote{This notation is borrowed from the way statisticians indicate an \emph{estimator}
for a parameter in a statistical model -- here estimators for the 
slope and the intercept in the linear model.}
The data points and the (approximate) regression line $y = .548 x + 5.23$ are plotted in Figure~\ref{fig:HypDataPlot}.
\begin{figure}
\begin{center}
\includegraphics[width=2.5in,width=2.5in]{HypDataReg.png}
\caption{A  ``Best Fit'' Regression Line}
\label{fig:HypDataPlot}
\end{center}
\end{figure}
The relatively close agreement between the dashed black line and the red data points indicates that this linear model
fits the data quite well.  $\triangle$
\end{example}

We now discuss a basic way to generate point plots, compute equations of regression lines, and plot
them with the point plots in Excel.  (There are several other
ways to do this as well; this is the simplest method.)  

To create a point plot of what Excel wants to call a \emph{bivariate data set} (that is a 
collection of $(x_i,y_i)$ points for $i = 1,\ldots,N$), you 
will follow these steps:
\begin{enumerate}
\item  Enter the $x_i$ and $y_i$ values into the spreadsheet
in two consecutive columns.  (To help you understand what you did
if you come back to the spreadsheet later, it is often helpful to 
enter text headings in the cells at the top of the columns, but
that is not necessary.)
\item  Highlight the range of cells containing the data.  (On a PC you do
this by holding down the left ``mouse'' button and dragging the cursor.  
On a Mac, use one finger to highlight and drag with the other.)
\item  With the data highlighted, press the Insert tab of 
the Excel window, and choose the option Scatter from the Plots
group.
\item  You should see a bare-bones version of the plot
generated at this point.  
\end{enumerate}

You will almost always want to edit your plot
to add axis labels, a chart title, regression lines (examples of what 
are called \emph{trend lines} in Excel), equation(s), etc.
To do this you will use various options in the Layout tab of the
Chart Tools (PC) or Chart Design (Mac) group.  

\begin{enumerate}
\item[(5)]  With the Layout tab of the Chart Tools group highlighted on a PC or the Add Chart Element
tab on Mac, you should
see options labeled Chart Title, Axis Titles, Legend, etc.  Each of those
buttons produces a menu that you use to add or
remove features of the chart.  The title options, for instance, add 
text boxes overlaying the graph that you type in to add the title
you want.  
\item[(6)] The Trendline menu contains
options for linear regression and other sorts of calculations, some of which we will use later.
The Linear Trendline button just adds the trendline, though.
If you want to be able to generate the equation of the line overlaid
on the plot, go to the bottom option in the menu (Other Trendline Options), 
select the trend/regression type you want, and check the box that
says Display Equation on Chart (this is down at the bottom of the list and 
you may need to scroll down to find it).
\end{enumerate}

Figure~\ref{fig:ExcelRegression} shows a screen shot of the Mac version of Excel 2016 showing results of
commands like those given above on the data set entered in cells B2-C7. 
\begin{figure}
\begin{center}
\includegraphics[width=4.0in,width=4.0in]{ExcelReg.png}
\caption{A  ``Best Fit'' Regression Line in Excel.}
\label{fig:ExcelRegression}
\end{center}
\end{figure}
The chart title \emph{Regression Example} was generated automatically
by the column heading in column C (the contents of cell C2).  The x-values and y-values 
labels on the axes in the plot were added manually.  The 
equation $y = 2.8226 x - 6.302$ is in a ``text box'' element.
This may come out covering part of the graph but you can move
it.  The same is true of the equation $R^2 = .94757$.  The $R^2$ statistic
is a measure of how close the regression line comes to passing through the
data points.  The closer that is to $1$, the better the ``fit,'' so this is quite
good. 

When evaluating whether a linear model is a good 
fit for a data set, experienced mathematical modelers and statisticians
usually do an additional step -- investigation of the so-called \emph{residuals}
for the regression.  The residuals are the differences
$$r_i = y_i - (\hat{m} x_i + \hat{b}),$$
where $\hat{m},\hat{b}$ are the solutions of the normal equations from 
Equation~\eqref{eq:NormalEqs}.
The residual $r_i$ is a measure of how far ``off'' the value for $y$ with 
$x = x_i$ predicted by the linear model is from the measured value $y_i$ from 
the data set.  The \emph{residual plot} is the point plot (Excel scatter plot)
for the data set consisting of the points $(x_i,r_i)$.  If the scatter plot
shows more or less random-looking up and down variations from $0$, then there
is no reason to be concerned.  On the other hand, if the residuals show some
systematic pattern--for instance, 
\begin{itemize}
\item if they are all negative for the first half
of the data, then all positive for the second half, or more generally
\item if the scatter plot has long intervals of consistent
curvature creating consistent sign patterns in the residuals,
\end{itemize}
then the linear model might
be missing something and use of another type of model might be suggested.  

For example, in Figure~\ref{fig:ExcelRegression}, even without computing the
numerical values of the residuals, we can see that $r_1$ will be 
slightly positive (the first point has $y_1$ greater than $\hat{m} x_1 + \hat{b}$
since the point $(x_1,y_1)$ lies above the regression line), then $r_2$ is slightly
negative, $r_3$ is slightly more negative than $r_2$, $r_4$ is positive and
finally $r_5$ is quite close to zero since that point comes close to lying 
on the regression line (it's actually slightly postive, but quite a bit less
positive than $r_1$ or $r_4$.  This is more or less the sort of ``random
up and down'' pattern that indicates a linear model is reasonable.  A few 
words of caution are probably in order here, though.  Our comments above
apply most directly when $N$ is large.  $N = 5$ is probably too small
to talk about any sort of patterns.  Moreover, when examining residuals
for a linear model, it is quite easy to start seeing patterns that are
probably only chance variations--our visual perception systems have evolved to 
work that way because there was probably some advantage to perceiving such
patterns even when they are not really there.\footnote{Think of 
the possible value of a \emph{flight response} generated by perceiving
a predator in the trees in the distance, even when there is really nothing there!} 
   

\specialsection{Chapter Project}

\subsection*{Background Information}
The Mauna Loa Observatory (located at an elevation of about 3400 meters on the Mauna Loa volcanic mountain on the 
``big island'' of  Hawaii) is a research station maintained and staffed by the National Oceanic and Atmospheric Administration 
(NOAA), the major agency of the US government that studies weather and climate phenomena.   NOAA maintains a web site, 
for instance, with the National Weather Service day-to-day forecasts and severe weather warnings that form the basis for most local 
weather reporting in the media.   Among the data collected regularly at  Mauna Loa are measurements of atmospheric concentrations of 
a host of trace gasses, including carbon dioxide, or CO2.  
The data set of those measurements goes back to 1958 and is one of the most complete records of the recent evolution of this aspect of the 
Earth's atmosphere.   In this project we will apply the modeling techniques introduced in this chapter to try to understand what this data set is saying about changes 
in atmospheric CO2 over time. 

\subsection*{Important Note}
This is a well-known data set and you can find all sorts of discussions of various aspects of it on the web, if you look.  
I am going to ask all of you \emph{not to look} until after you have worked through at least questions A through D below, though.   
The idea is for you to approach this entirely ``fresh'' and make your own observations and analysis and draw your own conclusions.

\subsection*{Getting Started}
The data we will be looking at is contained in a (large) Excel spreadsheet called {\tt MaunaLoaCO2Data.xls}  that you will download from 
the course homepage.  Begin by getting the spreadsheet and opening it in Excel.  Note the layout: 
\begin{itemize}
\item Column A gives the year the measurement was taken
\item Column B gives the month (1 = January, through 12 = December)
\item Column C gives a decimal equivalent of the middle of the month, so for instance January 1958 is given as 1958.04, 
since 1 month = 1/12 year = .08 year (roughly), and .04 year is about 15 days.
\item Column D gives the average CO2 level observed at Mauna Loa that month in units of parts per million
\item In Column D, if you look closely, you will see that a few of the entries near the start are -99.99.  What do you suppose that means?  
\item In Column E, you will notice that most of the entries are the same as the corresponding entries in Column D, but the -99.99 
entries have been replaced by other values.  These are ``interpolated'' (estimated) values based on the trends from the nearby months.   We will use 
Column E for all our values so that the -99.99's are not included.   
\end{itemize}

\subsection*{Questions}
The first thing you will notice if you look at the CO2  levels is that there is \emph{a lot} of
up-and-down variation.  Is it completely random, though?  And is there an underlying trend?
\vskip 10pt
\noindent
(A)  To start to answer this question:
\vskip 5pt
\begin{enumerate}
\item[(1)] Create scatter plots of the CO2 monthly averages for the calendar years 1965, 1975, 1985 (individually), versus the 
decimal year from Column C.   This will require picking out the correct range of rows in Columns C and E for each of these years, 
and you may want to copy those values to other cells to create the scatter plots.  \vskip 5pt
\item[(2)] Looking at these scatter plots, what do you notice about the way CO2 levels vary over these years?  Describe what happens 
over the course of a typical year, and hypothesize a reason why the annual pattern works this way.  Note:  Mauna Loa is in the Northern 
Hemisphere and typical mixing patterns in the atmosphere mean that most of the air that passes over this location has come from other 
areas in the Northern Hemisphere.   What happens through the months of May, June, July, August in the Northern Hemisphere, and 
how might that affect atmospheric CO2 levels?  \vskip 5pt
\item[(3)]  \emph{Extra Credit}: How might you model the yearly variation of the CO2 readings?  Suggest mathematical 
function(s) that might be useful and how you might apply them.
\end{enumerate}
\vskip 10pt
\noindent
(B) Condensing the Data to a More Manageable Form.
Our goal is to model how atmospheric CO2 levels have been changing over this period (but on the year-to-year level, not 
on the much more variable month-to-month level).  This will be much more manageable if we identify some way to compute 
a ``summary value'' for each year to use as the representative CO2 level for that year.   \vskip 5pt
\begin{enumerate}
\item[(1)]  Identify (at least) three different ways that might be used to produce that sort of ``summary value'' and describe why
they would be suitable.  \vskip 5pt
\item[(2)]  Choose one of your proposed ways to do this and give a reason for why you think  that will be a reasonable way to ``condense'' the data for each year. \vskip 5pt
\item[(3)] Create new columns in your spreadsheet giving the number of years since 1959, and your summary CO2 value for the year.    Since 
we don't have complete values for the years 1958 or 2017, just use the years 1959 ? 2016 (58 years in all).
\end{enumerate}
\vskip 10pt
\noindent
(C)  ``Let the modeling begin!'' \vskip 5pt
\begin{enumerate}
\item[(1)]	Using Excel, fit a linear model to your ``condensed'' data set and record your results.  Give
the equation of the regression line as a function of the years since 1959 (that is, $x = 0$ correspond to 1959, 
$x = 1$ corresponds to 1960, and so forth).  Also give the $R^2$ value reported by Excel as a measure of goodness of the fit, and discuss the residuals for the linear model (in particular, is there a consistent pattern there)?  \vskip 5pt
\item[(2)]  What does your model predict concerning the CO2 level in 2020?  (This is slightly outside the interval 1959 to 2016 of course, but not too far outside. So extrapolation from 
the linear model is at least a possibility!)
\end{enumerate}
\vskip 10pt
\noindent
(D) Atmospheric CO2 levels are of concern, of course, because of the ``greenhouse gas'' properties of this compound--the way 
atmospheric CO2 can trap energy from reflected solar radiation and increase temperatures near the surface of the Earth.  
Some greenhouse effect is necessary for life on Earth, of course (our water-based form of life could not exist at the temperatures 
that would prevail with no greenhouse effect at all because all water would be frozen as ice).   But have there been times in the past when CO2 concentrations were 
significantly higher than they are now?  What were the Earth's climate and sea levels like then?  (This may require some 
research--be sure to give the sources you used to compile your information.)
\vskip 10pt
\noindent
Assignment:  Submit your edited Excel spreadsheet with the data and write up answers to the questions above
in a separate document.

\begin{xcb}{Chapter Exercises}
\begin{enumerate}
\item Consider the function defined by $f(x) = \sqrt{x - 6}$.  What is the domain of $f$ according to
the ``rule of thumb'' discussed in the text.  
\vskip 5pt
\item Same question as in (1) for $g(x) = \frac{x}{x^2 - 5x + 6}$.
\vskip 5pt
\item Does the equation $x^2 + y^2 = 9$ define $y$ as a function of $x$?   (Hint: What curve in the 
plane is defined by that equation?)  If yes, plot that function.  If not, 
explain why not and say how you could find subsets of the curve defined by this equation
that are graphs of functions of $x$ and give their domains.
\vskip 5pt
\item We discussed an example of a family of circles with areas in an \emph{arithmetic} progression in 
Figure~\ref{fig:ArithCircs}.  Formally, an arithmetic progression is any sequence of numbers of the
form 
$$a, a + d, a + 2\cdot d, a + 3\cdot d, \ldots, a + n\cdot d$$
where $a,d$ are fixed numbers.  Show how such an arithmetic progression can be obtained
as the values $f(0), f(1), \ldots, f(n)$ for some linear function.  (What is the formula of that linear function?)
\vskip 5pt
\item Given that the line plotted in Figure~\ref{fig:LinearSlope} passes through the points $(1,5)$ and $(2,7)$,
what is the equation of the corresponding linear function $f = m x + b$? 
\vskip 5pt
\item Here are some additional problems concerning equations of lines.
\vskip 5pt
\begin{enumerate}
\item What is the equation of the line passing through the point $(3.2,5.2)$ and parallel 
to the line given by $y = 7x + 1$?  (Hint:  What is true about the slopes of parallel lines?) \vskip 5pt
\item  What is the equation of the line passing through the point $(0,1)$ and perpendicular 
to the line from part (a)?  (Hint: The slopes of perpendicular lines are negative reciprocals 
of one another.)
\end{enumerate}
\vskip 5pt
\item In least squares regression, we minimize the sum of the squares of the 
vertical distances $y_i - (m x_i + b)$ in terms of $m, b$ to find the regression 
line.  Another way to think about the distance from a point to a line is to consider
the straight line distance from the point to the \emph{closest point on the 
line}.\footnote{That is different unless the line is horizontal!}
In a formula, if the point $P$ is $(x_0,y_0)$ and the line $L$ is given by 
an equation written in the form $A x + B y + C = 0$, 
the distance from $P$ to $L$ measured this way is 
$$d(P,L) = \frac{|A x_0 + B y_0 + C|}{\sqrt{A^2 + B^2}}$$
Find the distance from $P = (1,2)$ to the line $y = 5x + 7$ measured this way.
\vskip 5pt
\begin{table}[ht]
\renewcommand\arraystretch{1.5}
\caption{Three tabulated functions}
\label{Exercise}
\noindent\[
\begin{array}{|c|c|c|c|c|c|c|}
\hline
x & 1.1 & 2.4 & 3.7 & 5.0 & 6.3 & 7.6\\ 
\hline
f(x) &1.445 & 1.844& 2.168& 2.449& 2.702& 2.933 \\ 
\hline
g(x) &1.525& -1.400& -4.325& -7.250& -10.175& -13.100\\
\hline
h(x) &1.111& 1.257& 1.423& 1.611& 1.823& 2.063\\ 
\hline
\end{array}
\]
\end{table}
\item Only one of the functions tabulated in Table~\ref{Exercise} is linear.
Identify which one it is and find the equation of the linear function.
\vskip 5pt
\item Refer to Exercise~\ref{ex:Kudzu} from Chapter 1.
Construct a linear model for the area in the U.S. covered
by kudzu as a function of time in years since 1876.  What are the units
of the slope and what does the slope represent?
\vskip 5pt
\item Compute the residuals for the linear model from Example~\ref{HypData} and discuss
them in relation to Figure~\ref{fig:HypDataPlot}.   
\vskip 5pt
\item Use Excel to find the least squares regression line for the hybrid vehicle market share 
data from Table~\ref{HybridMarketShare}.  What is the slope of the regression line, what are its 
units, and what does that slope represent?
\vskip 5pt
\item In this problem we continue the study of the California population data from 
Table~\ref{CalPop}.  \vskip 5pt
\begin{enumerate}
\item Using Excel (or doing the calculations by hand), find the regression line for
the subset of the data set corresponding to the years 1900, 1910, and 1920.  How
does the slope compare to the slope found for the data from 1870 through 1900?  Does
this match what you thought by looking at Figure~\ref{fig:CalPops}? \vskip 5pt
\item Now find the regression line for the whole data set (the years 1870--1920).
Plot this together with the point plot (or Excel scatter plot) and discuss the results. 
\item Find the residuals for this linear model and generate a point plot (scatter plot).
Is there a systematic trend? 
\end{enumerate}
\vskip 5pt
\item By researching U.S. Federal Census data online, \vskip 5pt
\begin{enumerate}
\item  Find the population of 
your home state for the censuses in 1900 through 2010.  (If your home state only 
joined the Union after 1900, use only the census data for the years since
statehood.)  \vskip 5pt
\item Find the least squares regression line and the $R^2$ statistic.  \vskip 5pt
\item Compute the residuals for your linear model and generate a point
plot (scatter plot).  Examine them for any patterns as we discussed in the text.  \vskip 5pt
\item What is your conclusion? 
Does a linear model fit your state's change in population over this 
period (at least approximately)?  
\end{enumerate}


\end{enumerate}
\end{xcb}
\endinput
