Linear regression with straight lines

Linear regression is a widely used mathematical technique in which a certain function (often a straight line or polynomial) is used to fit a set of data or raw measurements, showing a correlation between 2 or more variables. The idea is that we choose a certain function which represents a measure of the error between the “line of best fit” and the actual data, and then we use techniques from Calculus to minimize that error by choosing the most optimum parameters for that line of best fit. In this article, I will walk the reader through the mathematics of linear regression with straight lines using the technique of least squares analysis (LSA).

*Note: it is assumed the reader is familiar with some multivariable calculus (e.g. partial derivatives) and linear algebra.

Mathematical derivation

We begin our discussion by the most common and basic of all regression types: straight line regression. As its name suggests, given a data set S(x), our goal is to find the equation of a straight line y(x) which minimises the distance between each point on the line and the data set S(x). In general, the equation of a straight line is written as

where the parameters b0 and b1 are constants to be determined. In order to calculate these parameters, we create a function that represents the sum of the squared error E for each point in y(x) and the data set S(x)

The idea is then to to minimise the function E(b0, b1) in terms of all of its parameters…