Linear and Logistic Regression

Abstract

Linear regression is the most representative “machine learning” method to build models for value prediction and classification from training data. It offers a study in contrasts:

• Linear regression has a beautiful theoretical foundation yet, in practice, this algebraic formulation is generally discarded in favor of faster, more heuristic optimization.

• Linear regression models are, by definition, linear. This provides an opportunity to witness the limitations of such models, as well as develop clever techniques to generalize to other forms.

• Linear regression simultaneously encourages model building with hundreds of variables, and regularization techniques to ensure that most of them will get ignored.

An unsophisticated forecaster uses statistics as a drunken man uses lamp posts – for support rather than illumination.

– Andrew Lang