Theory choice, non-epistemic values, and machine learning

Abstract

I use a theorem from machine learning, called the “No Free Lunch” theorem (NFL) to support the claim that non-epistemic values are essential to theory choice. I argue that NFL entails that predictive accuracy is insufficient to favor a given theory over others, and that NFL challenges our ability to give a purely epistemic justification for using other traditional epistemic virtues in theory choice. In addition, I argue that the natural way to overcome NFL’s challenge is to use non-epistemic values. If my argument holds, non-epistemic values are entangled in theory choice regardless of human limitations and regardless of the subject matter. Thereby, my argument overcomes objections to the main lines of argument revealing the role of values in theory choice. At the end of the paper, I argue that, contrary to common conception, the epistemic challenge arising from NFL is distinct from Hume’s problem of induction and other forms of underdetermination.

Appendix: The no free lunch theorem(s)

“No Free Lunch” is the name of a family of theorems. Differences between No Free Lunch theorems include differences between the kinds of algorithms they consider. For example, initially No Free Lunch theorems were proven for optimization algorithms (Wolpert and Macready 1993). Wolpert (1996, 2001, 2012) proves No Free Lunch theorems for supervised learning algorithms, and this is what I have focused on in this paper. Schaffer (1994) gives an elegant formulation of Wolpert’s main No Free Lunch Theorem for classification learning algorithms, based on a preprint of Wolpert (1996). In this appendix, I state Schaffer’s formulation to illustrate what NFL theorems say more formally (Schaffer calls it the “Law of Conservation of Generalization of Performance”). See Montanez (2017, chapter 2) for a review of various No Free Lunch results, and see Schaffer (1994) and Wolpert (1996) for a proof of the theorem which I will state here.

We start with defining cases in a classification problem. Each case in a classification problem, $A_{i}$, is a vector of attributes. For simplicity, we assume that each component in the vector is a finite number. $\left\{ {A_{1} , \ldots ,A_{m} } \right\}$ is the set of all possible attribute vectors, where m is finite. C is a class probability vector, which defines the relationship between attribute vectors and classes. Each component of C, $C_{\text{i}}$, is the probability that a case with attribute $A_{i}$ belongs to class 1. We assume that data is generated in the same way in training and testing a learner. Attribute vectors are sampled with replacement according to an arbitrary distribution D and a class is assigned to them using C. We also assume that the training set contains n samples. A learning situation S is a triple (D, C, n).

The Generalization Accuracy of a learner (GA_L) is the expected prediction performance of a learner on cases with attribute vectors not represented in the training set. For example, the generalization accuracy of a random guesser in a two-class problem is 1/2 for every D and C. We use the generalization accuracy of a random guesser as a baseline and define Generalization Performance of a learner (GP_L) the difference between its generalization accuracy and the generalization accuracy of a random guesser:

$$GP_{L} = GA_{L} - GA_{random\,guesser}$$

Generalization performance greater than zero means better than chance performance. $GP_{L} \left( S \right)$ is the generalization performance of learner L in learning situation S.

Using the notation above we can write Schaffer’s Law of Conservation of Generalization Performance:

$$\mathop \sum \limits_{S} GP_{L} \left( S \right) = 0,\;{\text{for}}\;{\text{every}}\;{\text{D}},{\text{n}}$$

In words, this law says that any positive performance by a learner in a certain learning situation must be exactly balanced by negative performance in other learning situations.

If we allow for the possibility of noise, then the law is properly written with an integral instead of a summation:

$$\mathop \int \limits_{S}^{{}} GP_{L} \left( S \right)ds = 0,\;{\text{for}}\;{\text{every}}\;{\text{D}},{\text{n}}$$

In this case, the components of C are taken from the real interval [0,1] and the integral runs over the space [0,1]^m of class probability vector. Without noise, the components of C are taken from {0,1} and the summation runs over 2 ^m possible class probability vectors.

From the conservation law, it follows that all learners have the same average generalization performance if we average over all possible learning situations (or, as I put it, that all algorithms have the same expected error if we make no assumptions about the problem we are trying to solve). Here’s why.

For any learner:

$$\mathop \sum \limits_{S} GP_{L} \left( S \right) = \mathop \sum \limits_{S} \left( {GA_{L} \left( {\text{S}} \right) - GA_{random\,guesser} \left( S \right)} \right) = 0$$

Add $\mathop \sum \nolimits_{S} GA_{random\,guesser} \left( S \right)$ to both sides and get:

$$\mathop \sum \limits_{S} GA_{L} \left( S \right) = \mathop \sum \limits_{S} GA_{random \,guesser} \left( S \right)$$

Divide by the number of learning situations and get the formulation that was used in this paper—that the average generalization performance of any learner L is equal, and in particular equal to that of the random guesser:

$$\frac{{\mathop \sum \nolimits_{S} GA_{L} \left( S \right)}}{\# S} = \frac{{\mathop \sum \nolimits_{S} GA_{random\, guesser} \left( S \right)}}{\# S}$$