Model Selection and Estimation Via Subjective User Preferences

Abstract

Subjective opinions of domain experts are often encountered in data analysis projects. Often, it is difficult to express the experts’ opinions in model form or integrate their professional knowledge in the analysis. In this paper, we approach the problem directly in the context of model selection and estimation: we ask the expert for subjective preferences between readily computed model solutions, and compute an optimal solution based on the recorded opinions. We consider the pre-computed models as graph nodes, and calculate the preferential relations between the nodes based on the recorded opinions as conditional probabilities. Using a random surfer model from the Web analysis community, we compute the stationary distribution of the preferences. The stationary distribution can be used in model selection by selecting the most probable model or in model estimation by averaging over the models according to their posterior probabilities. We present a real-life application in a regression problem of tree-ring width series data.