Principal Components Identify MLP Hidden Layer Size for Optimal Generalisation Performance

Abstract

One of the major concerns when implementing a supervised artificial neural network solution to a classification or prediction problem, is the network’s performance on unseen data. The phenomenon of the network overfitting the training data, is understood and reported in the literature. Most researchers recommend a ‘trial and error’ approach to selecting the optimal number of weights for the network, which is time consuming, or start with a large network and prune to an optimal size. Current pruning techniques based on approximations of the Hessian matrix of the error surface are computationally intensive and prone to severe approximation errors if a suitable minimal training error has not been achieved. We propose a novel and simple design heuristic for a three layer multi-layer perceptron (MLP) based on an eigenvalue decomposition of the covariance matrix of the middle layer output. This technique identifies the neurons which are contributing to the redundancy of data through the network and as such are additional effective network parameters which have a deleterious effect on the classifier surface smoothness. This technique identifies redundancy in the network data and so is not dependant on the network training having reached a minimal error value making the Levenberg-Marquardt approximation valid. We report on simulations using the double-convex benchmark which show the utility of the proposed method.