Abstract
A new algorithm is developed to train feed-forward neural networks for non-linear input-to-output mappings with small incomplete data in arbitrary distributions. The developed Training-EStimation-Training (TEST) algorithm consists of 3 steps, i.e., (1) training with the complete portion of the training data set, (2) estimation of the missing attributes with the trained neural networks, and (3) re-training the neural networks with the whole data set. Error back propagation is still applicable to estimate the missing attributes. Unlike other training methods with missing data, it does not assume data distribution models which may not be appropriate for small training data. The developed TEST algorithm is first tested for the Iris benchmark data. By randomly removing some attributes from the complete data set and estimating the values latter, accuracy of the TEST algorithm is demonstrated. Then it is applied to the Diabetes benchmark data, of which about 50% contains missing attributes. Compared with other existing algorithms, the proposed TEST algorithm results in much better recognition accuracy for test data.
Similar content being viewed by others
References
LeCun, Y., Denker, J. S. and Solla, S. A.: Optimal brain damage, In: D. Touretzky (ed.), Advances in Neural Information Processing Systems 2, Morgan Kaufmann, 1990, pp. 598–605.
Weigend, A. S., Rumelhart, D. E. and Huberman, B. A.: Generalization by weight elimination with application to forecasting, In: R. Lippmann, J. Moody, and D. Touretzky (eds.), Advances in Neural Information Processing Systems 3, San Mateo, CA, 1991, pp. 875–882.
Jeong, D. G. and Lee, S. Y.: Merging Hebbian and error back-propagation learning rules for robust classifications, Neural Networks 9 (1996), 1213–1222.
Timm, H. and Klawonn, K.: Classification of data with missing data, Proc. 6th European Congress on Intelligent Techniques and Soft Computing 1 (September 7, 1998), 639–643, Aachen, Germany.
Little, R. J. A.: Regression with missing X's: a review, Journal of the American Statistical Association 87 (1992), 1227–1237.
Beale, E. M. L. and Little, R. J. A.: Missing values in multivariate analysis, Journal of Royal Statistical Society, B37 (1975), 129–145.
Dempster, A. P., Laird, N. M. and Rubin, D. B.: Maximum likelihood from incomplete data via the EM algorithm, Journal of Royal Statistical Society B39 (1977), 1–38.
Chen, C. F.: A Bayesian approach to nested missing-data problems, In: P. Goel and A. Zellner (eds.), Bayesian Inference and Decision Techniques, P. Goel and A. Zellner (eds.), Elsevier Press, New York, 1986, pp. 355–361.
Heitjan, D. F. and Little, R. J. A.: Multiple imputation in the fatal accident reporting system, Applied Statistics 40 (1991), 13–29.
Lange, K., Little, R. J. A. and Taylor, J. M. G.: Robust statistical inference using the t distribution, Journal of the American Statistical Association 84 (1989), 881–896.
Tresp, V., Ahmad, S. and Neuneier, R.: Training neural networks with deficient data, In: J. Cowan, G. Tesauro and J. Alspector (eds.), Advances in Neural Information Processing Systems 6 (1994), Morgan Kaufmann, pp. 128–135.
Jordan, M. I. and Jacobs, R. A.: Hierarchical mixtures of experts and the EM algorithm, Neural Computation 6 (1994), 181–214.
Amari, S.: Information geometry of the EM and em algorithms for neural networks, Neural Networks 8 (1995), 1379–1408.
Ghahramani, Z. and Jordan, M.: Supervised learning from incomplete data via an EMapproach, In: J. Cowan, G. Tesauro, and J. Alspector (eds.), Advances in Neural Information Processing Systems 6 (1994), Morgan Kaufmann, pp. 120–127.
Holmström, L. and Koistinen, P.: Using additive noise in backpropagation training, IEEE Trans. Neural Networks, 3 (1992), 24–38.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Yoon, SY., Lee, SY. Training Algorithm with Incomplete Data for Feed-Forward Neural Networks. Neural Processing Letters 10, 171–179 (1999). https://doi.org/10.1023/A:1018772122605
Issue Date:
DOI: https://doi.org/10.1023/A:1018772122605