Abstract
Dimension reduction for regression (DRR) deals with the problem of finding for high-dimensional data such low-dimensional representations, which preserve the ability to predict a target variable. We propose doing DRR using a neural network with a low-dimensional “bottleneck” layer. While the network is trained for regression, the bottleneck learns a low-dimensional representation for the data. We compare our method to Covariance Operator Inverse Regression (COIR), which has been reported to perform well compared to many other DRR methods. The bottleneck network compares favorably with COIR: it is applicable to larger data sets, it is less sensitive to tuning parameters and it gives better results on several real data sets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Parviainen, E.: Deep bottleneck classifiers in supervised dimension reduction. In: Proc. of ICANN 2010 (to appear, 2010)
Kim, M., Pavlovic, V.: Dimensionality reduction using covariance operator inverse regression. In: Proc. of CVPR (Computer Vision and Pattern Recognition) (2008)
Kim, M., Pavlovic, V.: Covariance operator based dimensionality reduction with extension to semi-supervised learning. In: Proc. of AISTATS. JMLR: W&CP, vol. 5, pp. 280–287 (2009)
Li, K.C.: Sliced inverse regression for dimension reduction. Journal of the American Statistical Association 86(414), 316–327 (1991)
Fukumizu, K., Bach, F.R., Jordan, M.I.: Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. Journal of Machine Learning Research 5, 73–99 (2004)
Fukumizu, K., Bach, F.R., Jordan, M.I.: Kernel dimension reduction in regression. The Annals of Statistics 37(1), 1871–1905 (2009)
Li, K.C.: High dimensional data analysis via the SIR/PHD approach. Lecture notes in progress (April 2000), http://www.stat.ucla.edu/~kcli/
Nilsson, J., Sha, F., Jordan, M.I.: Regression on manifolds using kernel dimension reduction. In: Proc. of ICML (2007)
Moon, K., Pavlović, V.: Regression using Gaussian Process manifold kernel dimensionality reduction. In: Proc. of MLSP (Machine Learning for Signal Processing), pp. 14–19 (2008)
DeMers, D., Cottrell, G.: Non-linear dimensionality reduction. In: Proc. of NIPS, pp. 580–587 (1993)
Schneider, J.J., Kirkpatrick, S.: Stochastic optimization. Springer, Heidelberg (2006)
Asuncion, A., Newman, D.: UCI machine learning repository (2007)
Georghiades, A.S., Belhumeur, P.N., Kriegman, D.: From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(6), 643–660 (2001)
Tsanas, A., Little, M.A., McSharry, P.E., Ramig, L.O.: Accurate telemonitoring of Parkinson s disease progression by non-invasive speech tests. IEEE Transactions on Biomedical Engineering (2009)
Redmond, M.A., Baveja, A.: A data-driven software tool for enabling cooperative information sharing among police departments. European Journal of Operational Research 141, 660–678 (2002)
Yeh, I.C.: Modeling of strength of high performance concrete using artificial neural networks. Cement and Concrete Research 28(12), 1797–1808 (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Parviainen, E. (2010). Dimension Reduction for Regression with Bottleneck Neural Networks. In: Fyfe, C., Tino, P., Charles, D., Garcia-Osorio, C., Yin, H. (eds) Intelligent Data Engineering and Automated Learning – IDEAL 2010. IDEAL 2010. Lecture Notes in Computer Science, vol 6283. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15381-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-15381-5_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-15380-8
Online ISBN: 978-3-642-15381-5
eBook Packages: Computer ScienceComputer Science (R0)