Abstract
While a variety of multiple classifier systems have been studied since at least the late 1950’s, this area came alive in the 90’s with significant theoretical advances as well as numerous successful practical applications. This article argues that our current understanding of ensemble-type multiclassifier systems is now quite mature and exhorts the reader to consider a broader set of models and situations for further progress. Some of these scenarios have already been considered in classical pattern recognition literature, but revisiting them often leads to new insights and progress. As an example, we consider how to integrate multiple clusterings, a problem central to several emerging distributed data mining applications. We also revisit output space decomposition to show how this can lead to extraction of valuable domain knowledge in addition to improved classification accuracy.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Selfridge, O.G.: Pandemonium: a paradigm for learning. Proc. of Symp. held at the National Physical Lab. (1958) 513–526
Nilsson, N.J.: Learning Machines: Foundations of Trainable Pattern-Classifying Systems. McGraw Hill, NY (1965)
Kanal, L.: Patterns in pattern recognition. IEEE Trans. Information Theory IT-20 (1974) 697–722
Minsky, M.: Logical versus analogical or symbolic versus connectionist or neat versus scruffy. AI Magazine 12 (1991) 34–51
Granger, C.W.J.: Combining forecasts-twenty years later. Journal of Forecasting 8 (1989) 167–173
French, S.: Group consensus probability distributions: A critical survey. In Bernardo et al., J.M., ed.: Bayesian Statistics 2. North Holland, New York (1985)
Benediktsson, J.A., Swain, P.H.: Consensus theoretic classification methods. IEEE Transactions on Systems, Man, and Cybernetics 22 (1992) 688–704
Luo, R.C., Kay, M.G.: Multisensor integration and fusion in intelligent systems. IEEE Transactions on Systems, Man, and Cybernetics 19 (1989) 901–931
Narendra, K., Balakrishnan, J., Ciliz, K.: Adaptation and learning using multiple models, switching and tuning. IEEE Control Systems Magazine (1995) 37–51
Murray-Smith, R., Johansen, T.A.: Multiple Model Approaches to Modelling and Control. Taylor and Francis, UK (1997)
Ghosh, J., Deuser, L., Beck, S.: A neural network based hybrid system for detection, characterization and classification of short-duration oceanic signals. IEEE Jl. of Ocean Engineering 17 (1992) 351–363
Sharkey, A.: On combining artificial neural networks. Connection Science 8 (1996) 299–314
Kittler, J., Hatef, M., Duin, R., Matas, J.: On combining classifiers. IEEE Trans. Pattern Analysis and Machine Intelligence 20 (1998) 226–239
Tumer, K.: Linear and order statistics combiners for reliable pattern classification. PhD thesis, Dept. of ECE, Univ. of Texas at Austin, Dec. (1996)
Perrone, M.P.: Improving Regression Estimation: Averaging Methods for Variance Reduction with Extensions to General Convex Measure Optimization. PhD thesis, Brown University (1993)
Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation and active learning. In G. Tesauro, D.T., Leen, T., eds.: Advances in Neural Information Processing Systems-7. (1995) 231–238
Hansen, L.K., Salamon, P.: Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence 12 (1990) 993–1000
Tumer, K., Ghosh, J.: Analysis of decision boundaries in linearly combined neural classifiers. Pattern Recognition 29 (1996) 341–348
Tumer, K., Ghosh, J.: Ensemble correlation and error reduction in ensemble classifiers. Connection Science 8 (1996) 385–404
Dietterich, T.G.: Ensemble methods in machine learning. In Kittler, J., Roli, F. eds.: Multiple Classifier Systems. LNCS Vol. 1857, Springer (2001) 1–15
Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Technical Report Dept. of Statistics, Stanford University. (1998)
Ghosh, J., Beck, S., Chu, C.: Evidence combination techniques for robust classification of short-duration oceanic signals. In: SPIE Conf. on Adaptive and Learning Systems, SPIE Proc. Vol. 1706. (1992) 266–276
Xu, L., Krzyzak, A., Suen, C.Y.: Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Transactions on Systems, Man and Cybernetics 22 (1992) 418–435
Ho, T.K., Hull, J.J., Srihari, S.N.: Decision combination in multiple classifier systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 16 (1994) 66–76
Kittler, J., Roli (Eds.), F.: Multiple Classifier Systems. LNCS Vol. 1857, Springer (2001)
Jordan, M., Jacobs, R.: Hierarchical mixture of experts and the EM algorithm. Neural Computation 6 (1994) 181–214
Holmstrom, L., Koistinen, P., Laaksonen, J., Oja, E.: Neural and statistical classifiers-taxonomy and two case studies. IEEE Transactions on Neural Networks 8 (1997) 5–17
Ramamurti, V., Ghosh, J.: Structurally adaptive modular networks for non-stationary environments. IEEE Transactions on Neural Networks 10 (1999) 152–60
Woods, K., Kegelmeyer, W.P., Bowyer, K.: Combination of multiple classifiers using local accuracy estimates. IEEE Transactions on Pattern Analysis and Machine Intelligence 19 (1997) 405–410
Jain, A.K., Dubes, R.C.: Algorithms for Clustering Data. Prentice Hall, New Jersey (1988)
Barthelemy, J.P., Laclerc, B., Monjardet, B.: On the use of ordered sets in problems of comparison and consensus of classifications. Jl. of Classification 3 (1986) 225–256
Neumann, D.A., Norton, V.T.: Clustering and isolation in the consensus problem for partitions. Journal of Classification 3 (1986) 281–298
Kim, J., Warnow, T.: Tutorial on phylogenetic tree estimation. In: Intelligent Systems for Molecular Biology, Heidelberg. (1999)
Johnson, E., Kargupta, H.: Collective, hierarchical clustering from distributed, heterogeneous data. In Zaki, M., Ho, C., eds.: Large-Scale Parallel KDD Systems. Volume 1759 of LNCS., Springer-Verlag (1999) 221–244
Fred, A.L.N., Jain, A.K.: Data clustering using evidence accumulation. In: Proc. ICPR. (2002) to appear
Bollacker, K.D., Ghosh, J.: Effective supra-classifiers for knowledge base construction. Pattern Recognition Letters 20(11–13) (1999) 1347–52
Kargupta, H., Chan, P., eds.: Advances in Distributed and Parallel Knowledge Discovery. AAAI/MIT Press, Cambridge, MA (2000)
Strehl, A., Ghosh, J.: Cluster ensembles — a knowledge reuse framework for combining partitionings. In: Proceedings of AAAI 2002, Edmonton, Canada, AAAI (2002) in press.
Strehl, A., Ghosh, J., Mooney, R.: Impact of similarity measures on web-page clustering. In: Proc. AAAI Workshop on AI for Web Search (AAAI 2000), Austin, AAAI (2000) 58–64
Kumar, S.: Modular learning through output space decomposition. PhD thesis, Dept. of ECE, Univ. of Texas at Austin, Dec. (2000)
Anand, R., Methrotra, K., Mohan, C.K., Ranka, S.: Efficient classification for multiclass problems using modular neural networks. IEEE Transactions on Neural Networks 6 (1995) 117–125
Berenstein, C., Kanal, L.N., Lavine, D.: Consensus rules. In Kanal, L.N., Lemmer, J.F., eds.: Uncertainty in Artificial Intelligence. North Holland, New York (1986)
Hastie, T., Tibshirani, R.: Classification by pairwise coupling. In Jordan, M.I., Kearns, M.J., Solla, S.A., eds.: Advances in Neural Information Processing Systems. Volume 10., The MIT Press (1998)
Kumar, S., Crawford, M.M., Ghosh, J.: A versatile framework for labelling imagery with a large number of classes. In: Proc. IJCNN. (1999)
Dietterich, T.G., Bakiri, G.: Solving multiclass learning problems via errorcorrecting output codes. Jl. of Artificial Intelligence Research 2 (1995) 263–286
Ricci, F., Aha, D.: Extending local learners with errorcorrecting output codes. Technical report, Technical Report 9701-08, IRST (1997)
Chakrabarti, S., Dom, B., Agrawal, R., Raghavan, P.: Scalable feature selection, classication and signature generation for organizing large text databases into hierarchical topic taxonomies. VLDB Journal 7 (1998) 163–178
Kumar, S., Ghosh, J., Crawford, M.M.: Hierarchical fusion of multiple classifiers for hyperspectral data analysis”, Pattern Analysis and Applications, spl. Issue on Fusion of Multiple Classifiers 5 (2002) In Press
Thrun, S., Pratt, L.: Learning To Learn. Kluwer Academic, Norwell, MA (1997)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ghosh, J. (2002). Multiclassifier Systems: Back to the Future. In: Roli, F., Kittler, J. (eds) Multiple Classifier Systems. MCS 2002. Lecture Notes in Computer Science, vol 2364. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45428-4_1
Download citation
DOI: https://doi.org/10.1007/3-540-45428-4_1
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-43818-2
Online ISBN: 978-3-540-45428-1
eBook Packages: Springer Book Archive