Skip to main content
SpringerLink
Log in

Decision Trees

  • Chapter
Data Mining and Knowledge Discovery Handbook

Abstract

Decision Trees are considered to be one of the most popular approaches for representing classifiers. Researchers from various disciplines such as statistics, machine learning, pattern recognition, and Data Mining have dealt with the issue of growing a decision tree from available data. This paper presents an updated survey of current methods for constructing decision tree classifiers in a top-down manner. The chapter suggests a unified algorithmic framework for presenting these algorithms and describes various splitting criteria and pruning methodologies.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 229.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Almuallim H., An Efficient Algorithm for Optimal Pruning of Decision Trees. Artificial Intelligence 83(2): 347–362, 1996.

    Article  Google Scholar 

  • Almuallim Hv and Dietterich T.G., Learning Boolean concepts in the presence of many irrelevant features. Artificial Intelligence, 69:1–2, 279–306, 1994.

    Article  MathSciNet  Google Scholar 

  • Alsabti K., Ranka S. and Singh V., CLOUDS: A Decision Tree Classifier for Large Datasets, Conference on Knowledge Discovery and Data Mining (KDD-98), August 1998.

    Google Scholar 

  • Attneave F, Applications of Information Theory to Psychology. Holt, Rinehart and Winston, 1959.

    Google Scholar 

  • Baker E., and Jain A. K., On feature ordering in practice and some finite sample effects. In Proceedings of the Third International Joint Conference on Pattern Recognition, pages 45–49, San Diego, CA, 1976.

    Google Scholar 

  • BenBassat M., Myopic policies in sequential classification. IEEE Trans. on Computing, 27(2): 170–174, February 1978.

    MathSciNet  Google Scholar 

  • Bennett X. and Mangasarian O.L., Multicategory discrimination via linear programming. Optimization Methods and Software, 3:29–39, 1994.

    Google Scholar 

  • Bratko I., and Bohanec M., Trading accuracy for simplicity in decision trees, Machine Learning 15: 223–250, 1994.

    MATH  Google Scholar 

  • Breiman L., Friedman J., Olshen R., and Stone C.. Classification and Regression Trees. Wadsworth Int. Group, 1984.

    Google Scholar 

  • Brodley C. E. and Utgoff. P. E., Multivariate decision trees. Machine Learning, 19:45–77, 1995.

    MATH  Google Scholar 

  • Buntine W., Niblett T, A Further Comparison of Splitting Rules for Decision-Tree Induction. Machine Learning, 8: 75–85, 1992.

    Google Scholar 

  • Catlett J., Mega induction: Machine Learning on Vary Large Databases, PhD, University of Sydney, 1991.

    Google Scholar 

  • Chan P.K. and Stolfo S.J, On the Accuracy of Meta-learning for Scalable Data Mining, J. Intelligent Information Systems, 8:5–28, 1997.

    Article  Google Scholar 

  • Crawford S. L., Extensions to the CART algorithm. Int. J. of ManMachine Studies, 31(2): 197–217, August 1989.

    Article  MathSciNet  Google Scholar 

  • Dietterich, T. G., Kearns, M., and Mansour, Y., Applying the weak learning framework to understand and improve C4.5. Proceedings of the Thirteenth International Conference on Machine Learning, pp. 96–104, San Francisco: Morgan Kaufmann, 1996.

    Google Scholar 

  • Duda, R., and Hart, P., Pattern Classification and Scene Analysis, New-York, Wiley, 1973.

    MATH  Google Scholar 

  • Esposito E, Malerba D. and Semeraro G., A Comparative Analysis of Methods for Pruning Decision Trees. EEE Transactions on Pattern Analysis and Machine Intelligence, 19(5):476–492, 1997.

    Article  Google Scholar 

  • Fayyad U., and Irani K. B., The attribute selection problem in decision tree generation. In proceedings of Tenth National Conference on Artificial Intelligence, pp. 104–110, Cambridge, MA: AAAI Press/MIT Press, 1992.

    Google Scholar 

  • Ferri C, Flach P., and Hernandez-Orallo J., Learning Decision Trees Using the Area Under the ROC Curve. In Claude Sammut and Achim Hoffmann, editors, Proceedings of the 19th International Conference on Machine Learning, pp. 139–146. Morgan Kaufmann, July 2002

    Google Scholar 

  • Fifield D. J., Distributed Tree Construction From Large Datasets, Bachelor’s Honor Thesis, Australian National University, 1992.

    Google Scholar 

  • Freitas X., and Lavington S. H., Mining Very Large Databases With Parallel Processing, Kluwer Academic Publishers, 1998.

    Google Scholar 

  • Friedman J. H., A recursive partitioning decision rule for nonparametric classifiers. IEEE Trans, on Comp., C26:404–408, 1977.

    MATH  Google Scholar 

  • Friedman, J. H., “Multivariate Adaptive Regression Splines”, The Annual Of Statistics, 19, 1–141, 1991.

    MATH  Google Scholar 

  • Gehrke J., Ganti V., Ramakrishnan R., Loh W., BOAT-Optimistic Decision Tree Construction. SIGMOD Conference 1999: pp. 169–180, 1999.

    Google Scholar 

  • Gehrke J., Ramakrishnan R., Ganti V., RainForest-A Framework for Fast Decision Tree Construction of Large Datasets, Data Mining and Knowledge Discovery, 4,2/3) 127–162, 2000.

    Article  Google Scholar 

  • Gelfand S. B., Ravishankar C. S., and Delp E. J., An iterative growing and pruning algorithm for classification tree design. IEEE Transaction on Pattern Analysis and Machine Intelligence, 13(2): 163–174, 1991.

    Article  Google Scholar 

  • Gillo M. W., MAID: A Honeywell 600 program for an automatised survey analysis. Behavioral Science 17: 251–252, 1972.

    Google Scholar 

  • Hancock T. R., Jiang T, Li M., Tromp J., Lower Bounds on Learning Decision Lists and Trees. Information and Computation 126(2): 114–122, 1996.

    Article  MathSciNet  MATH  Google Scholar 

  • Holte R. C, Very simple classification rules perform well on most commonly used datasets. Machine Learning, 11:63–90, 1993.

    Article  MATH  Google Scholar 

  • Hyafil L. and Rivest R.L., Constructing optimal binary decision trees is NP-complete. Information Processing Letters, 5(1):15–17, 1976

    Article  MathSciNet  MATH  Google Scholar 

  • Janikow, C.Z., Fuzzy Decision Trees: Issues and Methods, IEEE Transactions on Systems, Man, and Cybernetics, Vol. 28, Issue 1, pp. 1–14. 1998.

    Google Scholar 

  • John G. H., Robust linear discriminant trees. In D. Fisher and H. Lenz, editors, Learning From Data: Artificial Intelligence and Statistics V, Lecture Notes in Statistics, Chapter 36, pp. 375–385. Springer-Verlag, New York, 1996.

    Google Scholar 

  • Kass G. V, An exploratory technique for investigating large quantities of categorical data. Applied Statistics, 29(2):119–127, 1980.

    Google Scholar 

  • Kearns M. and Mansour Y., A fast, bottom-up decision tree pruning algorithm with near-optimal generalization, in J. Shavlik, ed., ‘Machine Learning: Proceedings of the Fifteenth International Conference’, Morgan Kaufmann Publishers, Inc., pp. 269–277, 1998.

    Google Scholar 

  • Kearns M. and Mansour Y, On the boosting ability of top-down decision tree learning algorithms. Journal of Computer and Systems Sciences, 58(1): 109–128, 1999.

    Article  MathSciNet  MATH  Google Scholar 

  • Kohavi R. and Sommerfield D., Targeting business users with decision table classifiers, in R. Agrawal, P. Stolorz & G. Piatetsky-Shapiro, eds, ‘Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining’, AAAI Press, pp. 249–253, 1998.

    Google Scholar 

  • Langley, P. and Sage, S., Oblivious decision trees and abstract cases. in Working Notes of the AAAI-94 Workshop on Case-Based Reasoning, pp. 113–117, Seattle, WA: AAAI Press, 1994.

    Google Scholar 

  • Last, M., Maimon, O. and Minkov, E., Improving Stability of Decision Trees, International Journal of Pattern Recognition and Artificial Intelligence, 16: 2, 145–159, 2002.

    Article  Google Scholar 

  • Li X. and Dubes R. C, Tree classifier design with a Permutation statistic, Pattern Recognition 19:229–235, 1986.

    Article  Google Scholar 

  • Lim X., Loh W.Y., and Shih X., A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms. Machine Learning 40:203–228, 2000.

    Article  MATH  Google Scholar 

  • Lin Y. K. and Fu K., Automatic classification of cervical cells using a binary tree classifier. Pattern Recognition, 16(1):69–80, 1983.

    Article  Google Scholar 

  • Loh W.Y.,and Shih X., Split selection methods for classification trees. Statistica Sinica, 7: 815–840, 1997.

    MathSciNet  MATH  Google Scholar 

  • Loh W.Y. and Shih X., Families of splitting criteria for classification trees. Statistics and Computing 9:309–315, 1999.

    Article  Google Scholar 

  • Loh W.Y. and Vanichsetakul N., Tree-structured classification via generalized discriminant Analysis. Journal of the American Statistical Association, 83: 715–728, 1988.

    Article  MathSciNet  MATH  Google Scholar 

  • Lopez de Mantras R., A distance-based attribute selection measure for decision tree induction, Machine Learning 6:81–92, 1991.

    Google Scholar 

  • Lubinsky D., Algorithmic speedups in growing classification trees by using an additive split criterion. Proc. AI&Statistics93, pp. 435–444, 1993.

    Google Scholar 

  • Martin J. K., An exact probability metric for decision tree splitting and stopping. An Exact Probability Metric for Decision Tree Splitting and Stopping, Machine Learning, 28,2–3):257–291, 1997.

    Article  Google Scholar 

  • Mehta M., Rissanen J., Agrawal R., MDL-Based Decision Tree Pruning. KDD 1995: pp. 216–221, 1995.

    Google Scholar 

  • Mehta M., Agrawal R. and Rissanen J., SLIQ: A fast scalable classifier for Data Mining: In Proc. If the fifth Int’l Conference on Extending Database Technology (EDBT), Avignon, France, March 1996.

    Google Scholar 

  • Mingers J., An empirical comparison of pruning methods for decision tree induction. Machine Learning, 4(2):227–243, 1989.

    Article  Google Scholar 

  • Morgan J. N. and Messenger R. C, THAID: a sequential search program for the analysis of nominal scale dependent variables. Technical report, Institute for Social Research, Univ. of Michigan, Ann Arbor, MI, 1973.

    Google Scholar 

  • Muller W., and Wysotzki F, Automatic construction of decision trees for classification. Annals of Operations Research, 52:231–247, 1994.

    Article  Google Scholar 

  • Murthy S. K., Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey. Data Mining and Knowledge Discovery, 2(4):345–389, 1998.

    Article  Google Scholar 

  • Naumov G.E., NP-completeness of problems of construction of optimal decision trees. Soviet Physics: Doklady, 36(4):270–271, 1991.

    MATH  MathSciNet  Google Scholar 

  • Niblett T. and Bratko I., Learning Decision Rules in Noisy Domains, Proc. Expert Systems 86, Cambridge: Cambridge University Press, 1986.

    Google Scholar 

  • Olaru C, Wehenkel L., A complete fuzzy decision tree technique, Fuzzy Sets and Systems, 138(2):221–254, 2003.

    Article  MathSciNet  Google Scholar 

  • Pagallo, G. and Huassler, D., Boolean feature discovery in empirical learning, Machine Learning, 5(1): 71–99, 1990.

    Article  Google Scholar 

  • Peng Y., Intelligent condition monitoring using fuzzy inductive learning, Journal of Intelligent Manufacturing, 15(3): 373–380, June 2004.

    Article  Google Scholar 

  • Quinlan, J.R., Induction of decision trees, Machine Learning 1, 81–106, 1986.

    Google Scholar 

  • Quinlan, J.R., Simplifying decision trees, International Journal of Man-Machine Studies, 27, 221–234, 1987.

    Article  Google Scholar 

  • Quinlan, J.R., Decision Trees and Multivalued Attributes, J. Richards, ed., Machine Intelligence, V. 11, Oxford, England, Oxford Univ. Press, pp. 305–318, 1988.

    Google Scholar 

  • Quinlan, J. R., Unknown attribute values in induction. In Segre, A. (Ed.), Proceedings of the Sixth International Machine Learning Workshop Cornell, New York. Morgan Kaufmann, 1989.

    Google Scholar 

  • Quinlan, J. R., C4.5: Programs for Machine Learning, Morgan Kaufmann, Los Altos, 1993.

    Google Scholar 

  • Quinlan, J. R. and Rivest, R. L., Inferring Decision Trees Using The Minimum Description Length Principle. Information and Computation, 80:227–248, 1989.

    Article  MathSciNet  MATH  Google Scholar 

  • Rastogi, R., and Shim, K., PUBLIC: A Decision Tree Classifier that Integrates Building and Pruning, Data Mining and Knowledge Discovery, 4(4):315–344, 2000.

    Article  MATH  Google Scholar 

  • Rissanen, J., Stochastic complexity and statistical inquiry. World Scientific, 1989.

    Google Scholar 

  • Rounds, E., A combined non-parametric approach to feature selection and binary decision tree design, Pattern Recognition 12, 313–317, 1980.

    Article  Google Scholar 

  • Schlimmer, J. C., Efficiently inducing determinations: A complete and systematic search algorithm that uses optimal pruning. In Proceedings of the 1993 International Conference on Machine Learning: pp 284–290, San Mateo, CA, Morgan Kaufmann, 1993.

    Google Scholar 

  • Sethi, K., and Yoo, J. H., Design of multicategory, multifeature split decision trees using perceptron learning. Pattern Recognition, 27(7):939–947, 1994.

    Article  Google Scholar 

  • Shafer, J. C, Agrawal, R. and Mehta, M., SPRINT: A Scalable Parallel Classifier for Data Mining, Proc. 22nd Int. Conf. Very Large Databases, T. M. Vijayaraman and Alejandro P. Buchmann and C. Mohan and Nandlal L. Sarda (eds), 544–555, Morgan Kaufmann, 1996.

    Google Scholar 

  • Sklansky, J. and Wassel, G. N., Pattern classifiers and trainable machines. SpringerVerlag, New York, 1981.

    MATH  Google Scholar 

  • Sonquist, J. A., Baker E. L., and Morgan, J. N., Searching for Structure. Institute for Social Research, Univ. of Michigan, Ann Arbor, MI, 1971.

    MATH  Google Scholar 

  • Taylor P. C, and Silverman, B. W., Block diagrams and splitting criteria for classification trees. Statistics and Computing, 3(4):147–161, 1993.

    Article  Google Scholar 

  • Utgoff, P. E., Perceptron trees: A case study in hybrid concept representations. Connection Science, 1(4):377–391, 1989.

    Google Scholar 

  • Utgoff, P. E., Incremental induction of decision trees. Machine Learning, 4: 161–186, 1989.

    Article  Google Scholar 

  • Utgoff, P. E., Decision tree induction based on efficient tree restructuring, Machine Learning 29,1):5–44, 1997.

    Article  MATH  Google Scholar 

  • Utgoff, P. E., and Clouse, J. A., A Kolmogorov-Smirnoff Metric for Decision Tree Induction, Technical Report 96-3, University of Massachusetts, Department of Computer Science, Amherst, MA, 1996.

    Google Scholar 

  • Wallace, C. S., and Patrick J., Coding decision trees, Machine Learning 11: 7–22, 1993.

    Article  MATH  Google Scholar 

  • Zantema, H., and Bodlaender H. L., Finding Small Equivalent Decision Trees is Hard, International Journal of Foundations of Computer Science, 11(2): 343–354, 2000.

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Industrial Engineering, Tel-Aviv University, Israel

    Lior Rokach & Oded Maimon

Authors
  1. Lior Rokach
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Oded Maimon
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Dept. of Industrial Engineering, Tel-Aviv University, 69978, Ramat-Aviv, Israel

    Oded Maimon  & Lior Rokach  & 

Rights and permissions

Reprints and permissions

Copyright information

© 2005 Springer Science+Business Media, Inc.

About this chapter

Cite this chapter

Rokach, L., Maimon, O. (2005). Decision Trees. In: Maimon, O., Rokach, L. (eds) Data Mining and Knowledge Discovery Handbook. Springer, Boston, MA. https://doi.org/10.1007/0-387-25465-X_9

Download citation

  • DOI: https://doi.org/10.1007/0-387-25465-X_9

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-0-387-24435-8

  • Online ISBN: 978-0-387-25465-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation