Skip to main content
SpringerLink
Log in

A Two-Sample Kolmogorov-Smirnov-Like Test for Big Data

  • Conference paper
  • First Online:
Data Mining (AusDM 2017)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 845))

Included in the following conference series:

Abstract

Exploratory data analysis (EDA) is an important component of modern data analysis and data mining. The Big Data setting has made many traditional and useful EDA tools impractical and ineffective. Among such useful tools is the two-sample Kolmogorov-Smirnov (TS-KS) goodness-of-fit (GoF) test for assessing whether or not two samples arose from the same population. A TS-KS like testing procedure is constructed using chunked and averaged (CA) estimation paradigm. The procedure is named the TS-CAKS GoF test. Distributed and streamed implementations of the TS-CAKS procedure are discussed. The consistency of the TS-CAKS test is proved. A numerical study is provided to demonstrate the effectiveness and computational efficiency of the procedure.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Buoncristiano, M., Mecca, G., Quitarelli, E., Roveri, M., Santoro, D., Tanca, L.: Database challenges for exploratory computing. ACM SIGMOD Rec. 44, 17–22 (2015)

    Article  Google Scholar 

  • DasGupta, A.: Asymptotic Theory of Statistics and Probability. Springer, New York (2008). https://doi.org/10.1007/978-0-387-75971-5

    Book  MATH  Google Scholar 

  • Dasu, T., Johnson, T.: Exploratory Data Mining and Data Cleaning. Wiley, New York (2003)

    Book  Google Scholar 

  • dos Reis, D., Flach, P., Matwin, S., Batista, G.: Fast unsupervised online drift detection using incremental Kolmogorov-Smirnov test. In: ACM SIGKDD International Conference on Knowledge Disocvery and Data Mining XXII. ACM (2016)

    Google Scholar 

  • Kifer, D., Ben-David, S., Gehrke, J.: Detecting change in data streams. In: Proceedings of the 30th VLDB Conference (2004)

    Chapter  Google Scholar 

  • Kim, P.J.: On the exact and approximate sampling distribution of the two sample Kolmogorov-Smirnov criterion \(D_{mn}\), \(m \le n\). J. Am. Stat. Assoc. 64, 1625–1637 (1969)

    Google Scholar 

  • Kim, P.J., Jennrich, R.I.: Selected tables in mathematical statistics 1, chapter tables of the exact sampling distribution of the two-sample Kolmogorov-Smirnov criterion \(D_{mn}\), \(m \le n\), pp. 80–129. Institute of Mathematical Statistics (1973)

    Google Scholar 

  • Lall, A.: Data streaming algorithm for the Kolmogorov-Smirnov test. In: Proceedings of the IEEE International Conference on Big Data, pp. 95–104 (2015)

    Google Scholar 

  • Li, R., Lin, D.K.J., Li, B.: Statistical inference in massive data sets. Appl. Stoch. Models Bus. Ind. 29, 399–409 (2013)

    MathSciNet  Google Scholar 

  • Matloff, N.: Software alchemy: turning complex statistical computations into embarrassingly-parallel ones. J. Stat. Softw. 71, 1–15 (2016)

    Article  Google Scholar 

  • Mecca, G.: Database exploration: problems and opportunities. In: IEEE 32rd International Conference on Data Engineering Workshop, pp. 153–156 (2016)

    Google Scholar 

  • Myatt, G.J., Johnson, W.P.: Making Sense of Data I: A Practical Guide to Exploratory Data Analysis and Data Mining. Wiley, New York (2014)

    Book  Google Scholar 

  • Nguyen, H.D.: A simple online parameter estimation technique with asymptotic guarantees. arXiv:1703.07039 (2017a)

  • Nguyen, H.D.: A stream-suitable Kolmogorov-Smirnov-type test for Big Data analysis. arXiv:1704.03721 (2017b)

  • Nguyen, H.D., McLachlan, G.J.: Chunked-and-averaged estimators for vector parameters. arXiv:1612.06492 (2017)

  • R Core Team: R: a language and environment for statistical computing. R Foundation for Statistical Computing (2016)

    Google Scholar 

  • Smirnov, N.V.: Estimating the deviation between the empirical distribution functions of two independent samples. Bulletin de l’Universite de Moscou, 2, 3–16 (1939)

    Google Scholar 

  • Tukey, J.W.: The future of data analysis. Ann. Math. Stat. 33, 1–67 (1962)

    Article  MathSciNet  Google Scholar 

  • Tukey, J.W.: Exploratory Data Analysis. Addison-Wesley, Reading (1977)

    MATH  Google Scholar 

  • Wang, J., Tsang, W.W., Marsaglia, G.: Evaluating Kolmogorov’s distribution. J. Stat. Softw. 8, 18 (2003)

    Google Scholar 

Download references

Acknowledgements

The author is personally supported by Australian Research Council grant number DE170101134.

Author information

Authors and Affiliations

  1. Department of Mathematics and Statistics, La Trobe University, Bundoora, VIC, 3086, Australia

    Hien D. Nguyen

Authors
  1. Hien D. Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hien D. Nguyen .

Editor information

Editors and Affiliations

  1. RMIT University, Melbourne, Victoria, Australia

    Yee Ling Boo

  2. University of Wollongong, Wollongong, New South Wales, Australia

    David Stirling

  3. La Trobe University, Melbourne, Victoria, Australia

    Lianhua Chi

  4. School of Information Technology and Mathematical Sciences, University of South Australia, Adelaide, South Australia, Australia

    Lin Liu

  5. La Trobe University, Melbourne, Victoria, Australia

    Kok-Leong Ong

  6. Microsoft Pty Ltd, Singapore, Singapore

    Graham Williams

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nguyen, H.D. (2018). A Two-Sample Kolmogorov-Smirnov-Like Test for Big Data. In: Boo, Y., Stirling, D., Chi, L., Liu, L., Ong, KL., Williams, G. (eds) Data Mining. AusDM 2017. Communications in Computer and Information Science, vol 845. Springer, Singapore. https://doi.org/10.1007/978-981-13-0292-3_6

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-0292-3_6

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-0291-6

  • Online ISBN: 978-981-13-0292-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation