Skip to main content
SpringerLink
Log in

MAD: A Monitor System for Big Data Applications

  • Conference paper
  • First Online:
Intelligence Science and Big Data Engineering. Big Data and Machine Learning Techniques (IScIDE 2015)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 9243))

Abstract

A big data application usually needs to build a pipeline on the top of workflow engine which connects relevant periodic workflow jobs. It’s crucial to timely alert pipeline issues, provide an issue diagnosis subsystem to find out root cause from a variety of sources, and measure pipeline/service by predefined metrics. In this paper, we identify three indispensable qualities monitor systems must fulfill namely timeliness, accuracy and flexibility. We find that the conventional monitoring tools lack at least one of three qualities, and introduce a general purpose MAD (Monitoring, Alerting and Diagnosis) system for big data applications to keep data freshness, collect measurement metrics to meet SLA.

This work is specially supported by the Science and Technology Plan General Program of Beijing Municipal Education Commission (KM201510037001), Chinese Mountaineering Association (CMA2014-B-A04) and Intelligence Logistics System Beijing Key Laboratory (NO:BZ0211)

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Khanna, G., Varadharajan, P., et al.: Automated online monitoring of distributed applications through external monitors. IEEE Trans. Dependable Secure Comput. 3(2), 115–129 (2006)

    Article  Google Scholar 

  2. Khanna, G., Cheng, M.Y., et al.: Automated rule-based diagnosis through a distributed monitor system. IEEE Trans. Dependable Secure Comput. 4(4), 266–279 (2007)

    Article  Google Scholar 

  3. Chen, H., Jiang, G., et al.: Invariants based failure diagnosis in distributed computing systems. In: IEEE Symposium on Reliable Distributed Systems, pp: 160–166 (2010)

    Google Scholar 

  4. Joshi, K.R., Hiltunen, M.A., et al.: Probabilistic model-driven recovery in distributed systems. IEEE Trans. Dependable Secure Comput. 8(6), 913–928 (2011)

    Article  Google Scholar 

  5. Ganglia - a scalable distributed monitoring system for high-performance computing systems. http://ganglia.sourceforge.net/

  6. Nagios - the industry standard in IT infrastructure monitoring. http://www.nagios.org/

  7. Splunk - the leading platform for Operational Intelligence. http://www.splunk.com/

  8. Apache Hadoop. http://wiki.apache.org/hadoop

  9. Jeffery, D., Sanjay, G.: MapReduce: simplified data processing on large clusters (2004). http://labs.google.com/papers/mapreduce.html

  10. Hadoop - Yahoo! Lauches world’s largest hadoop production applications. http://developer.yahoo.com/blogs/hadoop/posts/2008/02/yahoo-worlds-largest-product-hadoop/

  11. Ronnie, C., Bob, J., et al.: SCOPE: easy and efficient parallel processing of massive data sets. In: VLDB 2008, pp. 24–30 (2008)

    Google Scholar 

  12. Mohammad, I., Angelo, K.H. Oozie: torwards a scalable workflow management system for hadoop. In: SWEET 2012, 20 May 2012

    Google Scholar 

  13. Patrick, H., Mahadev, K., et al.: ZooKeeper: wait-free coordination for internet-scale. In: Usenix (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Beijing Wuzi University, Beijing, 100123, China

    Mingruo Shi & Ruiping Yuan

Authors
  1. Mingruo Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruiping Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingruo Shi .

Editor information

Editors and Affiliations

  1. Zhejiang University, Hangzhou, China

    Xiaofei He

  2. Xidian University, Xi'an, China

    Xinbo Gao

  3. Northwestern Polytechnical University, Xi'an, China

    Yanning Zhang

  4. Nanjing University, Nanjing, China

    Zhi-Hua Zhou

  5. Chinese Academy of Sciences, Beijing, China

    Zhi-Yong Liu

  6. Suzhou University of Science and Technology, Suzhou, China

    Baochuan Fu

  7. Suzhou University of Science and Technology, Jiangsu, China

    Fuyuan Hu

  8. Suzhou University of Science and Technology, Jiangsu, China

    Zhancheng Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Shi, M., Yuan, R. (2015). MAD: A Monitor System for Big Data Applications. In: He, X., et al. Intelligence Science and Big Data Engineering. Big Data and Machine Learning Techniques. IScIDE 2015. Lecture Notes in Computer Science(), vol 9243. Springer, Cham. https://doi.org/10.1007/978-3-319-23862-3_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-23862-3_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-23861-6

  • Online ISBN: 978-3-319-23862-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation