HDFS Logfile Analysis Using ElasticSearch, LogStash and Kibana

Abstract

Hadoop is an open-source software framework for storing and processing large sets of data on a platform consisting of commodity hardware. Hadoop is mostly designed to handle large amounts of data, which can easily run into many petabytes and even exabytes. Hadoop file sizes are usually very large, ranging from gigabytes to terabytes, and large Hadoop clusters store millions of these files. Hadoop depends on large number of servers so it can parallelize work across them. Server and storage failures are to be expected, and the system is not affected by non-functioning storage units or even failed servers. Traditional databases are geared mostly for fast access to data and not for batch processing. Hadoop was originally designed for batch processing, such as the indexing of millions of web pages, and provides streaming access to datasets. Data consistency issues that may arise in an updatable database are not an issue with Hadoop file systems, because only a single writer can deal with write operation. Activity on the server will be captured by logs. There are two types of logs, and they can be generated from web servers or application servers or both. Access logs and error logs are two types of log files. An access log will have client info, whereas an error log consists of exceptions and error info. This chapter will address the log file analysis process using ElasticSearch, LogStash and Kibana. We can show the frequency of errors by the given time period using different forms such as trend graphs, bar graphs, pie charts and gauge charts.