Abstract
Hadoop Distributed File System (HDFS) is designed to reliably storage and manage large-scale files. All the files in HDFS are managed by a single server, the NameNode. The NameNode stores metadata, in its main memory, for each file stored into HDFS. HDFS suffers the penalty of performance with increased number of small files. It imposes a heavy burden to the NameNode to store and manage a mass of small files. The number of files that can be stored into HDFS is constrained by the size of NameNode’s main memory. In order to improve the efficiency of storing and accessing the small files on HDFS, we propose Small Hadoop Distributed File System (SHDFS), which bases on original HDFS. Compared to original HDFS, we add two novel modules in the proposed SHDFS: merging module and caching module. In merging module, the correlated files model is proposed, which is used to find out the correlated files by user-based collaborative filtering and then merge correlated files into a single large file to reduce the total number of files. In caching module, we use Log - linear model to dig out some hot-spot data that user frequently access to, and then design a special memory subsystem to cache these hot-spot data. Caching mechanism speeds up access to hot-spot data.
The experimental results indicate that SHDFS is able to reduce the metadata footprint on NameNode’s main memory and also improve the efficiency of storing and accessing large number of small files.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Taunk, A., Parmar, A., et al.: The Hadoop distributed file system. Int. J. Comput. 8(1), 8–15 (2013)
Dean, J., Ghemawat, S., et al.: MapReduce:a flexible data processing tool. Commun. ACM 53(1), 72–77 (2010)
Ghemawat, S., Gobioff, H., Leung, S.: The Google file system. ACM 37(5), 29–43 (2003)
Zhou, Y., Zeng, F., Zhao, H., et al.: Hierarchical visual perception and two-dimensional compressive sensing for effective content-based color image retrieval. Cognit. Comput. 8(5), 877–889 (2016)
White, T.: Hadoop: The Definitive Guide. O’Reilly Media, Inc., Newton
Cloudera Engineer Blog: http://www.cloudera.com. Accessed 21 Mar 2018
Herlocker, J., Konstan, J., Riedl, J.: An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms. Inf. Retrieval 5(4), 287–310 (2002)
Szabo, G., Huberman, B.A.: Predicting the popularity of online content. Commun. ACM 53(8), 80–88 (2010)
Zhao, H., Ren J., Zhan, J., et al.: Compressive sensing based secret signals recovery for effective image Steganalysis in secure communications. Multimed. Tools Appl. 1–14 (2018)
Yan, Y., Ren, J., Sun, G., et al.: Unsupervised image saliency detection with Gestalt-laws guided optimization and visual attention based refinement. Pattern Recogn. 79(7), 65–78 (2018)
Zhao, H., Dai, Q., Ren, J.C., et al.: Robust information hiding in low-resolution videos with quantization index modulation in DCT-CS domain. Multimed. Tools Appl. 1, 1–21 (2017)
Zhao, H., Ren, J.: Cognitive computation of compressed sensing for watermark signal measurement. Cognit. Comput. 8(2), 246–260 (2016)
Acknowledgements
This work was supported in part by the Science and Technology Project of Guangdong Province (2015B010131017, 2017B030306016) and Guangzhou City (201802020019, 201806040010).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Peng, Jf. et al. (2018). Hadoop Massive Small File Merging Technology Based on Visiting Hot-Spot and Associated File Optimization. In: Ren, J., et al. Advances in Brain Inspired Cognitive Systems. BICS 2018. Lecture Notes in Computer Science(), vol 10989. Springer, Cham. https://doi.org/10.1007/978-3-030-00563-4_50
Download citation
DOI: https://doi.org/10.1007/978-3-030-00563-4_50
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00562-7
Online ISBN: 978-3-030-00563-4
eBook Packages: Computer ScienceComputer Science (R0)