Abstract
The rapid development of the Internet, especially mobile Internet, makes it much easier for people to make social contacts online. Nowadays people tend to spend more and more time on social network service, and produce a lot of image files. This brings a challenge to traditional standalone framework on handing the continued increasing image files. Therefore, it is advisable to find a new way to settle the challenge. Hadoop is a notable, widely-used project for distributed storage and computations with high efficiency, data integrity, reliability and fault tolerance. Hadoop Distributed File System and MapReduce are two primary subprojects respectively for big data storage and computations. However, Hadoop does not provide any interface for image processing. Moreover, both Hadoop Distributed File System and MapReduce have trouble in processing large amount of small files, which result in decreasing efficiency of files access and distributed computations. This prevents us from performing images processing actions on Hadoop. In view of this, this paper proposes a new method to optimize the storage of small image files on Hadoop and self-defines an input/output format to enable Hadoop to process image files.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ren, J., Zabalza, J., Marshall, S., Zheng, J.: Effective feature extraction and data reduction in remote sensing using hyperspectral imaging. IEEE Signal Process. Mag. 31(4), 149–154 (2014)
Qiao, T., Yang, Z., Ren, J., et al.: Joint bilateral filtering and spectral similarity-based sparse representation: a generic framework for effective feature extraction and data classification in hyperspectral imaging. Pattern Recogn. 77, 316–328 (2017)
Zabalza, J., et al.: Novel two dimensional singular spectrum analysis for effective feature extraction and data classification in hyperspectral imaging. IEEE Trans. Geosci. Remote Sens. 53, 4418–4433 (2015)
Qiao, T., Ren, J., et al.: Effective denoising and classification of hyperspectral images using curve let transform and singular spectrum analysis. IEEE Trans. Geosci. Remote Sens. 55, 119–133 (2017)
Cao, F., Yang, Z., Ren, J., Ling, W.K., Zhao, H., Marshall, S.: Extreme sparse multinomial logistic regression: a fast and robust framework for hyperspectral image classification. Remote Sens. 9(12), 1255 (2017)
Mohandas, N., Thampi, S.M.: Improving Hadoop performance in handling small files. In: Abraham, A., Mauri, J.L., Buford, J.F., Suzuki, J., Thampi, S.M. (eds.) ACC 2011. CCIS, vol. 193, pp. 187–194. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22726-4_20
Bende, S., Shedge, R.: Dealing with small files problem in Hadoop distributed file system. Proced. Comput. Sci. 79, 1001–1012 (2016)
Ghazi, M.R., Gangodkar, D.: Hadoop, MapReduce and HDFS: a developers perspective. Proced. Comput. Sci. 48, 45–50 (2015)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51, 107–113 (2008)
He, H., Du, Z., Zhang, W., Chen, A.: Optimization strategy of Hadoop small file storage for big data in healthcare. J. Supercomput. 72, 3696–3707 (2015)
Mackey, G., Sehrish, S., Wang, J.: Improving metadata management for small files in HDFS. In: IEEE International Conference on Cluster Computing, pp. 1–4 (2009)
Cao, Z., Lin, J., Wan, C., Song, Y., Taylor, G., Li, M.: Hadoop-based framework for big data analysis of synchronised harmonics in active distribution network. IET Gener. Transm. Distrib. 11, 3930–3937 (2017)
Zhao, S., Medhi, D.: Application-aware network design for Hadoop MapReduce optimization using software-defined networking. IEEE Trans. Netw. Serv. Manage. 14, 804–816 (2017)
George, J., Chen, C.-A., Stoleru, R., Xie, G.G.: Hadoop MapReduce for mobile clouds. IEEE Trans. Cloud Comput. 3(1), 1–14 (2014)
Won, H., Nguyen, M., Gil, M., Moon, Y.: Advanced resource management with access control for multitenant Hadoop. J. Commun. Netw. 17, 592–601 (2015)
Acknowledgements
This paper is supported by the National Nature Science Foundation of China (No. 61370103), Guangdong Province Application Major Fund (2015B010131013) and Zhongshan Produce & Research Fund (2017A1014).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Su, Q., Lu, L., Feng, Q. (2018). An Optimal Solution of Storing and Processing Small Image Files on Hadoop. In: Ren, J., et al. Advances in Brain Inspired Cognitive Systems. BICS 2018. Lecture Notes in Computer Science(), vol 10989. Springer, Cham. https://doi.org/10.1007/978-3-030-00563-4_63
Download citation
DOI: https://doi.org/10.1007/978-3-030-00563-4_63
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00562-7
Online ISBN: 978-3-030-00563-4
eBook Packages: Computer ScienceComputer Science (R0)