Abstract
Everyone can easily use the Internet because it is the source of information. The Internet holds a tremendous amount of information, get the relevant information and high efficiency is challenging issue. The crawler is helping to extract relevant information easily. We propose smart crawler which provides better results than another crawler. Smart crawler has two stages as site locating and in-site exploring. Site locating can fetch the relevant information and in-site exploring rank the sites as per their relevancy. We implement smart crawler using a hybrid algorithm. The hybrid algorithm is a combination of bisect K-means algorithm and bottom up agglomerative clustering algorithm. The bisect K-means clustering algorithm split cluster into sub clusters and agglomerative algorithm computes the centroid of the clusters and merge them on the basis of similarities or approximate similar centroid value and form singleton cluster hence it is simple to use and provides efficient and better result than K-means clustering algorithm as it generates uniform size clusters.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Olston, C., Najork, M.: Web crawling. Found. Trends Inf. Retr. 4, 175–246 (2010)
Zhao, F., Zhou, J., Nie, C., Jin, H.: SmartCrawler: a two-stage crawler for efficiently harvesting deep-web interfaces. IEEE Trans. Serv. Comput. 99, 1–14 (2015)
Bangoria, B., Mankad, N., Pambhar, V.: Enhanced K-means clustering algorithm to reduce the time complexity for numeric values. Int. J. Adv. Eng. Res. Dev. 1, 1–9 (2014)
Wagsta, K., Cardie, C.: Constrained K-means clustering with background knowledge. In: Proceedings of the Eighteenth International Conference on Machine Learning, pp. 577–584 (2001)
Purohit, P.: A new efficient approach towards K-means clustering algorithm. Int. J. Comput. Appl. 65, 7–10 (2013)
Singh, G., Kaur, N.: Implementation of hybrid clustering algorithm with enhanced K-means and hierarchal clustering. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 3, 608–618 (2013)
Sasirekha, K., Baby, P.: Agglomerative hierarchical clustering algorithm-a review. Int. J. Sci. Res. Publ. 3, 1–3 (2013)
http://metaquerier.cs.uiuc.edu/repository/datasets/tel-8/documentation.html
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Bhoi, S.G., Patil, U.M. (2018). Hybrid Clustering Based Smart Crawler. In: Deshpande, A., et al. Smart Trends in Information Technology and Computer Communications. SmartCom 2017. Communications in Computer and Information Science, vol 876. Springer, Singapore. https://doi.org/10.1007/978-981-13-1423-0_16
Download citation
DOI: https://doi.org/10.1007/978-981-13-1423-0_16
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-1422-3
Online ISBN: 978-981-13-1423-0
eBook Packages: Computer ScienceComputer Science (R0)