Abstract
Poor road conditions, such as potholes, are a nuisance to society, which would annoy passengers, damage vehicles, and even cause accidents. Thus, detecting potholes is an important step toward pavement maintenance and rehabilitation to improve road conditions. Potholes have different shapes, scales, shadows, and illumination effects, and highly complicated backgrounds can be involved. Therefore, detection of potholes in road images is still a challenging task. In this study, we focus on pothole detection in 2D vision and present a new method to detect potholes based on location-aware convolutional neural networks, which focuses on the discriminative regions in the road instead of the global context. It consists of two main subnetworks: the first localization subnetwork employs a high recall network model to find as many candidate regions as possible, and the second part-based subnetwork performs classification on the candidates on which the network is expected to focus. The experiments using the public pothole dataset show that the proposed method could achieve high precision (95.2%), recall (92.0%) simultaneously, and outperform the most existing methods. The results also demonstrate that accurate part localization considerably increases classification performance while maintains high computational efficiency. The source code is available at https://github.com/hanshenchen/pothole-detection.
Similar content being viewed by others
References
National Safety Council (2018) Safety on the road. https://www.nsc.org/road-safey. Accessed 7 Jan 2019
European Commission (2018) Road safety in the European Union trends, statistics and main challenges. https://ec.europa.eu/transport/road_safety/sites/roadsafety/files/vademecum_2018.pdf. Accessed 7 Jan 2019
European Commission (2017) Public consultation on road infrastructure and tunnel safety. https://ec.europa.eu/transport/modes/road/consultations/2017-road-infrastructure-safety_en. Accessed 7 Jan 2019
Federal Highway Administration (2018) Asset management overview. https://www.fhwa.dot.gov/asset/if08008/amo_06.cfm. Accessed 7 Jan 2019
Kim T, Ryu S (2014) Review and analysis of pothole detection methods. J Emerg Trends Comput Inf Sci 5(8):603–608
Fan R, Ai X, Dahnoun N (2018) Road surface 3d reconstruction based on dense subpixel disparity map estimation. IEEE Trans Image Process 27(6):3025–3035. https://doi.org/10.1109/TIP.2018.2808770
Tsai YC, Chatterjee A (2017) Pothole detection and classification using 3D technology and watershed method. J Comput Civ Eng 32(2):04017078. https://doi.org/10.1061/(asce)cp.1943-5487.0000726
Zhang A, Wang KCP, Ai C (2017) 3D shadow modeling for detection of descended patterns on 3D pavement surface. J Comput Civ Eng 31(4):04017019. https://doi.org/10.1061/(asce)cp.1943-5487.0000661
Li S, Yuan C, Liu D, Cai H (2016) Integrated processing of image and GPR data for automated pothole detection. J Comput Civ Eng 30(6):04016015. https://doi.org/10.1061/(asce)cp.1943-5487.0000582
Eriksson J, Girod L, Hull B et al (2008) The pothole patrol: using a mobile sensor network for road surface monitoring. In: Proceedings of 6th international conference on. mobile systems, applications, and services, pp 29–39. https://doi.org/10.1145/1378600.1378605
Wang HW, Chen CH, Cheng DY et al (2015) A real-time pothole detection approach for intelligent transportation system. Math Probl Eng. https://doi.org/10.1155/2015/869627
Xue G, Zhu H, Hu Z et al (2017) Pothole in the dark: perceiving pothole profiles with participatory urban vehicles. IEEE Trans Mobile Comput 16(5):1408–1419. https://doi.org/10.1109/tmc.2016.2597839
Buza E, Omanovic S, Huseinovic A (2013) Pothole detection with image processing and spectral clustering. In: Proceedings of 2nd international conference on information technology and computer networks, pp 48–53
Koch C, Brilakis I (2011) Pothole detection in asphalt pavement images. Adv Eng Inf 25(3):507–515. https://doi.org/10.1016/j.aei.2011.01.002
Pereira V, Tamura S, Hayamizu S, Fukai H (2018) A deep learning-based approach for road pothole detection in timor leste. In: Proc 2018 IEEE international conference on service operations and logistics, and informatics, pp 279–284. https://doi.org/10.1109/soli.2018.8476795
Anand S, Gupta S, Darbari V, Kohli S (2018) Crack-pot: autonomous road crack and pothole detection. In: Proceedings of 2017 international conference on digital image computing: techniques and applications, pp 1–6. https://doi.org/10.1109/DICTA.2018.8615819
Zhang L, Yang F, Zhang YD, Zhu YJ (2016) Road crack detection using deep convolutional neural network. In: Proceedings of IEEE international conference on image processing, pp 3708–3712. https://doi.org/10.1109/icip.2016.7533052
Eisenbach M, Stricker R, Seichter D et al (2017) How to get pavement distress detection ready for deep learning? A systematic approach. In: Proceedings of 2017 international joint conference on neural networks, pp 2039–2047. https://doi.org/10.1109/ijcnn.2017.7966101
Fu J, Zheng H, Mei T (2017) Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 4476–4484. https://doi.org/10.1109/cvpr.2017.476
Liu H, Feng J, Qi M et al (2017) End-to-end comparative attention networks for person re-identification. IEEE Trans Image Process 26(7):3492–3506. https://doi.org/10.1109/tip.2017.2700762
Zhang N, Donahue J, Girshick R, Darrell T (2014) Part-based R-CNNs for fine-grained category detection. In: Proceedings of European conference on computer vision. Springer, Cham, pp 834–849. https://doi.org/10.1007/978-3-319-10590-1_54
Huang S, Xu Z, Tao D, Zhang Y (2016) Part-stacked CNN for fine-grained visual categorization. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 1173–1182. https://doi.org/10.1109/cvpr.2016.132
Yang Z, Luo T, Wang D, Hu Z, Gao J, Wang L (2018) Learning to navigate for fine-grained classification. In: Proceeding of European conference on computer vision. Springer, Cham, pp 420–435. https://doi.org/10.1007/978-3-030-01264-9_26
Machine Intelligence Institute of Africa (2017) MIIA deep learning Hackathon. http://machineintelligenceafrica.org/activities/hackathon/. Accessed 7 Jan 2019
Tedeschi A, Benedetto F (2017) A real-time automatic pavement crack and pothole recognition system for mobile Android-based devices. Adv Eng Inf 32:11–25. https://doi.org/10.1061/(asce)cp.1943-5487.0000726
Radopoulou SC, Brilakis I (2016) Automated detection of multiple pavement defects. J Comput Civ Eng 31(2):04016057. https://doi.org/10.1061/(asce)cp.1943-5487.0000623
Nienaber S, Booysen MJ, Kroon RS (2015) Detecting potholes using simple image processing techniques and real-world footage. In: Proceedings of 34th South African transport conference, Southern African transport conference
Akagic A, Buza E, Omanovic S (2017) Pothole detection: an efficient vision based method using RGB color space image segmentation. In: Proceedings of 40th international convention on information and communication technology, electronics and microelectronics, pp 1104–1109. https://doi.org/10.23919/MIPRO.2017.7973589
Koch C, Jog GM, Brilakis I (2012) Automated pothole distress assessment using asphalt pavement video data. J Comput Civ Eng 27(4):370–378. https://doi.org/10.1061/(asce)cp.1943-5487.0000232
Feng C, Liu MY, Kao CC, Lee TY (2017) Deep active learning for civil infrastructure defect detection and classification. In: Proceedings of ASCE international workshop on computing in civil engineering, pp 298–306. https://doi.org/10.1061/9780784480823.036
Maeda H, Sekimoto Y, Seto T, Kashiyama T, Omata H (2018) Road damage detection using deep neural networks with images captured through a smartphone. Comput Aided Civ Infrastruct Eng 33:1127–1141. https://doi.org/10.1111/mice.12387
Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 3431–3440. https://doi.org/10.1109/cvpr.2015.7298965.
Xiao B, Wu H, Wei Y (2018) Simple baselines for human pose estimation and tracking. In: Proceedings of European conference on computer vision, pp 472–487. https://doi.org/10.1007/978-3-030-01231-1_29
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/cvpr.2016.90
Ioffe S, Szegedy C (2015). Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of 32td international conference on machine learning, international machine learning society (IMLS), vol 37, pp 448–456
Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Proceedings of 26th Conference on neural information processing systems, neural information processing systems foundation, pp 1097–1105
Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 815–823. https://doi.org/10.1109/cvpr.2015.7298682
Scherer D, Muller A, Behnke S (2010) Evaluation of pooling operations in convolutional architectures for object recognition. In: Proceedings of international conference on artificial neural networks, pp 92–101. https://doi.org/10.1007/978-3-642-15825-4_10
Nienaber S, Kroon RS, Booysen MJ (2015) A comparison of low-cost monocular vision techniques for pothole distance estimation. In: Proceedings of 2015 IEEE symposium series on computational intelligence, pp 419–426. https://doi.org/10.1109/ssci.2015.69
Chen HS (2019) pothole-detection. https://github.com/hanshenChen/pothole-detection/tree/master/preprocessing. Accessed 7 Jan 2019
Yang Y, Ramanan D (2013) Articulated human detection with flexible mixtures of parts. IEEE Trans Pattern Anal Mach Intell 35(12):2878–2890. https://doi.org/10.1109/tpami.2012.261
Chollet F et al (2015) Keras. https://github.com/fchollet/keras. Accessed 7 Jan 2019
Abadi M, Agarwal A, Barham P et al (2016) TensorFlow: large-scale machine learning on heterogeneous distributed systems. Preprint, submitted Mar 14, 2016. Software available from tensorflow.org. http://arxiv.org/abs/1603.04467
Russakovsky O, Deng J, Su H et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vision 115(3):211–252. https://doi.org/10.1007/s11263-015-0816-y
Kingma D, Ba J (2015) Adam: A method for stochastic optimization. In: Proceedings of international conference on learning representations
Marais J et al (2017) PotholeDetection. https://github.com/jandremarais/PotholeDetection. Accessed 7 Jan 2019
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 4700–4708. https://doi.org/10.1109/CVPR.2017.243
Acknowledgements
This work was supported by the National Science Foundation of China (No. 61871350) and the Department of Communication of Zhejiang Province, China (No. 2017JY04).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Chen, H., Yao, M. & Gu, Q. Pothole detection using location-aware convolutional neural networks. Int. J. Mach. Learn. & Cyber. 11, 899–911 (2020). https://doi.org/10.1007/s13042-020-01078-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13042-020-01078-7