Abstract
Autonomous driving systems in internet of vehicles (IoV) applications usually adopt a cloud computing mode. In these systems, information got at the edge of the cloud computing center for data analysis and situation response. However, the conventional IoV face enormous challenges to meet the requirements in terms of storage, communication, and computing problems because of the considerable amount of information on the traffic environment. The environment perception during the nighttime is poorer than that during the daytime that this problem also requires addressing. To solve these problems, we propose a nighttime object detection scheme based on a lightweight deep learning model in the edge computing mode. First, the pedestrian detection and the vehicle detection algorithm that using the thermal images based on the YOLO architecture. We can implement the model on edge devices that can achieve real-time detection through the designed lightweight strategy. Next, a spatial prior information and temporal prior information into the detection algorithm and divide the frames into key and non-key frames to increase the performance and speed of the system simultaneously. Finally, we implemented the detection network for performance and feasibility verification on the Jetson TX2 edge device. The experimental results show that the proposed system can achieve real-time and high-accuracy object detection on edge devices.
Similar content being viewed by others
References
Zhang, J., Wang, F., Wang, K., Lin, W., Xu, X., Chen, C.: Data-driven intelligent transportation systems: a survey. Trans. Intell. Transp. Syst. 12(4), 1624–1639 (2011)
Guo, J., Hsia, C., Wong, K., Wu, J., Wu, Y., Wang, N.: Nighttime vehicle detection and tracking with adaptive mask training. IEEE Trans. Veh. Technol. 65(6), 4023–4032 (2016)
Boden, M.A.: Creativity and artificial intelligence. Artif. Intell. 103, 347–356 (1998)
Hsia, C., Yen, S., Jang, J.: An intelligent IoT-based vision system for nighttime vehicle detection and energy saving. Sens. Mater. 31(6), 1803–1814 (2019)
Yang, F., Wang, S., Li, J., Liu, Z., Sun, Q.: An overview of internet of vehicles. China Commun. 11(10), 1–15 (2014)
Liu, Q., Kumar, S., Mago, V.: SafeRNet: safe transportation routing in the era of internet of vehicles and mobile crowd sensing. In: IEEE Annual Consumer Communications & Networking Conference, pp. 299–304 (2017)
Sun, Y., Wang, B., Li, S., Sun, Z., Nguyen, H., Duong, T.Q.: Manipulation with domino effect for cache- and buffer-enabled social IIoT: preserving stability in tripartite graphs. IEEE Trans. Ind. Inf. 16(8), 5389–5400 (2020)
Cao, H., Wu, S., Aujla, G., Wang, Q., Yang, L., Zhu, H.: Dynamic embedding and quality of service-driven adjustment for cloud networks. IEEE Trans. Ind. Inf. 16(2), 1406–1416 (2020)
Cheng, J., Yuan, G., Zhou, M., Gao, S., Liu, C., Duan, H., Zeng, Q.: Accessibility analysis and modeling for IoV in an urban scene. IEEE Trans. Veh. Technol. 69(4), 4246–4256 (2020)
Wang, B., Sun, Y., Li, S., Cao, Q.: Hierarchical matching with peer effect for low-latency and high-reliable caching in social IoT. IEEE Internet Things J. 6(1), 1193–1209 (2019)
Alexe, B., Deselaers, T., Ferrari, V.: Measuring the objectness of image windows. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2189–2202 (2012)
Zhu, J., Zeng, H., Huang, J., Liao, S., Lei, Z., Cai, C., Zheng, L.: Vehicle re-identification using quadruple directional deep learning features. IEEE Trans. Intell. Transp. Syst. 21(1), 410–420 (2020)
Wang, Y., Piao, Y.: Enhancement system of nighttime infrared video image and visible video image. In: Selected Proceedings from CSOE (2016)
Bhowmik, M., Saha, K., Majumder, S., Majumder, G., Saha, A., Sarma, A., Bhattacharjee, D., Basu, D., Nasipuri, M.: Thermal infrared face recognition—a biometric identification technique for robust security system. In: Reviews Refinements and New Ideas in Face Recognition (2011)
Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3962–3971 (2019)
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.: MobileNetV2: inverted residuals and linear bottlenecks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp/ 4510–4520 (2018)
Amert, T., Otterness, N., Yang, M., Anderson, J. H., Smith, F. D.: GPU scheduling on the NVIDIA TX2: hidden details revealed. In: IEEE Real-Time Systems Symposium, pp. 104–115 (2017)
Bourlai, T., Cukic, B.: Multi-spectral face recognition: identification of people in difficult environments. In: IEEE International Conference on Intelligence and Security Informatics, pp. 196–201 (2012)
Fernández-Caballero, A., Castillo, J.C., Martínez-Cantos, J., Martínez-Tomás, R.: Optical flow or image subtraction in human detection from infrared camera on mobile robot. Robot. Auton. Syst. 58(12), 1273–1281 (2010)
Davis, J., Sharma, V.: Fusion-based background-subtraction using contour saliency. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, p. 11 (2005)
Jeon, E.S., Kim, J.H., Hong, H.G., Batchuluun, G., Park, K.R.: Human detection based on the generation of a background image and fuzzy system by using a thermal camera. Sensors 16, 453 (2016)
Lin, C., Chen, C., Hwang, W., Hwang, C., Chang, C.: Novel outline features for pedestrian detection system with thermal images. Pattern Recogn. 48(11), 3440–3450 (2015)
Zhao, X., He, Z., Zhang, S., Liang, D.: Robust pedestrian detection in thermal infrared imagery using a shape distribution histogram feature and modified sparse representation classification. Pattern Recogn. 48(6), 1947–1960 (2015)
Qi, Y., An, G.: Infrared moving targets detection based on optical flow estimation. Int. Conf. Comput. Sci. Netw. Technol. 4, 2452–2455 (2011)
Gilmore, E.T., Ugbome, C., Kim, C.: An IR-based pedestrian detection system implemented with Matlab-equipped laptop and low-cost microcontroller. Int. J. Comput. Sci. Inf. Technol. 3(5), 79–87 (2015)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1, 886–893 (2005)
Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)
Xu, Y., Yu, G., Wang, Y., Wu, X., Ma, Y.: A hybrid vehicle detection method based on Viola-Jones and HOG + SVM from UAV images. Sensors 16(8), 1325 (2016)
Cilulko, J., Janiszewski, P., Bogdaszewski, M., Szczygielska, E.: Infrared thermal imaging in studies of wild animals. Eur. J. Wildl. Res. 59, 17–23 (2012)
Goodall, T., Bovik, A., Paulter, N.: Tasking on natural statistics of infrared images. IEEE Trans. Image Process. 25, 65–79 (2016)
Lee, E., Ko, B., Nam, J.: Recognizing pedestrian’s unsafe behaviors in far-infrared imagery at night. Infrared Phys. Technol. 76, 261–270 (2016)
Rodger, I., Connor, B., Robertson, N. M.: Classifying objects in LWIR imagery via CNNS. Secur. Defence 9987, 99870–99884 (2016)
Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)
Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6517–6525 (2017)
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C., Berg, A.: SSD: single shot multibox detector. In: European Conference on Computer Vision (2016)
Zhang, H., Luo, C., Wang, Q., Kitchin, M., Parmley, A., Monge-Álvarez, J., Casaseca-de-la-Higuera, P.: A novel infrared video surveillance system using deep learning based techniques. Multim. Tools Appl. 77, 26657–26676 (2018)
Ghose, D., Desai, S.M., Bhattacharya, S., Chakraborty, D., Fiterau, M., Rahman, T.: Pedestrian detection in thermal images using saliency maps. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 988–997 (2019)
Chen, Y., Li, G., Jhong, S., Chen, P., Tsai, C., Chen, P.: Nighttime pedestrian detection based on thermal imaging and convolutional neural networks. Sens. Mater. 32(10), 3157–3167 (2020)
Devaguptapu, C., Akolekar, N., Sharma, M., Balasubramanian, V.: Borrow from anywhere: pseudo multi-modal object detection in thermal imagery. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 1029–1038 (2019)
Heo, D., Lee, E., Ko, B.C.: Pedestrian detection at night using deep neural networks and saliency maps. J. Imaging Sci. Technol. 61(6), 060403-1–060403-9 (2017)
Abbott, R., Del Rincon, J. M., Connor, B., Robertson, N.: Deep object classification in low resolution LWIR imagery via transfer learning. In: IMA Conference on Mathematics in Defence (2017)
Dai, X., Yuan, X., Wei, X.: TIRNet: object detection in thermal infrared images for autonomous driving. Appl. Intell. 51, 1–18 (2020)
Krišto, M., Ivasic-Kos, M., Pobar, M.: Thermal object detection in difficult weather conditions using YOLO. IEEE Access 8, 125459–125476 (2020)
Zagoruyko, S., Komodakis, N.: Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In: International Conference on Learning Representations (2017)
Chen, Y., Zhang, Y.: Drift-free tracking surveillance based on online latent structured SVM and Kalman filter modules. IEICE Trans. Inf. Syst. 101-D, 491–503 (2018)
Bochkovskiy, A., Wang, C.Y., Liao, H.: Yolov4: optimal speed and accuracy of object detection. arXiv:2004.10934 (2020)
He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. In: IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)
Lin, T., Goyal, P., Girshick, R.B., He, K., Dollár, P.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 42, 318–327 (2020)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Jhong, SY., Chen, YY., Hsia, CH. et al. Nighttime object detection system with lightweight deep network for internet of vehicles. J Real-Time Image Proc 18, 1141–1155 (2021). https://doi.org/10.1007/s11554-021-01110-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11554-021-01110-1