Abstract
Automatic airplane detection in aerial imagery has a variety of applications. Two of the significant challenges in this task are variations in the scale and direction of the airplanes. To solve these challenges, we present a rotation-and-scale-invariant airplane proposal generator. We call this generator symmetric line segments (SLS) that is developed based on the symmetric and regular boundaries of airplanes from the top view. Then, the generated proposals are used to train a deep convolutional neural network for removing non-airplane proposals. Since each airplane can have multiple SLS proposals, where some of them are not in the direction of the fuselage, we collect all proposals corresponding to one ground truth as a positive bag and the others as the negative instances. To have multiple instance deep learning, we modify the loss function of the network to learn from each positive bag at least one instance as well as all negative instances. Finally, we employ non-maximum suppression to remove duplicate detections. Our experiments on NWPU VHR-10 and DOTA datasets show that our method is a promising approach for automatic airplane detection in very high-resolution images. Moreover, we estimate the direction of the airplanes using box-level annotations as an extra achievement.
Similar content being viewed by others
References
Amores, J.: Multiple instance classification: review, taxonomy and comparative study. Artif. Intell. 201, 81–105 (2013)
An, Z., Shi, Z., Teng, X., Yu, X., Tang, W.: An automated airplane detection system for large panchromatic image with high spatial resolution. Optik Int. J. Light Electron Opt. 125(12), 2768–2775 (2014)
Bai, X., Zhang, H., Zhou, J.: Vhr object detection based on structural feature extraction and query expansion. IEEE Trans. Geosci. Remote Sens. 52(10), 6508–6520 (2014)
Bo, S., Jing, Y.: Region-based airplane detection in remotely sensed imagery. In: 2010 3rd International Congress on Image and Signal Processing (CISP), vol. 4, pp. 1923–1926. IEEE (2010)
Cai, H., Su, Y.: Airplane detection in remote sensing image with a circle-frequency filter. In: International Conference on Space Information Technology, vol. 5985, p. 59852T. International Society for Optics and Photonics (2006)
Chen, J., Wan, L., Zhu, J., Xu, G., Deng, M.: Multi-scale spatial and channel-wise attention for improving object detection in remote sensing imagery. IEEE Geosci. Remote Sens. Lett. 17(4), 681–685 (2019)
Cheng, G., Han, J.: A survey on object detection in optical remote sensing images. ISPRS J. Photogram. Remote Sens. 117, 11–28 (2016)
Cheng, G., Han, J., Guo, L., Qian, X., Zhou, P., Yao, X., Hu, X.: Object detection in remote sensing imagery using a discriminatively trained mixture model. ISPRS J. Photogram. Remote Sens. 85, 32–43 (2013)
Cheng, G., Han, J., Zhou, P., Guo, L.: Multi-class geospatial object detection and geographic image classification based on collection of part detectors. ISPRS J. Photogram. Remote Sens. 98, 119–132 (2014)
Cheng, G., Han, J., Zhou, P., Guo, L.: Scalable multi-class geospatial object detection in high-spatial-resolution remote sensing images. In: Geoscience and Remote Sensing Symposium (IGARSS), 2014 IEEE International, pp. 2479–2482. IEEE (2014)
Cheng, G., Zhou, P., Han, J.: Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images. IEEE Trans. Geosci. Remote Sens. 54(12), 7405–7415 (2016)
Cheng, G., Zhou, P., Yao, X., Yao, C., Zhang, Y., Han, J.: Object detection in VHR optical remote sensing images via learning rotation-invariant hog feature. In: 2016 4th International Workshop on Earth Observation and Remote Sensing Applications (EORSA), pp. 433–436. IEEE (2016)
Cheng, M.M., Zhang, Z., Lin, W.Y., Torr, P.: Bing: binarized normed gradients for objectness estimation at 300fps. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3286–3293 (2014)
Chollet, F., et al.: Keras. https://keras.io (2015)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, vol. 1, pp. 886–893. IEEE (2005)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 248–255. IEEE (2009)
Deng, Z., Sun, H., Zhou, S., Zhao, J., Lei, L., Zou, H.: Multi-scale object detection in remote sensing imagery with convolutional neural networks. ISPRS J. Photogram. Remote Sens. 145, 3–22 (2018)
Ding, J., Xue, N., Long, Y., Xia, G.S., Lu, Q.: Learning roi transformer for detecting oriented objects in aerial images. arXiv preprint arXiv:1812.00155 (2018)
Duda, R.O., Hart, P.E.: Use of the Hough transformation to detect lines and curves in pictures. Commun. ACM 15(1), 11–15 (1972)
Farooq, A., Hu, J., Jia, X.: Efficient object proposals extraction for target detection in VHR remote sensing images. In: 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 3337–3340. IEEE (2017)
Gao, F., Xu, Q., Li, B.: Aircraft detection from VHR images based on circle-frequency filter and multilevel features. Sci. World J. 2013, 1–7 (2013)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)
Hosang, J., Benenson, R., Dollár, P., Schiele, B.: What makes for effective detection proposals? IEEE Trans. Pattern Anal. Mach. Intell. 38(4), 814–830 (2016)
Hosomura, T., et al.: Airplane extraction from high resolution satellite image using boundary feature. In: Proceedings of ISPRS Technical Com. VIII Symposium (2010)
Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261–2269 (2017). https://doi.org/10.1109/CVPR.2017.243
Humayun, A., Li, F., Rehg, J.M.: Rigor: Reusing inference in graph cuts for generating object regions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 336–343 (2014)
Inglada, J.: Automatic recognition of man-made objects in high resolution optical remote sensing images by SVM classification of geometric image features. ISPRS J. Photogram. Remote Sens. 62(3), 236–248 (2007)
James, J., Ford, J.J., Molloy, T.L.: Below horizon aircraft detection using deep learning for vision-based sense and avoid. In: 2019 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 965–970. IEEE (2019)
Kawato, S., Tetsutani, N.: Circle-frequency filter and its application. In: ITE Technical Report 25.12, pp. 49–54. The Institute of Image Information and Television Engineers (2001)
Khan, M.J., Khan, H.S., Yousaf, A., Khurshid, K., Abbas, A.: Modern trends in hyperspectral image analysis: a review. IEEE Access 6, 14118–14129 (2018)
Khan, M.J., Yousaf, A., Javed, N., Nadeem, S., Khurshid, K.: Automatic target detection in satellite images using deep learning. J. Space Technol. 7(1), 44–49 (2017)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kumar, G., Bhatia, P.K.: A detailed review of feature extraction in image processing systems. In: 2014 Fourth International Conference on Advanced Computing & Communication Technologies (ACCT), pp. 5–12. IEEE (2014)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Li, Y., Fu, K., Sun, H., Sun, X.: An aircraft detection framework based on reinforcement learning and convolutional neural networks in remote sensing images. Remote Sensing 10(2), 243 (2018)
Li, Y., Sun, X., Wang, H., Sun, H., Li, X.: Automatic target detection in high-resolution remote sensing images using a contour-based spatial model. IEEE Geosci. Remote Sens. Lett. 9(5), 886–890 (2012)
Li, Z., Itti, L.: Saliency and gist features for target detection in satellite images. IEEE Trans. Image Process. 20(7), 2017–2029 (2011)
Li, Z., Xu, K., Xie, J., Bi, Q., Qin, K.: Deep multiple instance convolutional neural networks for learning robust scene representations. IEEE Trans. Geosci. Remote Sens. 58(5), 3685–3702 (2020)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. (2018). https://doi.org/10.1109/TPAMI.2018.2858826
Liu, Q., Xiang, X., Wang, Y., Luo, Z., Fang, F.: Aircraft detection in remote sensing image based on corner clustering and deep learning. Eng. Appl. Artif. Intell. 87, 103333 (2020)
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: SSD: single shot multibox detector. In: European Conference on Computer Vision, pp. 21–37. Springer, Berlin (2016)
Long, Y., Gong, Y., Xiao, Z., Liu, Q.: Accurate object localization in remote sensing images based on convolutional neural networks. IEEE Trans. Geosci. Remote Sens. 55(5), 2486–2498 (2017)
Luo, Q., Shi, Z.: Airplane detection in remote sensing images based on object proposal. In: 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 1388–1391. IEEE (2016)
Petridis, S., Geyer, C., Singh, S.: Learning to detect aircraft at low resolutions. In: International Conference on Computer Vision Systems, pp. 474–483. Springer (2008)
Qiu, S., Wen, G., Deng, Z., Fan, Y., Hui, B.: Automatic and fast PCM generation for occluded object detection in high-resolution remote sensing images. IEEE Geosci. Remote Sens. Lett. 14, 1730–1734 (2017)
Qiu, S., Wen, G., Fan, Y.: Occluded object detection in high-resolution remote sensing images using partial configuration object model. IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 10(5), 1909–1925 (2017)
Qiu, S., Wen, G., Liu, J., Deng, Z., Fan, Y.: Unified partial configuration model framework for fast partially occluded object detection in high-resolution remote sensing images. Remote Sens. 10(3), 464 (2018)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)
Seelan, S.K., Laguette, S., Casady, G.M., Seielstad, G.A.: Remote sensing applications for precision agriculture: a learning community approach. Remote Sens. Environ. 88(1–2), 157–169 (2003)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Sizkouhi, A.M.M., Aghaei, M., Esmailifar, S.M., Mohammadi, M.R., Grimaccia, F.: Automatic boundary extraction of large-scale photovoltaic plants using a fully convolutional network on aerial imagery. IEEE J. Photovolt. 10(4), 1061–1067 (2020)
Sun, H., Sun, X., Wang, H., Li, Y., Li, X.: Automatic target detection in high-resolution remote sensing images using spatial sparse coding bag-of-words model. IEEE Geosci. Remote Sens. Lett. 9(1), 109–113 (2012)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
Tan, M., Le, Q.V.: Efficientnet: rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946 (2019)
Uijlings, J.R., Van De Sande, K.E., Gevers, T., Smeulders, A.W.: Selective search for object recognition. Int. J. Comput. Vision 104(2), 154–171 (2013)
Von Gioi, R.G., Jakubowicz, J., Morel, J.M., Randall, G.: LSD: a fast line segment detector with a false detection control. IEEE Trans. Pattern Anal. Mach. Intell. 32(4), 722–732 (2010)
Wang, S., Wang, M., Zhao, X., Liu, D.: Two-stage object detection based on deep pruning for remote sensing image. In: International Conference on Knowledge Science, Engineering and Management, pp. 137–147. Springer (2018)
Wu, H., Zhang, H., Zhang, J., Xu, F.: Fast aircraft detection in satellite images based on convolutional neural networks. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 4210–4214. IEEE (2015)
Xia, G.S., Bai, X., Ding, J., Zhu, Z., Belongie, S., Luo, J., Datcu, M., Pelillo, M., Zhang, L.: Dota: A large-scale dataset for object detection in aerial images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3974–3983 (2018)
Xu, C., Li, C., Cui, Z., Zhang, T., Yang, J.: Hierarchical semantic propagation for object detection in remote sensing imagery. IEEE Trans. Geosci. Remote Sens. 58(6), 4353–4364 (2020)
Xu, Z., Xu, X., Wang, L., Yang, R., Pu, F.: Deformable convnet with aspect ratio constrained NMS for object detection in remote sensing imagery. Remote Sens. 9(12), 1312 (2017)
Yu, Y., Guan, H., Ji, Z.: Rotation-invariant object detection in high-resolution satellite imagery using superpixel-based deep Hough forests. IEEE Geosci. Remote Sens. Lett. 12(11), 2183–2187 (2015)
Yu, Y., Guan, H., Zai, D., Ji, Z.: Rotation-and-scale-invariant airplane detection in high-resolution satellite images based on deep-Hough-forests. ISPRS J. Photogram. Remote Sens. 112, 50–64 (2016)
Zhang, W., Jiao, L., Liu, X., Liu, J.: Multi-scale feature fusion network for object detection in vhr optical remote sensing images. In: IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, pp. 330–333. IEEE (2019)
Zhang, W., Lv, W., Zhang, Y., Tian, J., Ma, J.: Unsupervised-learning airplane detection in remote sensing images. In: MIPPR 2015: Remote Sensing Image Processing, Geographic Information Systems, and Other Applications, vol. 9815, p. 981503. International Society for Optics and Photonics (2015)
Zhang, X., Zhu, K., Chen, G., Tan, X., Zhang, L., Dai, F., Liao, P., Gong, Y.: Geospatial object detection on high resolution remote sensing imagery based on double multi-scale feature pyramid network. Remote Sens. 11(7), 755 (2019)
Zheng, J., Xi, Y., Feng, M., Li, X., Li, N.: Object detection based on bing in optical remote sensing images. In: 2016 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 504–509. IEEE (2016)
Zhong, Y., Han, X., Zhang, L.: Multi-class geospatial object detection based on a position-sensitive balancing framework for high spatial resolution remote sensing imagery. ISPRS J. Photogram. Remote Sens. 138, 281–294 (2018)
Zitnick, C.L., Dollár, P.: Edge boxes: locating object proposals from edges. In: European Conference on Computer Vision, pp. 391–405. Springer, Berlin (2014)
Zou, Z., Shi, Z.: Random access memories: a new paradigm for target detection in high resolution aerial remote sensing images. IEEE Trans. Image Process. 27(3), 1100–1111 (2018)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Mohammadi, M.R. Deep multiple instance learning for airplane detection in high-resolution imagery. Machine Vision and Applications 32, 36 (2021). https://doi.org/10.1007/s00138-020-01153-7
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00138-020-01153-7