Abstract
Pain is an essential physiological phenomenon of human beings. Accurate assessment of pain is important to develop proper treatment. Recently, deep learning has been exploited rapidly and successfully to solve a large scale of image processing tasks. In this paper, we propose a Multi-stream Integrated Neural Networks with Different Frame Rate (MINN) for detecting facial expression of pain. There are four-stream inputs of the MINN for facial expression feature extraction, including the spatial information, the temporal information, the static information, and the dynamic information. The dynamic facial features are learned in both implicit and explicit manners to better represent the facial changes that occur during pain experience. The experiments are conducted on publicly available pain datasets to evaluate the performance of proposed MINN, and the performance is compared with several deep learning models. The experimental results illustrate the superiority of the proposed model on pain assessment with binary and multi-level classification.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gholami, B., Haddad, W.M., Tannenbaum, A.R.: Relevance vector machine learning for neonate pain intensity assessment using digital imaging. IEEE Trans. Biomed. Eng. 57(6), 1457–1466 (2010)
Prkachin, K.M., Solomon, P.E.: The structure, reliability and validity of pain expression: evidence from patients with shoulder pain. Pain 139(2), 267–274 (2008)
Johnston, C.C., Strada, M.E.: Acute pain response in infants: a multidimensional description. Pain 24(3), 373–382 (1986)
Kremer, E., Atkinson, J.H., Ignelzi, R.J.: Measurement of pain: patient preference does not confound pain measurement. Pain 10, 241–248 (1981)
Ho, K., Spence, J., Murphy, M.F.: Review of pain-measurement tools. Ann. Emerg. Med. 27(4), 427–432 (1996)
Willis, M.H.W., Merkerl, S.I., Voepeo-Lewis, T., Malviya, S.: FLACC behavioral pain assessment scale: comparison with the child’s self-report. Pediatr. Nurs. 29(3), 195–198 (2003)
Walter, S., et al.: The biovid heat pain database data for the advancement and systematic validation of an automated pain recognition system. In: 2013 IEEE International Conference on Cybernetics (CYBCO), pp. 128–131. IEEE, Switzerland (2013)
Haque, M.A., et al.: Deep multimodal pain recognition: a database and comparison of spatio-temporal visual modalities. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 250–257. IEEE, China (2018)
Martinez, D.L., Rudovic, O., Picard, R.: Personalized automatic estimation of self-reported pain intensity from facial expressions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 70–79. IEEE, USA (2017)
Zhi, R., Zamzmi, G., Goldgof, D., Ashmeade, T., Su, Y.Y.: Automatic infants’ pain assessment by dynamic facial representation: effects of profile view, gestational age, gender, and race. J. Clin. Med. 7(7), 1–16 (2018)
Chen, J., Chi, Z., Fu, H.: A new framework with multiple tasks for detecting and locating pain events in video. Comput. Vis. Image Underst. 155, 113–123 (2017)
Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size. In: International Conference on Learning Representations (ICLR), Toulon, France (2017)
Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Neural Information Processing Systems (NIPS), pp. 568–576, Montréal, Canada (2014)
Hubel, D.H., Wiesel, T.N.: Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. J. Neurophysiol. 28, 229–289 (1965)
Livingstone, M., Hubel, D.: Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science 240, 740–749 (1988)
Feichtenhofer, C., Fan, H., Malik, J., He, K.: SlowFast networks for video recognition (2018). arXiv:1812.03982
Lin, T.Y., Dollar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Processing of Computer Vision and Pattern Recognition (CVPR), pp. 1–10. IEEE, USA (2017)
Acknowledgement
This research was funded by the National Natural Science Foundation of China, grant number 61673052, the National Research and Development Major Project, grant numbers 2017YFD0400100, the Fundamental Research Fund for the Central Universities of China, grant numbers FRF-GF-19-010A, FRF-TP-18-014A2, FRF-IDRY-19-011. The computing work is supported by USTB MatCom of Beijing Advanced Innovation Center for Materials Genome Engineering.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhi, R., Zhou, C., Yu, J., Liu, S. (2021). Multi-stream Integrated Neural Networks for Facial Expression-Based Pain Recognition. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12662. Springer, Cham. https://doi.org/10.1007/978-3-030-68790-8_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-68790-8_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68789-2
Online ISBN: 978-3-030-68790-8
eBook Packages: Computer ScienceComputer Science (R0)
