Abstract
In this work, a method is presented for the continuous estimation of pain intensity based on fusion of bio-physiological and video features. Furthermore, a method is proposed for the adaptation of the system to unknown test persons based on unlabeled data. First, an analysis is presented that shows which modalities and feature sets are suited best for the task of recognizing pain levels in a person-independent setting. For this, a large set of features is extracted from the available bio-physiological channels (ECG, EMG and skin conductivity) and the video stream. We then propose a method to learn the confidence of a regression system using a multi-stage ensemble classifier. Based on the outcome of the classifier, which is realized by a neural network, confident samples are selected by the adaptation procedure. In various experiments, we show that the algorithm is able to detect highly confident samples which can be used to improve the overall performance. We furthermore discuss the current limitations of automatic pain intensity estimation—in light of the presented approach and beyond.
Similar content being viewed by others
Notes
Although the estimated quantity is based on the regression error, in accordance to the literature, we will call it confidence as it is a value in (0, 1] indicating how certain a classifier is about a sample. Similar techniques have for example been presented by Platt in the context of Support Vector Machines (Platt 1999).
Here, only part A of the dataset is used. See Kächele et al. (2015) for details.
The experiments here have been carried out using the SCL (Ledalab) features, as they are virtually as good as the early fusion, however with only 22 dimensions much more manageable than the 3154 dimensions of all the features concatenated. Further, the results here are based on standardized features, for which the baseline result for SCL (Ledalab) is 1.1122 RMSE.
References
Abdelwahab M, Busso C 2015) Supervised domain adaptation for emotion recognition speech. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19–24, 2015, pp 5058–5062
AlZoubi O, Fossati D, D’Mello S, Calvo RA (2014) Affect detection from non-stationary physiological data using ensemble classifiers. Evol Syst 6(2):79–92
AlZoubi O, Rafael C, Stevens R (2009) Classification of eeg for affect recognition: An adaptive approach. In: Nicholson A, Li X (eds) AI 2009: Advances in Artificial Intelligence, vol 5866., Lecture Notes in Computer ScienceSpringer, Berlin Heidelberg, pp 52–61
Artemiadis PK, Kyriakopoulos KJ (2010) An EMG-based robot control scheme robust to time-varying EMG signal features. Trans Info Tech Biomed 14(3):582–588
Ashraf AB, Lucey S, Cohn JF, Chen T, Ambadar Z, Prkachin KM, Solomon PE (2009) The painful face—pain expression recognition using active appearance models. Image Vision Comput 27(12):1788–1796
Benedek M, Kaernbach C (2010) Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology 47(4):647–658
Cao C, Slobounov S (2011) Application of a novel measure of EEG non-stationarity as ’Shannon- entropy of the peak frequency shifting’ for detecting residual abnormalities in concussed individuals. Clinical neurophysiology : Official Journal of the International Federation of Clinical Neurophysiology 122(7):1314–1321
Cauwenberghs G, Poggio T (2001) Incremental and decremental support vector machine learning. In: Advances in neural information processing systems, vol 13, pp 409–415
Chen W, Zhuang J, Yu W, Wang Z (2009) Measuring complexity using FuzzyEn, ApEn, and SampEn. Medl Eng Phys 31(1):61–68
Duviella E, Serir L, Sayed-Mouchaweh M (2013) An evolving classification approach for fault diagnosis and prognosis of a wind farm. In: Conference on Control and Fault-Tolerant Systems (SysTol) IEEE, pp 377–382
Glodek M, Tschechne S, Layher G, Schels M, Brosch T, Scherer S, Kächele M, Schmidt M, Neumann H, Palm G, Schwenker F (2011) Multiple classifier systems for the classification of audio-visual emotional states. In: D’Mello S, Graesser A, Schuller B, Martin JC (eds) Proceedings of the Fourth International Conference on Affective Computing and Intelligent Interaction (ACII’11) - Part II, LNCS 6975, Springer, pp 359–368
Hammal Z, Cohn JF (2012) Automatic detection of pain intensity. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction, ICMI ’12, ACM, pp 47–52
He L, Jiang D, Yang L, Pei E, Wu P, Sahli H (2015) Multimodal affective dimension prediction using deep bidirectional long short-term memory recurrent neural networks. In: Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, ACM , pp 73–80
Kächele M, Glodek M, Zharkov D, Meudt S, Schwenker F (2014) Fusion of audio-visual features using hierarchical classifier systems for the recognition of affective states and the state of depression. In: M. De Marsico, A. Tabbone, A. Fred (eds.) Proceedings of the International Conference on Pattern Recognition Applications and Methods (ICPRAM), SciTePress, pp 671–678
Kächele M, Schels M, Meudt S, Palm G, Schwenker F (2016) Revisiting the EmotiW challenge: how wild is it really? J Multimodal User Interfaces 10(2):151–162. doi:10.1007/s12193-015-0202-7
Kächele M, Schels M, Schwenker F (2014) Inferring depression and affect from application dependent meta knowledge. In: Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, AVEC ’14, ACM, pp 41–48
Kächele M, Thiam P, Amirian M, Schwenker F, Palm G (2016) Methods for person-centered continuous pain intensity assessment from bio-physiological channels. IEEE J Sel Topics Signal Proces (99), pp 1–1
Kächele M, Thiam P, Amirian M, Werner P, Walter S, Schwenker F, Palm G (2015) Multimodal data fusion for person-independent, continuous estimation of pain intensity. In: Iliadis L, Jayne C (eds) Engineering Applications of Neural Networks, Communications in Computer and Information Science, vol. 517, pp. 275–285. Springer International Publishing
Kächele M, Thiam P, Palm G, Schwenker F, Schels M (2015) Ensemble methods for continuous affect recognition: Multi-modality, temporality, and challenges. In: Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, AVEC ’15, ACM, pp 9–16
Kächele M, Werner P, Al-Hamadi A, Palm G, Walter S, Schwenker F (2015) Bio-visual fusion for person-independent recognition of pain intensity. In: Schwenker F, Roli F, Kittler J (eds) Multiple classifier systems, Lecture notes in computer science, vol. 9132, pp 220–230. Springer International Publishing
Kaltwang S, Rudovic O, Pantic M (2012) Continuous pain intensity estimation from facial expressions. In: Bebis G, Boyle R, Parvin B, Koracin D, Fowlkes C, Wang S, Choi MH, Mantler S, Schulze J, Acevedo D, Mueller K, Papka M (eds) Advances in visual computing, LNCS, vol 7432. Springer, Berlin, pp 368–377
Kuncheva L (2004) Combining pattern classifiers: methods and algorithms. Wiley, Hoboken, New Jersey
Lucey P, Cohn JF, Prkachin KM, Solomon PE, Matthews I (2011) Painful data: the UNBC-McMaster shoulder pain expression archive database. In: IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), IEEE, pp 57–64
Ludl MC, Lewandowski A, Dorffner G (2008) Adaptive machine learning in delayed feedback domains by selective relearning. Appl Artif Intell 22(6):543–557
Phinyomark A, Limsakul C, Phukpattaranont P (2009) A novel feature extraction for robust EMG pattern recognition. J Comput 1(1):71–80
Platt JC (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in large margin classifiers, pp 61–74
Schels M, Glodek M, Meud, S, Scherer S, Schmidt M, Layher G, Tschechne S, Brosch T, Hrabal D, Walter S, Traue H, Palm G, Neumann H, Schwenker F (2013) Multi-modal classifier-fusion for the recognition of emotions, 1 edn. Coverbal synchrony in human-machine interaction. CRC Press, USA, pp 73–97
Schwenker F, Dietrich CR, Thiel C, Palm G (2006) Learning of decision fusion mappings for pattern recognition. Int J Artif Intell Mach Learn (AIML) 6:17–21
Tkach D, Huang H, Kuiken TA (2010) Research study of stability of time-domain features for electromyographic pattern recognition. J Neuroeng Rehabil 7:21
Treister R, Kliger M, Zuckerman G, Aryeh IG, Eisenberg E (2012) Differentiating between heat pain intensities: the combined effect of multiple autonomic parameters. Pain 153(9):1807–1814
Walter S, Gruss S, Ehleiter H, Tan J, Traue H, Werner P, Al-Hamadi A, Crawcour S, Andrade A, Moreira da Silva G (2013) The BioVid heat pain database data for the advancement and systematic validation of an automated pain recognition system. In: IEEE International Conference on Cybernetics (CYBCONF), pp 128–131
Walter S, Gruss S, Limbrecht-Ecklundt K, Traue HC, Werner P, Al-Hamadi A. Diniz N, Silva GMd, Andrade AO (2014) Automatic pain quantification using autonomic parameters. Psychol Neurosci 7:363–380
Wang W, Enescu V, Sahli H (2015) Adaptive real-time emotion recognition from body movements. ACM Trans Interactive Intell Syst 5(4):18:1–18:21
Werner P, Al-Hamadi A, Niese R, Walter S, Gruss S, Traue HC (2014) Automatic pain recognition from video and biomedical signals. In: International Conference on Pattern Recognition, pp 4582–4587
Wöllmer M, Kaiser M, Eyben F, Schuller B, Rigoll G (2013) LSTM-modeling of continuous emotions in an audiovisual affect recognition framework. Image Vision Comput 31(2):153–163
Yang J, Yan R, Hauptmann AG (2007) Cross-domain video concept detection using adaptive svms. In: Proceedings of the 15th ACM International Conference on Multimedia, MM ’07 , ACM pp 188–197
Zhang C Hamid R, Zhang Z (2008) Taylor expansion based classifier adaptation: application to person detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp 1–8
Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. Pattern Analy Mach Intell IEEE Trans 29(6):915–928
Zhao Q, Zhang L (2005) ECG feature extraction and classification using wavelet transform and support vector machines. Int Conf Neural Netw Brain 2:1089–1092
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kächele, M., Amirian, M., Thiam, P. et al. Adaptive confidence learning for the personalization of pain intensity estimation systems. Evolving Systems 8, 71–83 (2017). https://doi.org/10.1007/s12530-016-9158-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12530-016-9158-4