Using Brain Painting at Home for 5 Years: Stability of the P300 During Prolonged BCI Usage by Two End-Users with ALS

Abstract

Brain painting (BP) is non-invasive electroencephalography (EEG) based Brain-Computer Interface (BCI) for creative expression based on a P300 matrix. The technology was transferred into a home setup for two patients with amyotrophic lateral sclerosis (ALS), who used the system for several years while being evaluated on performance and satisfaction. Holz and colleagues found that the use of BP increased quality of life. Additionally, they described that changes in the amplitude of the P300 ERPs could be observed between recalibrations of the BCI. In this paper, we quantified the evolution of the P300 peaks in the two BCI end-users (HP and JT). For HP, the P300 peak amplitude increased during 9 months, then progressively decreased for the following 51 months, but the BCI accuracy remained stable. JT’s P300 peak amplitude did not significantly decrease during 32 months that separated the calibrations. Yet, JT’s BCI accuracy declined which we may attribute to a decline in physical functioning due to ALS. Painters used online BCI for hundreds of hours (HP 755, JT 223) and both finished more than 50 named brain paintings. HP could use BP autonomously and regularly at home for 33 months without recalibration of the system, and JT for 10 months, suggesting the stability of P300 and SWLDA online classifiers in the long-term, and demonstrating the feasibility of having a P300 based system at home that requires few involvement of BCI experts.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Brain painting (BP) is a brain-computer interface (BCI) controlled application that allows for creative expression. It is based on the well known P300 Speller paradigm [4], but with the particularity of presenting a matrix of icons that send commands to a virtual canvas. This non-invasive electroencephalography (EEG) based system was developed and successfully tested with healthy participants and patients [11]. To fulfill the translational purpose of the project, we adapted BP for independent home use, by simplifying the user interface, therefore allowing caregivers to operate the system without the direct assistance of BCI experts. BP was installed at the home of two end-users (HP, aged 78 and JT, aged 77). Both end-users, in the locked-in state due to amyotrophic lateral sclerosis (ALS), used the system assisted by their caregivers and family without requiring the presence of researchers on site. Further software and development cycles, following the user centered design (see [10] for guidelines) were implemented and led to a new BP version allowing users to draw lines, curves and select new forms. BP2 was tested with healthy participants, and installed at JT’s home [1]. During several years and hundreds of sessions of painting, we regularly assessed HP’s and JT’s satisfaction, perceived control level and impact of BP on their quality of life [5, 6]. Holz and colleagues previously found that the peak amplitude at Cz increased in 9 months of evaluation and decreased after 26 months, while Sellers and colleagues [12] showed that the ERPs remained stable across 18 months in a patients with ALS in the locked-in-state.

2 Objective

The amplitude of the P300 ERP is the physiological marker for BCI performance, which we acquired during recalibrations of the BCI over the extended period of 60 months for HP, and of 32 months for JT. We report in the present paper the evolution of the P300 ERP across the evaluated period and discuss its stability for the independent home use of BP.

3 Methods

3.1 BCI Feedback

To bring BP out of the lab to the end-users’ home, we were strictly following the user-centered design [10]. We adapted the BP system (BP1 [11]) such that it could be used independently at home without requiring an expert to be present. Satisfaction, frustration, workload and quality of life were assessed and were reported in other studies [5, 6]. As a first upgrade, we introduced a new feedback modality that replaced classical P300 intensification of the characters by overlays with the face of Einstein [7, 8], which were proven to lead to better performance, also in patients. Additionally, a minor upgrade of the feedback was also implemented, allowing the end-users to immediately see their selections within the matrix, which they initially had to deduce from a status bar. JT received directly the upgraded BP1, with the face paradigm. After a while using BP1, participants expressed the wish to extend the functionalities of the interface, notably the ability to draw lines, obtain more shapes than only circles or squares, and then be able to fill the whole screen at once. Thus, BP2 was developed and integrated all those requests, with even more features such as writing text or inserting external images [1]. Those were evaluated in healthy participants [3], but the number of functionalities proposed in the matrices provided to the end-users was simplified to match their initial wishes, while avoiding overload with too many options.

Yet, BP2 contains line drawing functions, gradients and additional shapes. The line drawing feature relies on placing crosses at specific locations on the canvas, which are then linked. For more than 3 crosses, lines were automatically connected to each others forming a path. The path could be closed using the “open/close” path command. Instead of tracing a line, the closed path could be filled with color using the “fill/trace” path command allowed to switch between tracing a line for the path or filling its content. A smoothing function was also added to allow more natural paths to be drawn. (See Fig. 1 for a simulation of BP2 use for drawing shapes, lines and gradients)

Fig. 1.

Screenshots of BP2 simulating painting based on the new features (i.e. lines, curves, gradients, fullscreen) in 30 steps totalizing 53 selections. The thumbnails indicate the commands that were selected within the BP matrices during the simulation.

BP2 also facilitated caregiver’s operation and allowed for short auto-calibration of the BCI, which provided a quick performance assessment, and a more robust adaptation to issues such as broken electrodes, wrong placement of the cap or lack of EEG gel. While those regular issues are easily spotted and solved in a laboratory environment, they represent a core challenge in home setups, especially during the first months of use.

3.2 Participants

Two patients with ALS in the locked-in state were provided with a BCI system at home. The study was approved by the Ethical Review Board of the Medical Faculty at the University of Tübingen.

HP is a female 77 years-old patient with ALS (spinal form). She was diagnosed in 2007, and is completely paralyzed except eye-movements, allowing her to use an eye-tracker for communicating, browsing Internet or sending emails. HP lives in her family house surrounded by her family and is permanently under caregiver supervision who oversee the operation of the artificial respirator and feeding. When she contacted the university of Würzburg, HP had an EEG amplifier (granted by g.tec¹). As a former hobby painter, HP expressed interest in trying BP in her home environment, and received the first iteration of BP adaptation to home use in Jan. 2012. She then kept using the system within a period of 5 years and is by now still using it albeit we stopped data collection for evaluation.

JT is a male 76 years-old patient with ALS. He was diagnosed in 2006, was able to normally talk until 2014. Afterwards, he nevertheless retained the ability to move facial muscles enabling people to read his lips. JT was surrounded by close relatives and was under permanent supervision to oversee the function of the artificial respirator and feeding. JT was a retired architect and professional painter. After loosing grasp function due to ALS, JT was determined to keep painting; after contacting the university of Würzburg, he received BP at home in Sept. 2013. In the course of the year 2014, JT presented fasciculations in the neck that were both painful and created artifacts on the EEG. Retraining the classifier was sufficient to solve the issue². JT kept actively painting until July 2015. But due to multiple health issues (e.g., neck pain) and practical reasons (assistant left his service), JT did not manage to regularly use BP after this period. JT deceased in Jan. 2017.

3.3 Calibrations

In the course of 5 years of use, HP had five calibration sessions of BP1. We visited HP four times to recalibrate the system, while the last session was remotely supervized using shared desktop and voice communication. On the third session, we introduced the Einstein face paradigm. JT used BP1 for 10 months, after which we installed BP2 and recalibrated the system three times in the following 11 months of painting. The auto-calibration function was used once under researchers supervision on site, and JT continued using BP for 3.5 months until his assistant left his service. Four auto-calibration sessions, which were conducted in 2016 by untrained caregivers, did not provide a good signal quality, and were excluded from the analysis. Helped by his former assistant, JT performed a successful auto-calibration in June 2016 that is reported here. In total, we report 5 of the 9 calibrations that JT performed during a period of 32 months. Details about the timeline of calibration sessions is shown in Table 1.

Table 1. Offline calibration runs used for both online classifier training and statistical analysis

3.4 Signal Processing and Classification

The feedback was provided by two LCD monitors directly in the field of view of the end-users, such that they could move their eyes between monitors. The P300 stimulation monitor was placed directly in front of them at a distance of about 60 cm, while the canvas monitor was placed on the right side.

The EEG was measured with a cap that allowed for easy set-up (g.GAMMAcap) holding eight active electrodes (Fz, Cz, P3, Pz, P4, PO3, PO4 and Oz, conform to the 10–20 system), and connected to a g.GAMMAbox and g.USBAmp amplifier (gtec.at, Austria). The EEG signal was recorded with a sampling rate of 256 Hz and was band-pass filtered between .1 and 40 Hz for both online and offline data analysis³. Both BP1 and BP2 relied on BCI2000 for signal acquisition and processing. For each calibration, a minimum of 5 target selections with 15 repetitions was collected in a supervised fashion, resulting in a minimum of 150 targets repetitions and 900 non-targets repetitions. The EEG activity was extracted between 0 to 800 ms post stimulation. Averaged features and their $r^2$ determination coefficient were calculated using the P300-GUI (BCI2000 toolbox) based on Matlab (The MathWorks). The features were used to train a stepwise linear discriminant analysis (SWLDA) classifier [9]. The minimum number of sequence to reach 100% cross-fold validation accuracy provided by P300GUI, plus two was used for online painting. This number was selected when installing BP1 and BP2. Then the number was manually adjusted by the experimenters based on end-users’ requests and cross-validation results.

3.5 P300 Peak Analysis

For the analysis of the P300 peaks, we selected the midline electrodes (Fz, Cz, Pz and Oz). To determine the temporal – time – position of target P300 peaks, we used the maximum $r^2$ between averaged target and non-targets amplitude, estimated for each calibration run and each electrode. No specific temporal constraint was applied for the identification of the peak (i.e. between 0 and 800 ms). Then, the peak amplitude at the corresponding time point was collected for every target stimulus.

To determine how the amplitude of the P300 ERP evolved during the years, and whether it was larger in different EEG channels, we conducted for each end-user a type III full-factorial ANOVA with target ERP peak as dependent variable. Calibration run and channel were used as between factors. For each combination of calibration run and channel, there was a minimum of 150 target peaks that entered the ANOVA. Post hoc tests were performed using Tukey’s honest significant differences test (HSD).

4 Results

4.1 HP

The average target peak amplitude on midline electrodes was $M=3.8\,\mathrm {\mu V}$, $SEM=.17$. The ANOVA for HP yielded an interaction between channel and calibration $(F(12,8980)=6.5$, $p<.001)$, a main effect of calibration $F(4,8980)=63.5$, $p<.001)$, and a main effect of electrode $(F(3,8980)=90$, $p<.001)$. Pairwise comparisons for the interaction between electrode and channel revealed that the target peak amplitude at Fz and Cz increased between the 1$^\mathrm{st}$ and the 3$^\mathrm{rd}$ calibration ($p_{adj}<.001$), then decreased between the 3rd and the 5th ($p_{adj}<.001$). For the parieto-occipital channels Pz and Oz, the target peak amplitude was significantly lower between the 1st calibration and calibrations 2, 3 and 5 ($p_{adj}<.001$), but did not significantly differ between each others for calibrations 2, 3, 4 and 5, excepted for calibrations 3 and 4 at Pz ($p_{adj}<.05$, see Fig. 2).

Fig. 2.

Average plots using data acquired during online BCI calibration of HP, arranged by EEG electrode and calibration session. (a) The target and non-target ERPs, with a number indicating their amplitude difference. Points indicate the onset of the flash stimuli, and the black bars represent the determination coefficient of the corresponding features. (b) Average amplitude and standard error of the mean of the target peaks. Adjusted $\alpha$ levels: .001***, .01**, .05*.

Considering only brain painting sessions that lasted more than five minutes; in 5 years of using BP, HP had 484 painting sessions totalizing 755 h of painting via BCI ($M=93.7$ min, $SD=50.6$). Hp made more than 50 named paintings (see Fig. 3).

Fig. 3.

Four original brain paintings made by HP using BP1 during 2016.

4.2 JT

For JT, average target peak amplitude on midline electrodes was $M=4.5\, \mathrm {\mu V}$, $SEM=.16$. The ANOVA for JT yielded no significant main effects of calibration $(F(4,5980)=1.4, p=.21)$, channel $(F(3,5980)=.54$, $p=.66)$ and no interaction between channel and calibration $(F(12,5980)=1.3$, $p=.24$, see Fig. 4). In a period of 2.5 years, JT had 225 painting sessions totalizing 223 h of painting via BCI ($M=59.6$ min, $SD=33.1$). Using BP1, JT created 31 named paintings, and 19 with BP2, plus an additional serie of 10 paintings that are meant to be later integrated as models to follow in a tutorial for beginners (see Fig. 5).

Fig. 4.

Average plots using data acquired during online BCI calibration of JT, arranged by EEG electrode and calibration session. (a) The target and non-target ERPs, with a number indicating their amplitude difference. Points indicate the onset of the flash stimuli, and the black bars represent the determination coefficient of the corresponding features. (b) Average amplitude and standard error of the mean of the target peaks.

Fig. 5.

Twelve original paintings by JT using BP2 during 2015. The serie of 10 smaller painting are intended to be translated into a BP2 beginners’ tutorial.

5 Discussion

The current study with two end-users demonstrates that people in the locked-in state due to neurodegenerative disease can use a BCI at home for many years. Both HP and JT could paint for years based on a single calibration. Between calibration 4 and 5, HP painted during 33 months before we remotely performed another calibration. This observation demonstrates the stability of the P300 and the user friendliness of the BCI. Interrestingly, HP’s P300 increased for 9 months which may indicate a better focus of attention due to training. The next calibrations revealed a reduced amplitude which may be attributed to disease progression but was not concomitant of changes made to the BCI paradigms (i.e. Einstein’s faces). JT’s P300 amplitude remained stable across the years despite physical decline from severe paralysis to the locked-in state, this decline was also reflected in his satisfaction and control levels [2]. Both artists continued to produce creative output through the years.

Although the autocalibration feature worked well in preliminary tests in the lab, the conditions in which such calibrations took place by JT were not reliable (i.e. untrained caregiver, no regular practice), underlining the necessity of thorough training of significant others who support BCI use at home. The auto-calibration function should not be used at the very beginning of BCI home use. Instead, experts must set-up the system properly and ensure well function through remote supervision. Another option would be to record a few selections before starting a daily session, such that the newly acquired data can be compared with the actual classifier. Such a functionality would be convenient to evaluate more accurately the bias between self-reported accuracy and true accuracy. Yet, later when end-users – caregivers and patients alike – are familiar with BCI set-up and use, the autocalibration function can be introduced.

6 Conclusion

In the light of these results, and previous findings [5, 6, 12], we suggest that the P300 is a sufficiently stable EEG component that may be used in a BCI setup for years in patients with neurodegenerative disease. Furthermore, the SWLDA classifier used for target selection is sensitive even to smaller ERP amplitudes. Therefore, BP – or any P300 based BCI – is an option for (creative) expression and interaction usable at home without experts being present, even after years in the locked in state due to the neurodegenerative ALS.