Towards a hybrid passive BCI for the modulation of sustained attention using EEG and fNIRS

• ¹ HEC Montréal, Université de Montréal, Canada

Context Advances in information technologies, such as artificial intelligence and robotics are reshaping the way in which we interact with technology (Autor D.H. 2015). Tasks commonly performed through human labour are becoming increasingly automated, creating vast subsets of tasks that require a high level of decision readiness and a high degree of sustained attention to monitor complex systems and the data they create. Users of modern software, ranging from critical systems infrastructure, network security frameworks to business logistics require the ability to quickly synthesize and interpret a wide variety of information, in order to make correct and timely decisions. However, the rapid adoption of automated process for administration and analysis tasks has resulted in a potentially hazardous business mindset that considers the human element as a secondary function (Warm et. al. 2008). Studies have shown that while automation has increased productivity through a reduction in information-processing and cognitive load, it has also decreased on-task safety profiles, resulting in incidents that range in scale from minor to major (such as air traffic controller error) . These incidents triggered by on-task monitoring surprise are often the result of a decrease in operator vigilance and sustained attention (Molloy & Parasuraman, 1996, De Boer & Dekkar 2017). Brain Computer Interfaces (BCI) have the potential to provide a means –through interactive countermeasures– to modulate sustained attention (SA) and reduce operator error. We report on progress towards a contribution in this area, through a passive hybrid brain-computer interface designed to modulate a user’s level of sustained attention measured using EEG and fNIRS and an autoadaptive information system interface which creates attentional signals to encourage changes in the operating environment determined by a user’s level of sustained attention. Methods From a proposed participant pool of N= 30, 16 participants (7 female) aged 18-43 (Avg. = 24), have so far taken part in the study. Participants were screened on the basis of good health, average hair density, and normal or corrected to normal vision. All participants signed consent in line with the University’s ethics board. Participants were provided with a mouse and keyboard and sat approximately 80 cm in front of a 24” computer screen. We utilise Pope et. al’s (1995,2002) engagement index to provide real-time assessment of a user’s attentional state and to drive the neurofeedback mechanism of the simulation task interface (see Figure 1.). We used a 32 electrode EEG (Brainvision, Morrisville, NC), to measure variations in brainwave activity in the θ (4-7Hz), α (8-12Hz) and β (13-21Hz) bands calculated as a ratio using (power) β / (α + θ) from F3, F4, O1, O2 on the international 10–20 system. In addition to EEG, we record NIRs data (NIRSport, NIRx, Berlin) concurrently from adjacent sites using a custom 6 channel montage, 2 channels each (1 source 2 detector) covering f3-f4, 1 channel each (1 source 1 detector) covering O1-O2. The experimental task is split into 2 parts: calibration, lasting 22 mins, and testing, lasting 90 mins. Calibration is composed of a 1 min baseline (passive observation), then an engagement task of 10 mins, then a 1 min baseline and a vigilance task of 10 mins. In the testing phase of task, participants are randomly assigned to one of 3 counter-measure conditions 1. no CM, 2. continuous CM and 3. event-based CM, both active CM conditions are controlled by a user’s level of sustained attention. Participants are asked to complete an ecologically valid information system task in an enterprise system (SAP, Waldorf) in a simulated business environment generated via ERPsim (ERPsimLab, Montreal) . The task itself involves maintaining stock levels in 3 locations, and participants are asked to make logistical decisions concerning stock allocation. Stock depletion rates are non-uniform and dependent on different demand functions. A maximum stock capacity is provided to force decisions as soon as new stock is received, and all correct, incorrect, and missed decisions are logged for later analysis. To induce a vigilance decrement in the participant, time moves at a slower pace within the simulation, this creates a monitoring task requiring a high level of sustained attention. Analysis Our primary hypothesis for the study is to determine if modulating sustained attention through interface countermeasures using a BCI improve task performance. However, the BCI artefact has two aims, 1.) To operate in real-time, present the simulated work environment, drive the SA neurofeedback mechanism used to provide countermeasures, and provide user performance metrics. 2.) To provide the experimental timeline and stimuli from which to record SA related fNIRS data for post hoc analysis. From these two aims we wish to answer a number of research questions: Is fNIRS a viable real-time measure of cortical activity involved in SA, during real-world tasks that do not conform to a standard laboratory baseline-stimulus-response iterative epoch protocol; Using the EEG derived SA index as a reference, can we synchronise the fNIRS data, to model the haemodynamic response function (HRF), for participants SA over the long duration experimental task to determine any significant commonality of effect; From the analysis of the fNIRS response data determine important features of the HRF for extraction and use within machine learning classification; Using the SA index data, performance statistics, participant subjective assessment of SA and the fNIRS data gathered during calibration, determine ground truth labels for training a machine learning algorithm and test the classification model(s) using the long duration task data. With respect to the NIRS data, if classification of these data proves successful we aim to propose a passive hybrid BCI framework for modulating SA that combines measures of EEG and fNIRS with computational and machine learning classification to drive interface countermeasure. We welcome discussion and feedback with regards to methods, classification models and overarching neurofeedback mechanism.