Multimodal Cognitive Workload Assessment Using EEG, fNIRS, ECG, EOG, PPG, and Eye-tracking

• ¹ School of Biomedical Engineering, Science and Health Systems, Drexel University, United States
• ² Lockheed Martin (United States), Advanced Technology Laboratories, United States
• ³ University of Pennsylvania, Department of Family and Community Health, United States
• ⁴ Children's Hospital of Philadelphia, Division of General Pediatrics, United States

Mental workload is a measure of the cognitive effort required to successfully perform a task, and is a function of task difficulty and individual expertise. Understanding the mental workload which is disassociated from the behavioral performance can be used to improve interface design of complex systems and efficiency of human-machine teaming (Ayaz et al 2012). Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) are portable neuroimaging modalities that can have wearable, wireless, and battery-operated sensors to capture neural correlates with complex and natural tasks, aligned with the neuroergonomic approach (Parasuraman and Wilson, 2008; Gramann et al 2017). Recent studies we have conducted have demonstrated the capability of fNIRS and EEG to measure workload in working memory tasks (Ayaz et al 2012; Choe et al 2016; Liu et al., 2017). In this study, we look at fNIRS, EEG, heart rate (ECG), photoplethysmograph (PPG), and several eye movement-related metrics (EOG and eye-tracking) to capture multi-dimensional biomarkers of workload in brain and body measures. Twenty subjects volunteered for this study and completed six unique tasks three times each over the course of four sessions held over one month (6 tasks x 3 repetitions / 4 sessions). Tasks were designed to specifically target six distinct cognitive modes: working memory, inhibitory control, sustained attention, risk assessment, multitasking, and situational awareness. Each task replicated a simplified version of parts of a militaristic mission control, such as using a radar image for working memory, searching unknown areas of interest for risk assessment, and viewing flight missions for situation awareness. Tasks were presented in block format with low workload and high workload conditions (excluding the continuous performance task for attention). Behavioral performance for the tasks within session and across days were consistent with the literature. In spatial working memory, the speed of clicks increased over time while accuracy remained consistent (Fig 1). In inhibitory control (modified go/stop task), accuracy improved for high workload stop blocks, but was consistent for go blocks (F2,353 = 3.04, p < 0.05) (Fig 2). In sustained attention (continuous performance task), accuracy decreased as reaction time increased over the course of each block (Fig 3). In risk assessment, the speed of decision making/clicks increased over sessions (F2,350 = 18.45, p < 0.01), whereas the total number of clicks in each condition was consistent (Fig 4). In multitasking (trail-making task), reaction time decreased for each condition over sessions (F2,346 = 12.51, p < 0.01), but there was a ceiling effect of accuracy for the low workload condition, and improvement in the high workload (F2,345 = 5.05, p < 0.01) (Fig 5). Overall, performance measures of the six tasks were consistent with the expected results. The next step of this study is to analyze the neuroimaging and physiological correlates of mental workload in the context of these results and develop classifiers that can be used to predict performance in subjects. Moreover, a complex protocol assimilating these tasks will be developed to construct a more realistic and ecologically valid model of real world tasks. Figure 1. Spatial working memory task plots demonstrating decreasing reaction time and consistent error that only differs between easy and difficult task conditions Figure 2. Inhibitory control task demonstrating increasing accuracy in more challenging condition (stop) over sessions Figure 3. Vigilance task demonstrating decreasing accuracy and increasing reaction time over 8 evenly timed segments of the 5-minute continuous task period Figure 4. Risk task demonstrating increasing speed but consistent clicks per session Figure 5. Multitasking (trail making task) demonstrating decreasing reaction time over sessions, and improvement in accuracy for high workload condition