fNIRS Feature Importance for Attentional State Prediction

• ¹ Langley Research Center, United States
• ² Booz Allen Hamilton (United States), United States
• ³ University of Virginia, United States
• ⁴ Analytical Mechanics Associates, United States

Introduction: Cognitive state prediction is attractive for the detection of human performance decrement and the mitigation of errors during safety-critical operational activities. Such methods are applicable in ground-based verification studies, and to support countermeasures during training or operational environments such as commercial piloting, air traffic control, or astronaut extra-vehicular activity. In particular, we aim to improve the discrimination of optimal from non-optimal states of attention by fusing data from multiple passive physiological sensors, down-selecting features and tuning machine learning algorithms. Prior work has explored the use of electroencephalography, functional Near Infrared Spectroscopy (fNIRS), galvanic skin response, eye tracking, heart rate, and respiration, but has not fully investigated classifier state prediction accuracy for various types of features extracted from fNIRS signals. Here we perform cognitive state prediction with, and assess the importance of, fNIRS features based on functionally-connected brain networks to inform future real-time cognitive state prediction methods. Methods: Six commercial aviation pilots were asked to perform “benchmark” tasks and motion-based flight simulation while wearing fNIRS and other physiological sensors. All participants consented to the study as approved by the Institutional Review Board of the NASA Langley Research Center. Here we report on the analysis of fNIRS benchmark task data to assess the usefulness of various fNIRS-derived features to discriminate cognitive states. Results of analysis of other physiological data recorded during benchmark tasks and flight simulation were reported elsewhere (Harrivel et al., 2017; Stephens et al., 2017). Use of the benchmark tasks was modeled after Hirshfield, et al. (2009). Selected tasks were used to induce five cognitive states for 6 minutes each: high workload, low workload, startle, channelized attention, and diverted attention (Harrivel et al., 2016a). Two fiber optic sensor arrays were used to record fNIRS signals at 6.25 Hz (Imagent, ISS, Inc.): one with 2 brain-sensing optodes placed over the Medial Frontal Gyrus (MFG, a node of the default mode or task-negative network), and another with 5 brain-sensing optodes placed over the right Dorsolateral Pre-Frontal Cortex (DLPFC, a node of an attentional or task-positive network). Additionally, a shallow physiology-sensing optode was employed on each sensor array. Time traces of relative hemoglobin concentration changes for both oxygenated and deoxygenated species were calculated for each channel. These were band-pass filtered to include 0.003 Hz to 0.1Hz to capture both functional hemodynamic responses, and low frequency spontaneous activity in functionally-connected default mode and attentional networks. Signals measured by the physiology-sensing optodes were filtered to include higher frequencies (0.008 Hz to 3 Hz) to capture a wide range of cardiac and respiratory fluctuations potentially to inform the classifiers. The “as-measured” traces were used as previously described (Harrivel et al., 2016b), without a-priori knowledge of task performance for processing, and without cleaning of physiological noise. Various feature types were extracted from the time series to assess their usefulness for improving classification accuracy. Features were based on spectral information (POWER), statistical summaries (SS), engagement indices (ENG), and correlation (COR). Spectral features included power up to 0.5Hz and the average, variance, and max power within this band. Statistical summaries included moments, coefficient of variation, percentiles, and multivariate and permutation entropy. “Engagement” indices were created using ratios or differences between the MFG and DLPFC traces, quantifying external task engagement using task-positive vs. task-negative network activity (Harrivel et al., 2013; Fox et al., 2005). Correlation-based features include Pearson correlations and eigenvalues of the correlation matrices designed to quantify more complicated network activations as potential indications of mind wandering (Smallwood et al., 2008; Christoff et al., 2009; Durantin et al., 2015). Data were partitioned into training and holdout sets using a 50/50 split accounting for the temporal nature of the data and class imbalance. Feature selection was performed on the training set to find an optimal subset of features using two models: (1) Logistic regression with L1 penalty, and (2) random forests. For logistic regression, one-vs-all models were fit for a total of five models and five sets of coefficients. An embedded feature selection technique was performed where features with non-zero coefficients were selected. For random forests, features with mean decrease importance scores (using the Gini Index) greater than the median importance were selected. For both classifiers, a recursive feature elimination (RFE) wrapper method was used to recursively identify the top 50% of features. Finally, an ensemble feature selector was also implemented by combining greater-than-median features for each model. Each feature subset was evaluated on the holdout data using a 4-fold stratified, temporal cross-validation. For comparison purposes, classifiers were also tested using all features. The evaluation metric was the weighted area under the curve, and weights were adjusted account for class imbalance. Results: Generally, the best results were obtained by performing feature selection prior to model evaluation. The embedded L1 feature selection method resulted in similar performance as the RFE method (6-pilot-average AUC of 0.711 vs 0.713) but used approximately one-fifth of the original feature set, whereas the RFE algorithm using half of the original feature set. For random forests, the best results were obtained using the ensemble selection method. The highest 6-pilot-average AUC was 0.784 +/- 0.110, which resulted from employing the random forest classifier with the ensemble-derived feature subset. The top selected features across all pilots were the spectral power-based features in four of the six pilots (Figure 1). Moreover, across all feature types and all pilots, the oxy-derived features were selected more often than deoxy-derived features. Conclusion: The results of the present study warrant the prioritization of SS, ENG and POWER features in future analyses. Importantly, the power features also include spectral respiratory information, perhaps partly explaining their superior ability to increase AUC. Future work includes state prediction using data collected during the flight simulation. Further, fNIRS will be combined with other physiological sensors as previously reported (Harrivel et al., 2017; Stephens et al., 2017). Real-time operator cognitive state information may be relayed to flight instructors during training, or to semi-autonomous systems to optimize human-automation interaction and teaming.