THEORIES AND METHOD FOR LABELING COGNITIVE WORKLOAD: CLASSIFICATION AND TRANSFER LEARNING

• ¹ Northrop Grumman (United States), United States
• ² George Mason University, United States

There are a number of key data-centric questions that must be answered when developing classifiers for operator functional states. First among which is, should a supervised or unsupervised learning approach be used? If a supervised approach is desired, we need to ask what degree of labeling and transformation must be performed on the data? When considering the classification method, we have to consider the trade-off between algorithm flexibility and model interpretability, as generally these features are at odds? Here we focus exclusively on labeling cognitive load data for supervised learning with neurophysiological data. We explored three methods of labeling cognitive states using a two-state classification, overload or adequate load. The first, and most simplistic, method uses a median split of the number of items an individual had to hold in mind during each trial of a spatial memory task. The second method was more adaptive; it employed a mixed effects stress-strain curve and estimated an individual’s performance asymptotes with respect to the same spatial task. These asymptotes were combined with a rule stating that if the task demand exceeded the asymptote and an error was made, classify this condition as overload, otherwise as adequate load. The final method was similar to the second approach: it employed a mixed effects Rasch model to estimate individual capacity limits within the context of item response theory for the memory task. We then applied the same rule as in approach two to differentiate overload and adequate load. To assess the strength of each of these labeling approaches we compared area under the curve (AUC) for receiver operating curves (ROC) as well the AUC of precision-recall ROCs (PR-ROC) from elastic net and random forest classifiers. We chose these classifiers based on their balance of interpretability, flexibility, and past modeling successes. We also utilized these classifiers on a synthetic intelligence, surveillance and reconnaissance (ISR) task to test the efficacy of transfer learning a cognitive load model between tasks. Our synthetic ISR task, like the spatial memory task, taxed the memory of participants and observed their performance. Our metrics for the model’s accuracy again included AUC, ROC, and PR-ROC curves. These metrics were compared to models we created specifically for the ISR task. We also created an artificial intelligence (AI) agent that used the output of the classifiers to adapt our ISR task in order to attempt to keep the operator in the adequate load state. To measure the agent’s success, we gathered metrics on the human-agent team performance. We end with a discussion of this measure of system of systems performance in relation to adaptive automation and transfer learning of fundamental cognitive constructs across tasks.