Cross-Subject Cognitive Load Recognition in VR Using Multimodal Fusion with EEG and Eye-Tracking

In virtual reality (VR) environments, excessive information density can lead to cognitive overload, while overly simplistic experiences may result in user disengagement. Therefore, effective cognitive load design is crucial in VR, however, cross-subject cognitive load recognition using multimodal physiological signals remains a significant challenge. In this paper, we proposed a new method for accurate cognitive load recognition in a VR driving environment using electroencephalogram (EEG) and eye-tracking data. Fifteen participants performed tasks designed to induce three levels of cognitive load while their EEG signals and eye-tracking data were recorded. We utilized a two-stage deep learning framework comprising pretraining and fine-tuning. During pretraining, a crossattention mechanism was employed to effectively fuse multimodal features, leveraging complementary information between EEG and eye-tracking data. Additionally, a domain-adversarial adaptation network and a shared encoder-decoder structure were introduced to extract subject-independent representations, enhancing the model's generalization to unseen subjects. In the fine-tuning stage, a classifier was added to refine cognitive load classification. Experimental results demonstrated that our method achieved the highest average accuracy on our dataset and a public dataset, outperforming existing approaches in cross-subject cognitive load recognition. These findings highlighted the potential of our method for cognitive load assessment in VR applications, providing new insights into cognitive load monitoring and applications according to the cognitive load.