Full Length Article
DOI: https://doi.org/10.54216/JCHCI.100203
Multimodal Cognitive Workload Recognition in Human-Computer Interaction Using Biosignals and Interaction Traces
The process of recognizing cognitive workload requires reliable methods because researchers need to use both physiological indicators and interaction traces while facing challenges of limited data and inconsistent feature sets. The paper develops a multimodal fusion system which uses weight-based reliability assessment to identify three different workload levels from Cognitive Lab data which is publicly accessible. The subset which focuses on workload includes N-Back and mental subtraction tasks together with electroen-cephalography and functional near-infrared spectroscopy and electrocardiography and electrodermal activity and respiration and accelerometry and gaze descriptors and keyboard-mouse interaction indicators. The method conducts separate training for every modality through multidimensional variable reduction which enables gradient-boosted learners to make predictions about branch reliability based on their validation log-loss scores and combine posterior probabilities using normalized reliability weights. The design preserves distinct modality structures while controlling unpredictable branch effects. The study tests different approaches by evaluating single-modality learners against three methods which include direct early fusion and uniform late fusion and the proposed fusion rule. The proposed model achieves its best performance with 0.842 accuracy and 0.836 macro F1-score on the three-class workload task which includes the medium-load category that presents the greatest challenge to differentiate. The research results from class-wise and sensitivity assessments showed that interaction traces together with fNIRS features produced the smallest improvement to the system, and moderate reliability temperatures showed the highest stability in fusion pro-file performance. The feature attribution demonstrates specific emphasis on how cursor-velocity variability together with fNIRS oxygenation slope and EEG theta-band power and fixation-duration statistics and phasic electrodermal activity function as primary discriminative signals. The research findings demonstrate that multiple modal workload estimation needs to be improved through branch-specific modeling which should use decision fusion based on reliability as its foundation model and work through adaptive learning systems which have to handle rising cognitive requirements.
Andino Maseleno,
Kharchenko Raisa,
Rahul Chauhan
visibility
278
download
337