Multimodal Cognitive Workload Recognition in
Human-Computer Interaction Using Biosignals and
Interaction Traces
Andino Maseleno1,*, Kharchenko Raisa2, Rahul Chauhan3
1Institut Bakti Nusantara, Lampung, Indonesia
2North-West Institute of Management, RANEPA, Russia
3Unitedworld Institution of Management, Karnavati University, Gandhinagar, India
Emails: andino.maseleno@ibnus.ac.id; kh9044947155r@gmail.com;
rahulchauhan@karnavatiuniversity.edu.in
Abstract
The process of recognizing cognitive workload requires reliable methods because researchers need to use both
physiological indicators and interaction traces while facing challenges of limited data and inconsistent feature sets. The
paper develops a multimodal fusion system which uses weight-based reliability assess-ment to identify three different
workload levels from Cognitive Lab data which is publicly accessible. The subset which focuses on workload includes
N-Back and mental subtraction tasks together with electroen-cephalography and functional near-infrared spectroscopy
and electrocardiography and electrodermal activity and respiration and accelerometry and gaze descriptors and
keyboard-mouse interaction indicators. The method conducts separate training for every modality through multidimensional
variable reduction which enables gradient-boosted learners to make predictions about branch reliability
based on their validation log-loss scores and combine posterior probabilities using normalized reliability weights. The
design preserves distinct modality structures while controlling unpredictable branch effects. The study tests different
ap-proaches by evaluating single-modality learners against three methods which include direct early fusion and uniform
late fusion and the proposed fusion rule. The proposed model achieves its best performance with 0.842 accuracy and
0.836 macro F1-score on the three-class workload task which includes the medium-load category that presents the
greatest challenge to differentiate.The research results from class-wise and sensitivity assessments showed that
interaction traces together with fNIRS features produced the smallest improvement to the system, and moderate
reliability temperatures showed the highest stability in fusion pro-file performance. The feature attribution demonstrates
specific emphasis on how cursor-velocity variability together with fNIRS oxygenation slope and EEG theta-band power
and fixation-duration statistics and pha-sic electrodermal activity function as primary discriminative signals. The
research findings demonstrate that multiple modal workload estimation needs to be improved through branch-specific
modeling which should use decision fusion based on reliability as its foundation model and work through adaptive
learning systems which have to handle rising cognitive requirements.
Keywords: Cognitive workload; Multimodal fusion; Biosignals; Human-computer interaction; Adaptive systems;
Explainable machine learning