Volume 10 • Issue 2 • PP: 15–20 • 2025
Multimodal Cognitive Workload Recognition in Human-Computer Interaction Using Biosignals and Interaction Traces
View PDF Abstract
The process of recognizing cognitive workload requires reliable methods because researchers need to use both physiological indicators and interaction traces while facing challenges of limited data and inconsistent feature sets. The paper develops a multimodal fusion system which uses weight-based reliability assessment to identify three different workload levels from Cognitive Lab data which is publicly accessible. The subset which focuses on workload includes N-Back and mental subtraction tasks together with electroen-cephalography and functional near-infrared spectroscopy and electrocardiography and electrodermal activity and respiration and accelerometry and gaze descriptors and keyboard-mouse interaction indicators. The method conducts separate training for every modality through multidimensional variable reduction which enables gradient-boosted learners to make predictions about branch reliability based on their validation log-loss scores and combine posterior probabilities using normalized reliability weights. The design preserves distinct modality structures while controlling unpredictable branch effects. The study tests different approaches by evaluating single-modality learners against three methods which include direct early fusion and uniform late fusion and the proposed fusion rule. The proposed model achieves its best performance with 0.842 accuracy and 0.836 macro F1-score on the three-class workload task which includes the medium-load category that presents the greatest challenge to differentiate. The research results from class-wise and sensitivity assessments showed that interaction traces together with fNIRS features produced the smallest improvement to the system, and moderate reliability temperatures showed the highest stability in fusion pro-file performance. The feature attribution demonstrates specific emphasis on how cursor-velocity variability together with fNIRS oxygenation slope and EEG theta-band power and fixation-duration statistics and phasic electrodermal activity function as primary discriminative signals. The research findings demonstrate that multiple modal workload estimation needs to be improved through branch-specific modeling which should use decision fusion based on reliability as its foundation model and work through adaptive learning systems which have to handle rising cognitive requirements.
Keywords
References
[1] M. A. Just and P. A. Carpenter, “Eye fixations and cognitive processes,” Cognitive Psychology, vol. 8, no. 4, pp. 441–480, 1976.
[2] J. H. Goldberg and X. P. Kotval, “Computer interface evaluation using eye movements: Methods and constructs,” International Journal of Industrial Ergonomics, vol. 24, no. 6, pp. 631–645, 1999.
[3] A. T. Duchowski, “A breadth-first survey of eyetracking applications,” Behavior Research Methods, Instruments, and Computers, vol. 34, no. 4, pp. 455–470, 2002.
[4] A. Poole and L. J. Ball, “Eye tracking in humancomputer interaction and usability research: Current status and future prospects,” in Encyclopedia of Human- Computer Interaction, C. Ghaoui, Ed. Hershey, PA: Idea Group Reference, 2005, pp. 211–219.
[5] S. P. Marshall, “The index of cognitive activity: Measuring cognitive workload,” in Proceedings of the 2002 IEEE 7th Conference on Human Factors and Power Plants. IEEE, 2002, pp. 7–5–7–9.
[6] O. Palinko, A. L. Kun, A. Shyrokov, and P. Heeman, “Estimating cognitive load using remote eye tracking in a driving simulator,” in Proceedings of the 2010 Symposium on Eye-Tracking Research and Applications. ACM, 2010, pp. 141–144.
[7] W. Chen, T. Sawaragi, and T. Hiraoka, “Comparing eyetracking metrics of mental workload caused by ndrts in semi-autonomous driving,” Transportation Research Part F: Traffic Psychology and Behaviour, vol. 89, pp. 109–128, 2022.
[8] E. Ktistakis et al., “Colet: A dataset for cognitive workload estimation based on eye-tracking,” Computer Methods and Programs in Biomedicine, vol. 224, p. 106989, 2022.
[9] S. G. Hart and L. E. Staveland, “Development of nasatlx (task load index): Results of empirical and theoretical research,” in Human Mental Workload, ser. Advances in Psychology, P. A. Hancock and N. Meshkati, Eds. Amsterdam: North-Holland, 1988, vol. 52, pp. 139–183.
[10] E. Haapalainen, S. Kim, J. F. Forlizzi, and A. K. Dey, “Psycho-physiological measures for assessing cognitive load,” in Proceedings of the 12th ACM International Conference on Ubiquitous Computing. ACM, 2010, pp. 301–310.
[11] R. Bixler and S. D’Mello, “Toward fully automated person-independent detection of mind wandering,” in User Modeling, Adaptation, and Personalization, ser. Lecture Notes in Computer Science, V. Dimitrova et al., Eds. Cham: Springer, 2014, vol. 8538, pp. 37–48.
[12] C. Ahlstr"om, K. Kircher, and A. Kircher, “A gazebased driver distraction warning system and its effect on visual behavior,” IEEE Transactions on Intelligent Transportation Systems, vol. 14, no. 2, pp. 965–973, 2013.
[13] G. Marquart, C. Cabrall, and J. de Winter, “Review of eye-related measures of drivers’ mental workload,” Procedia Manufacturing, vol. 3, pp. 2854–2861, 2015.
[14] T. Kosch et al., “A survey on measuring cognitive workload in human-computer interaction,” ACM Computing Surveys, vol. 55, no. 13s, pp. 1–39, 2023.
[15] F. Liu et al., “Small-world properties of eye-movement time series assisted in identifying children at high risk for dyslexia,” Biomedical Signal Processing and Control, vol. 93, p. 106148, 2024.
[16] S. T. Iqbal and B. P. Bailey, “Using eye gaze patterns to identify user tasks,” in The Grace Hopper Celebration of Women in Computing, 2004, pp. 5–10. [17] J. Klingner, R. Kumar, and P. Hanrahan, “Measuring the task evoked pupillary response with a remote eye tracker,” in Proceedings of the 2008 Symposium on Eye- Tracking Research and Applications. ACM, 2008, pp. 69–72.
[18] C. Belkhiria and V. Peysakhovich, “Eog metrics for cognitive workload detection,” Procedia Computer Science, vol. 192, pp. 1875–1884, 2021.
Cite This Article
Choose your preferred format