Volume 10 • Issue 2 • PP: 23-35 • 2025
Multimodal Cognitive Workload Recognition in Human-Computer Interaction Using Biosignals and Interaction Traces
View PDF Abstract
The process of recognizing cognitive workload requires reliable methods because researchers need to use both physiological indicators and interaction traces while facing challenges of limited data and inconsistent feature sets. The paper develops a multimodal fusion system which uses weight-based reliability assessment to identify three different workload levels from Cognitive Lab data which is publicly accessible. The subset which focuses on workload includes N-Back and mental subtraction tasks together with electroen-cephalography and functional near-infrared spectroscopy and electrocardiography and electrodermal activity and respiration and accelerometry and gaze descriptors and keyboard-mouse interaction indicators. The method conducts separate training for every modality through multidimensional variable reduction which enables gradient-boosted learners to make predictions about branch reliability based on their validation log-loss scores and combine posterior probabilities using normalized reliability weights. The design preserves distinct modality structures while controlling unpredictable branch effects. The study tests different approaches by evaluating single-modality learners against three methods which include direct early fusion and uniform late fusion and the proposed fusion rule. The proposed model achieves its best performance with 0.842 accuracy and 0.836 macro F1-score on the three-class workload task which includes the medium-load category that presents the greatest challenge to differentiate. The research results from class-wise and sensitivity assessments showed that interaction traces together with fNIRS features produced the smallest improvement to the system, and moderate reliability temperatures showed the highest stability in fusion pro-file performance. The feature attribution demonstrates specific emphasis on how cursor-velocity variability together with fNIRS oxygenation slope and EEG theta-band power and fixation-duration statistics and phasic electrodermal activity function as primary discriminative signals. The research findings demonstrate that multiple modal workload estimation needs to be improved through branch-specific modeling which should use decision fusion based on reliability as its foundation model and work through adaptive learning systems which have to handle rising cognitive requirements.
Keywords
References
Ahlstrom, U., & Friedman-Berg, F. J. (2006). Using eye movement activity as a correlate of cognitive workload. International Journal of Industrial Ergonomics, 36(7), 623–636. https://doi.org/10.1016/j. ergon.2006.04.002
Aksu, S， . H., C， akıt, E., & Da˘gdeviren, M. (2024). Mental workload assessment using machine learning techniques based on EEG and eye tracking data. Applied Sciences, 14(6), 2282. https://doi.org/10.3390/ app14062282
Belkhiria, C., & Peysakhovich, V. (2021). Eog metrics for cognitive workload detection. Procedia Computer Science, 192, 3797–3806. https://doi.org/10.1016/j.procs.2021.08.193
Bixler, R., & D’Mello, S. (2014). Toward fully automated person-independent detection of mind wandering. In V. Dimitrova, T. Kuflik, D. Chin, F. Ricci, P. Dolog, & G.-J. Houben (Eds.), User modeling, adaptation, and personalization (pp. 37–48, Vol. 8538). Springer. https://doi.org/10.1007/978-3-319- 08786-3 4
Chen, W., Sawaragi, T., & Hiraoka, T. (2022). Comparing eye-tracking metrics of mental workload caused by ndrts in semi-autonomous driving. Transportation Research Part F: Traffic Psychology and Behaviour, 89, 109–128. https://doi.org/10.1016/j.trf.2022.05.004
Duchowski, A. T. (2002). A breadth-first survey of eye-tracking applications. Behavior Research Methods, Instruments, and Computers, 34(4), 455–470. https://doi.org/10.3758/BF03195475
Goldberg, J. H., & Kotval, X. P. (1999). Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics, 24(6), 631–645. https://doi.org/10.1016/ S0169-8141(98)00068-7
Haapalainen, E., Kim, S., Forlizzi, J. F., & Dey, A. K. (2010). Psycho-physiological measures for assessing cognitive load. Proceedings of the 12th ACM International Conference on Ubiquitous Computing, 301–310. https://doi.org/10.1145/1864349.1864395 Hart, S. G., & Staveland, L. E. (1988). Development of nasa-tlx (task load index): Results of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 139–183, Vol. 52). North-Holland. https://doi.org/10.1016/S0166-4115(08)62386-9
Iqbal, S. T., Zheng, X. S., & Bailey, B. P. (2004). Task-evoked pupillary response to mental workload in human-computer interaction. CHI ’04 Extended Abstracts on Human Factors in Computing Systems, 1477–1480. https://doi.org/10.1145/985921.986094
Jo, W., Wang, R., Cha, G.-E., Sun, S., Senthilkumaran, R. K., Foti, D., & Min, B.-C. (2025). Mocas: A multimodal dataset for objective cognitive workload assessment on simultaneous tasks. IEEE Transactions on Affective Computing, 16(1), 116–132. https://doi.org/10.1109/TAFFC.2024.3414330 Just, M. A., & Carpenter, P. A. (1976). Eye fixations and cognitive processes. Cognitive Psychology, 8(4), 441–480. https://doi.org/10.1016/0010-0285(76)90015-3
Klingner, J., Kumar, R., & Hanrahan, P. (2008). Measuring the task-evoked pupillary response with a remote eye tracker. Proceedings of the 2008 Symposium on Eye Tracking Research & Applications, 69–72. https://doi.org/10.1145/1344471.1344489
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Wozniak, P. W. (2023). A survey on measuring cognitive workload in human-computer interaction. ACM Computing Surveys, 55(13s), 1–39. https://doi.org/10.1145/3582272
Ktistakis, E., Skaramagkas, V., Manousos, D., Tachos, N. S., Tripoliti, E., Fotiadis, D. I., & Tsiknakis, M. (2022). COLET: A dataset for cognitive workload estimation based on eye-tracking. Computer Methods and Programs in Biomedicine, 224, 106989. https://doi.org/10.1016/j.cmpb.2022.106989
Mark, J. A., Curtin, A., Kraft, A. E., Ziegler, M. D., & Ayaz, H. (2024). Mental workload assessment by monitoring brain, heart, and eye with six biomedical modalities during six cognitive tasks. Frontiers in Neuroergonomics, 5, 1345507. https://doi.org/10.3389/fnrgo.2024.1345507
Marquart, G., Cabrall, C., & deWinter, J. (2015). Review of eye-related measures of drivers’ mental workload. Procedia Manufacturing, 3, 2854–2861. https://doi.org/10.1016/j.promfg.2015.07.783
Palinko, O., Kun, A. L., Shyrokov, A., & Heeman, P. (2010). Estimating cognitive load using remote eye tracking in a driving simulator. Proceedings of the 2010 Symposium on Eye-Tracking Research and Applications, 141–144. https://doi.org/10.1145/1743666.1743701
Poole, A., & Ball, L. J. (2006). Eye tracking in HCI and usability research. In C. Ghaoui (Ed.), Encyclopaedia of human-computer interaction (pp. 211–219). Idea Group Inc. https://doi.org/10.4018/978-1-59140- 562-7.ch034
Sevcenko, N., Appel, T., Ninaus, M., Moeller, K., & Gerjets, P. (2023). Theory-based approach for assessing cognitive load during time-critical resource-managing human-computer interactions: An eye-tracking study. Journal on Multimodal User Interfaces, 17(1), 1–19. https://doi.org/10.1007/s12193-022- 00398-y
Silveira, I., Varandas, R., & Gamboa, H. (2025). Cognitive lab: A dataset of biosignals and hci features for cognitive process investigation. Computer Methods and Programs in Biomedicine, 269, 108863. https: //doi.org/10.1016/j.cmpb.2025.108863
Cite This Article
Choose your preferred format