Volume 11 • Issue 1 • PP: 14–19 • 2026
An Interaction-Centric Wireless Multimodal Fusion Model for Cognitive State Recognition in Computer Interfaces
View PDF Abstract
Wireless human-computer interaction increasingly depends on distributed sensing, yet adaptive computer interfaces are still commonly modelled from isolated evidence streams. This paper presents an interaction-centric wireless multimodal fusion model for recognizing cognitive state during computer-based task execution. The model integrates wearable physiology, ocular behaviour, compact neurophysiological summaries, and direct interaction evidence obtained from the task interface, then adjusts each sensing channel through a reliability term that reflects wireless degradation. The experimental workflow follows a public stress-resilience human-computer interaction protocol involving synchronized task phases and computer interaction logs. The analysis shows that interaction variables such as task error, response latency, and click activity are among the strongest indicators of cognitive state and complement physiological information in a meaningful way. The results support the design of adaptive computer interfaces that respond not only to what the user is doing on the screen, but also to how reliably the supporting wireless sensing infrastructure is functioning.
Keywords
References
[1] T. Kosch, J. Karolus, J. Zagermann, H. Reiterer, A. Schmidt, and P. W. Wozniak, “A survey on measuring cognitive workload in humancomputer interaction,” ACM Computing Surveys, vol. 55, no. 13s, pp. 1–39, 2023.
[2] N. Sevcenko, T. Seufert, and M. Schweinberger, “A theory-based approach for assessing cognitive load during interaction with a serious game,” Cognitive Technology and Work, vol. 25, pp. 361–374, 2023.
[3] W. Jo, R. Wang, S. Sun, R. K. Senthilkumaran, D. Foti, and B.-C. Min, “MOCAS: A multimodal dataset for objective cognitive workload assessment on simultaneous tasks,” IEEE Transactions on Affective Computing, vol. 16, no. 1, pp. 302–316, 2025.
[4] I. Silveira, R. Varandas, and H. Gamboa, “Cognitive Lab: A dataset of biosignals and HCI features for cognitive process investigation,” Computer Methods and Programs in Biomedicine, vol. 269, article 108863, 2025.
[5] Y. Hou, Q. Xie, N. Zhang, and J. Lv, “Cognitive load classification of mixed reality human computer interaction tasks based on multimodal sensor signals,” Scientific Reports, vol. 15, article 13342, 2025.
[6] B. He, H. Zhang, T. Qin, B. Shi, Q. Wang, and W. Dong, “A simultaneous EEG and eye-tracking dataset for remote sensing object detection,” Scientific Data, vol. 12, article 651, 2025.
[7] R. Khan and M. Vernooij, “Assessing cognitive load using EEG and eye-tracking in 3-D learning environments: A systematic review,” Multimodal Technologies and Interaction, vol. 9, no. 9, article 99, 2025.
[8] X. Yu, Y. Zhang, and colleagues, “Human operators’ cognitive workload recognition with a dual attention-enabled network based on multisensory information fusion,” Expert Systems with Applications, 2025.
[9] K. Jin, Y. Liu, and colleagues, “Human-centric cognitive state recognition using multimodal sensing: A systematic review,” Sensors, vol. 25, 2025.
[10] F. Li, Z. Wang, and colleagues, “Multimodal physiological signals from wearable sensors for affective computing: A systematic review,” Biomedical Signal Processing and Control, 2025.
[11] S. Roy and J. Nuamah, “Neurophysiological Dataset of Stress Resilience During Human-Computer Interaction,” PhysioNet, version 1.0.0, 2026, doi: 10.13026/x3vc-p627.
[12] A. L. Goldberger et al., “PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals,” Circulation, vol. 101, no. 23, pp. e215–e220, 2000.
[13] R. L. Charles and J. Nixon, “Measuring mental workload using physiological measures: A systematic review,” Applied Ergonomics, vol. 74, pp. 221–232, 2019.
[14] J. Nuamah, “Effect of recurrent task-induced acute stress on task performance, vagally mediated heart rate variability, and task-evoked pupil response,” International Journal of Psychophysiology, article 112325, 2024.
[15] J. Uba and J. Nuamah, “Investigating human physiological responses to work-related stress,” Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 67, no. 1, pp. 2285–2290, 2023.
[16] A. Pascual-Leone and D. Bartres-Faz, “Human brain resilience: A call to action,” Annals of Neurology, vol. 90, no. 3, pp. 336–349, 2021.
[17] M. Gjoreski, T. Kolenik, T. Knez, M. Lustrek, M. Gams, H. Gjoreski, and V. Pejovic, “Datasets for cognitive load inference using wearable sensors and psychological traits,” Applied Sciences, vol. 10, no. 11, article 3843, 2020.
[18] F. Paas and J. Sweller, “Implications of cognitive load theory for multimedia learning,” The Cambridge Handbook of Multimedia Learning, 3rd ed., Cambridge University Press, 2022.
[19] J. Kim and E. Andre, “Emotion recognition based on physiological changes in music listening,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[20] I. H. Sarker, “Machine learning: Algorithms, real-world applications and research directions,” SN Computer Science, vol. 2, article 160, 2021.
Cite This Article
Choose your preferred format