Volume 9 • Issue 1 • PP: 20–25 • 2025
Interaction-Stability Gated Multimodal Learning for Cognitive Friction Detection in Human-Computer Interaction
View PDF Abstract
Cognitive human-computer interaction (HCI) requires systems that do not only estimate whether a user is under high cognitive load, but also detect moments when interaction demands become unstable, disruptive, or cognitively misaligned with the user’s current state. Existing cognitive-load models commonly treat workload estimation as a static classification task, which limits their usefulness for adaptive interfaces. This paper introduces an interaction-stability gated multimodal learning framework for detecting cognitive friction during HCI. The proposed model combines subject-normalized physiological and gaze features with a temporal stability gate that adjusts the contribution of electrocardiography (ECG), electrodermal activity (EDA), electroencephalography (EEG), and gaze streams according to local signal reliability and cognitive-state fluctuation. A Cognitive Friction Index is further proposed to identify transition periods where cognitive load rises sharply or remains unstable across modalities. The study is designed for reproducibility using the public CLARE dataset, which contains multimodal physiological and gaze recordings from participants performing Multi-Attribute Task Battery II (MATB-II) computer-based workload tasks. Baseline evidence from the CLARE benchmark shows that multimodal learning improves cognitive-load estimation, but leave-one-subject-out performance remains lower than random 10-fold validation, indicating a strong personalization challenge. The proposed framework addresses this gap by modeling temporal instability and subject-level calibration rather than only point-level workload labels. The paper contributes a reproducible Cognitive HCI model, a friction oriented interpretation layer, and an adaptive-interface decision mechanism.
Keywords
References
[1] J. Sweller, “Cognitive load during problem solving: Effects on learning,” Cognitive Science, vol. 12, no. 2, pp. 257–285, 1988, doi: 10.1207/s15516709cog1202_4.
[2] F. G. W. C. Paas, “Training strategies for attaining transfer of problem-solving skill in statistics: A cognitiveload approach,” Journal of Educational Psychology, vol. 84, no. 4, pp. 429–434, 1992.
[3] S. G. Hart and L. E. Staveland, “Development of NASATLX (Task Load Index): Results of empirical and theoretical research,” in Advances in Psychology, vol. 52, Elsevier, 1988, pp. 139–183, doi: 10.1016/S0166- 4115(08)62386-9.
[4] S. K. Card, T. P. Moran, and A. Newell, The Psychology of Human-Computer Interaction. Hillsdale, NJ, USA: Lawrence Erlbaum Associates, 1983.
[5] C. D. Wickens, “Multiple resources and mental workload,” Human Factors, vol. 50, no. 3, pp. 449–455, 2008, doi: 10.1518/001872008X288394.
[6] L. Fridman, B. Reimer, B. Mehler, and W. T. Freeman, “Cognitive load estimation in the wild,” in Proc. 2018 CHI Conf. Human Factors in Computing Systems, 2018, Paper 652, pp. 1–9, doi: 10.1145/3173574.3174226.
[7] M. I. Ahmad, I. Keller, D. A. Robb, and K. S. Lohan, “A framework to estimate cognitive load using physiological data,” Personal and Ubiquitous Computing, vol. 27, pp. 2027–2041, 2023, doi: 10.1007/s00779-020-01455- 7.
[8] P. Antonenko, F. Paas, R. Grabner, and T. van Gog, “Using electroencephalography to measure cognitive load,” Educational Psychology Review, vol. 22, no. 4, pp. 425–438, 2010, doi: 10.1007/s10648-010-9130-y.
[9] K. Krejtz, A. T. Duchowski, A. Niedzielska, C. Biele, and I. Krejtz, “Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze,” PLOS ONE, vol. 13, no. 9, Art. no. e0203629, 2018, doi: 10.1371/journal.pone.0203629.
[10] L. Perkhofer and O. Lehner, “Using gaze behavior to measure cognitive load,” in Information Systems and Neuroscience, Lecture Notes in Information Systems and Organisation, vol. 29, Springer, 2019, pp. 73–83, doi: 10.1007/978-3-030-01087-4_9.
[11] P. Li, Y. Li, Y. Yao, C. Wu, B. Nie, and S. E. Li, “Sensitivity of electrodermal activity features for driver arousal measurement in cognitive load: The application in automated driving systems,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 9, pp. 14954– 14967, 2022, doi: 10.1109/TITS.2021.3135266.
[12] S. Koldijk, M. Sappelli, S. Verberne, M. A. Neerincx, and W. Kraaij, “The SWELL knowledge work dataset for stress and user modeling research,” in Proc. 16th Int. Conf. Multimodal Interaction, 2014, pp. 291–298, doi: 10.1145/2663204.2663257.
[13] P. Schmidt, A. Reiss, R. Duerichen, C. Marberger, and K. Van Laerhoven, “Introducing WESAD, a multimodal dataset for wearable stress and affect detection,” in Proc. 20th ACM Int. Conf. Multimodal Interaction, 2018, pp. 400–408, doi: 10.1145/3242969.3242985.
[14] W. L. Lim, O. Sourina, and L. P. Wang, “STEW: Simultaneous task EEG workload data set,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 26, no. 11, pp. 2106–2114, 2018, doi: 10.1109/TNSRE.2018.2872924.
[15] M. Gjoreski, T. Kolenik, T. Knez, M. Lustrek, M. Gams, H. Gjoreski, and V. Pejovic, “Datasets for cognitive load inference using wearable sensors and psychological traits,” Applied Sciences, vol. 10, no. 11, Art. no. 3843, 2020, doi: 10.3390/app10113843.
[16] W. Jo, R. Wang, S. Sun, R. K. Senthilkumaran, D. Foti, and B.-C. Min, “MOCAS: A multimodal dataset for objective cognitive workload assessment on simultaneous tasks,” arXiv preprint arXiv:2210.03065, 2022.
[17] P. Angkan, B. Behinaein, Z. Mahmud, A. Bhatti, D. Rodenburg, P. Hungler, and A. Etemad, “Multimodal brain-computer interface for in-vehicle driver cognitive load measurement: Dataset and baselines,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 6, pp. 5949–5964, 2024, doi: 10.1109/TITS.2023.3345846.
[18] A. Bhatti, P. Angkan, B. Behinaein, Z. Mahmud, D. Rodenburg, H. Braund, P. J. Mclellan, A. Ruberto, G. Harrison, D. Wilson, A. Szulewski, D. Howes, A. Etemad, and P. Hungler, “CLARE: Cognitive Load Assessment in REaltime with Multimodal Data,” arXiv preprint arXiv:2404.17098v1, 2024, doi: 10.48550/arXiv.2404.17098.
[19] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2016, pp. 785–794, doi: 10.1145/2939672.2939785.
[20] A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, 2017.
[21] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Advances in Neural Information Processing Systems, vol. 30, 2017.
[22] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, “EEGNet: A compact convolutional neural network for EEG-based braincomputer interfaces,” Journal of Neural Engineering, vol. 15, no. 5, Art. no. 056013, 2018, doi: 10.1088/1741- 2552/aace8c.
Cite This Article
Choose your preferred format