Interaction-Stability Gated Multimodal Learning for Cognitive

Friction Detection in Human-Computer Interaction

Andino Maseleno1,* Aa Hubur2

1 Institut Bakti Nusantara, Lampung, Indonesia

2 Universitas Trisakti, Jakarta, Indonesia

Received: October 12, 2024 Revised: December 07, 2024 Accepted: January 03, 2025 ⋆ Corresponding author

ABSTRACT

Cognitive human-computer interaction (HCI) requires systems that do not only estimate whether a user is under

high cognitive load, but also detect moments when interaction demands become unstable, disruptive, or cognitively

misaligned with the user’s current state. Existing cognitive-load models commonly treat workload estimation

as a static classification task, which limits their usefulness for adaptive interfaces. This paper introduces an

interaction-stability gated multimodal learning framework for detecting cognitive friction during HCI. The proposed

model combines subject-normalized physiological and gaze features with a temporal stability gate that adjusts the

contribution of electrocardiography (ECG), electrodermal activity (EDA), electroencephalography (EEG), and gaze

streams according to local signal reliability and cognitive-state fluctuation. A Cognitive Friction Index is further

proposed to identify transition periods where cognitive load rises sharply or remains unstable across modalities. The

study is designed for reproducibility using the public CLARE dataset, which contains multimodal physiological and

gaze recordings from participants performing Multi-Attribute Task Battery II (MATB-II) computer-based workload

tasks. Baseline evidence from the CLARE benchmark shows that multimodal learning improves cognitive-load

estimation, but leave-one-subject-out performance remains lower than random 10-fold validation, indicating a

strong personalization challenge. The proposed framework addresses this gap by modeling temporal instability and

subject-level calibration rather than only point-level workload labels. The paper contributes a reproducible Cognitive

HCI model, a friction-oriented interpretation layer, and an adaptive-interface decision mechanism.

Keywords: Cognitive HCI Cognitive load Cognitive friction Multimodal learning EEG Gaze tracking

Adaptive interfaces

1. INTRODUCTION

Human-computer interaction has moved beyond static usability

evaluation toward systems that sense, interpret, and

respond to the cognitive state of users. In safety-critical, educational,

medical, and productivity environments, a user’s interaction

with an interface may deteriorate when the interface

imposes excessive mental effort, interrupts task continuity, or

requires attention switching beyond the user’s current cognitive

capacity. Classical cognitive-load theory explains that

working memory has limited processing capacity, and excessive

task demands may reduce learning, decision quality, and

task performance [1, 2, 5]. Within HCI, this makes cognitiveload

estimation important not only as a measurement problem

but also as a mechanism for interface adaptation.

Most existing computational approaches formulate cognitiveload

estimation as binary or multiclass classification, such as

low versus high workload. Although this formulation is use-