An Interaction-Centric Wireless Multimodal Fusion Model for

Cognitive State Recognition in Computer Interfaces

Khaled Sh. Gaber1,* Mahmoud Elshabrawy Mohamed1

1 Computer Science and Intelligent Systems Research Center, Blacksburg 24060, Virginia, USA

Received: October 07, 2025 Revised: November 16, 2025 Accepted: December 26, 2025 ⋆ Corresponding author

ABSTRACT

Wireless human-computer interaction increasingly depends on distributed sensing, yet adaptive computer interfaces

are still commonly modelled from isolated evidence streams. This paper presents an interaction-centric wireless

multimodal fusion model for recognizing cognitive state during computer-based task execution. The model integrates

wearable physiology, ocular behaviour, compact neurophysiological summaries, and direct interaction evidence

obtained from the task interface, then adjusts each sensing channel through a reliability term that reflects wireless

degradation. The experimental workflow follows a public stress-resilience human-computer interaction protocol

involving synchronized task phases and computer interaction logs. The analysis shows that interaction variables

such as task error, response latency, and click activity are among the strongest indicators of cognitive state and

complement physiological information in a meaningful way. The results support the design of adaptive computer

interfaces that respond not only to what the user is doing on the screen, but also to how reliably the supporting

wireless sensing infrastructure is functioning.

Keywords: Human-computer interaction Wireless interaction Cognitive state recognition Multimodal fusion Adaptive

interfaces

1. INTRODUCTION

Interaction-centric human-computer interaction research increasingly

treats the computer interface as a dynamic cognitive

environment rather than a static presentation layer. In

modern computer work, the user continuously alternates between

pointing, clicking, tracking, reading, monitoring alerts,

and recovering from interruptions. These micro-interactions

reflect the user’s cognitive state and also shape it. For this reason,

workload-aware interface design has become an important

direction in cognitive HCI, especially in settings where

the interface must remain usable under stress, divided attention,

or prolonged task engagement [1].

The shift toward wireless sensing has expanded the available

evidence for computer interaction analysis. Smartwatches,

mobile eye trackers, unobtrusive biosensors, and wireless neurophysiological

devices make it possible to observe workload

without interrupting the task [2–6]. However, a practical challenge

remains: when these sensing channels are integrated

with computer interaction logs, they do not all arrive with the

same reliability. Packet loss, synchronization delay, and temporary

sensor dropout can distort inference precisely when the

interface is expected to adapt. As a result, interaction-aware

modelling should consider both the content of the observed

signal and the reliability with which that signal reaches the

system.

A second challenge concerns representation. Many workload

studies use physiological or ocular measures but treat the

computer interface itself as secondary. In computer interaction,

this is limiting. A rise in pupil diameter is informative