2690-6791ISSN (Online)
2769-786XISSN (Print)
Volume 16 , Issue 2 , PP: 286-307, 2025 | Cite this article as | XML | Html | PDF | Full Length Article
Asmaa Badran 1 , Ahmad Salah 2 , A. A. Soliman 3 , Dina A. Elmanakhly 4 , Ahmed Fathalla 5 *
Doi: https://doi.org/10.54216/JISIoT.160221
Spatio-temporal human activity recognition (HAR) is an emerging field that uses spatial and temporal data to identify and classify human activities accurately. It has been effectively applied in areas like healthcare for monitoring daily activities, detecting anomalies, and aiding rehabilitation with real time feedback. However, there is a gap in research specifically focusing on integrating spatio temporal data with advanced machine and deep learning techniques for HAR based on sensor data. Existing reviews do not comprehensively cover spatio-temporal HAR based on sensor data, resulting in a lack of summaries on recent models, datasets, sensor technologies, applications, and machine/deep learning techniques used in this field. This systematic review provides a comprehendsive overview of spatio-temporal HAR based on sensor data, tracing its development from the origin of sensor-based spatio-temporal HAR field to the present. It highlights the main challenges in spatio- temporal HAR. The review also examines model trends over the years, including the distribution of models used in HAR and the identification of those frequently combined to form hybrid models. Additionally, it analyzes accuracy trends of the commonly used datasets and identifies the datasets that are widely used in spatio-temporal HAR research. Furthermore, various application domains and sensor technologies used in spatio-temporal HAR are identified.
Deep learning , Human activity recognition , Machine learning , Systematic review , Spatio-temporal , Wearable sensors
[1] C.-M. Forke and M. Tropmann-Frick, “Feature engineering techniques and spatio-temporal data processing,” Datenbank-Spektrum, vol. 21, pp. 237–244, 2021.
[2] S. Diwakar, D. Dwivedi, S. P. Singh, and M. Sharma, “Self-attention-based human activity detection using wearable sensors,” in International Conference on Signals, Machines, and Automation. Springer, 2022, pp. 629–636
[3] S. M. Wagh, “Precise human activity recognition for the openpack challenge 2022,” in 2023 IEEE In- ternational Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). IEEE, 2023, pp. 259–261.
[4] O. Nafea, W. Abdul, G. Muhammad, and M. Alsulaiman, “Sensor-based human activity recognition with spatio-temporal deep learning,” Sensors, vol. 21, no. 6, p. 2141, 2021.
[5] E. Ferrara, “Large language models for wearable sensor-based human activity recognition, health mon- itoring, and behavioral modeling: A survey of early trends, datasets, and challenges,” Sensors, vol. 24, no. 15, p. 5045, 2024.
[6] F. Al Machot and H. C. Mayr, “Improving human activity recognition by smart windowing and spatio- temporal feature analysis,” in Proceedings of the 9th ACM International Conference on PErvasive Tech- nologies Related to Assistive Environments, 2016, pp. 1–7.
[7] R. Chen, T. Chu, K. Liu, J. Liu, and Y. Chen, “Inferring human activity in mobile devices by computing multiple contexts,” Sensors, vol. 15, no. 9, pp. 21 219–21 238, 2015.
[8] A. Sarkar, S. S. Hossain, and R. Sarkar, “Human activity recognition from sensor data using spatial attention-aided cnn with genetic algorithm,” Neural Computing and Applications, vol. 35, no. 7, pp. 5165–5191, 2023.
[9] D. Khan, A. Alshahrani, A. Almjally, N. Al Mudawi, A. Algarni, K. Al Nowaiser, and A. Jalal, “Ad- vanced iot-based human activity recognition and localization using deep polynomial neural network,” IEEE Access, 2024.
[10] Ł. Czekaj, M. Kowalewski, J. Domaszewicz, R. Kitłowski, M. Szwoch, and W. Duch, “Real-time sensor- based human activity recognition for efitness and ehealth platforms,” Sensors, vol. 24, no. 12, p. 3891, 2024.
[11] H. Wu, Z. Zhang, X. Li, K. Shang, Y. Han, Z. Geng, and T. Pan, “A novel pedal musculoskeletal response based on differential spatio-temporal lstm for human activity recognition,” Knowledge-Based Systems, vol. 261, p. 110187, 2023.
[12] A. Bolatov, A. Yessenbayeva, and A. Yazici, “Glula: Linear attention-based model for efficient human activity recognition from wearable sensors,” Wearable Technologies, vol. 5, p. e10, 2024.
[13] A. Hamdi, K. Shaban, A. Erradi, A. Mohamed, S. K. Rumi, and F. D. Salim, “Spatiotemporal data mining: a survey on challenges and open problems,” Artificial Intelligence Review, pp. 1–48, 2022.
[14] Y. Isoda, S. Kurakake, and K. Imai, “Ubiquitous sensor-based human behaviour recognition using the spatio-temporal representation of user states,” International Journal of Wireless and Mobile Computing, vol. 3, no. 1-2, pp. 46–55, 2008.
[15] K. Chen, D. Zhang, L. Yao, B. Guo, Z. Yu, and Y. Liu, “Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities,” ACM Computing Surveys (CSUR), vol. 54, no. 4, pp. 1–40, 2021.
[16] V. Bijalwan, V. B. Semwal, and V. Gupta, “Wearable sensor-based pattern mining for human activity recognition: Deep learning approach,” Industrial Robot: the international journal of robotics research and application, vol. 49, no. 1, pp. 21–33, 2022.
[17] M. Straczkiewicz, P. James, and J.-P. Onnela, “A systematic review of smartphone-based human activity recognition methods for health research,” NPJ Digital Medicine, vol. 4, no. 1, p. 148, 2021.
[18] E. De-La-Hoz-Franco, P. Ariza-Colpas, J. M. Quero, and M. Espinilla, “Sensor-based datasets for human activity recognition – a systematic review of literature,” IEEE Access, vol. 6, pp. 59 192–59 210, 2018.
[19] J. Qi, P. Yang, A. Waraich, Z. Deng, Y. Zhao, and Y. Yang, “Examining sensor-based physical activity recognition and monitoring for healthcare using internet of things: A systematic review,” Journal of Biomedical Informatics, vol. 87, pp. 138–153, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S153204641830176X 305
[20] M. M. Baig, S. Afifi, H. GholamHosseini, and F. Mirza, “A systematic review of wearable sensors and iot-based monitoring applications for older adults–a focus on ageing population and independent living,” Journal of medical systems, vol. 43, pp. 1–11, 2019.
[21] A. Kristoffersson and M. Lind´en, “A systematic review of wearable sensors for monitoring physical activity,” Sensors, vol. 22, no. 2, p. 573, 2022.
[22] B. Kitchenham, S. Charters et al., “Guidelines for performing systematic literature reviews in software engineering version 2.3,” Engineering, vol. 45, no. 4ve, p. 1051, 2007.
[23] M. M. Islam and T. Iqbal, “Hamlet: A hierarchical multimodal attention-based human activity recogni- tion algorithm,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 10 285–10 292.
[24] O. Konak, P. Wegner, and B. Arnrich, “Imu-based movement trajectory heatmaps for human activity recognition,” Sensors, vol. 20, no. 24, p. 7179, 2020.
[25] X. Sun, H. Xu, Z. Dong, L. Shi, Q. Liu, J. Li, T. Li, S. Fan, and Y. Wang, “Capsganet: deep neural network based on capsule and gru for human activity recognition. ieee syst. j.(2022).”
[26] P. Khan, Y. Kumar, and S. Kumar, “Capslstm-based human activity recognition for smart healthcare with scarce labeled data,” IEEE Transactions on Computational Social Systems, vol. 11, no. 1, pp. 707–716, 2022.
[27] M. Sethi, M. Yadav, M. Singh, and P. G. Shambharkar, “Attnhar: Human activity recognition using data collected from wearable sensors,” in 2023 6th International Conference on Information Systems and Computer Networks (ISCON). IEEE, 2023, pp. 1–6.
[28] H. Najeh, C. Lohr, and B. Leduc, “Convolutional neural network bootstrapped by dynamic segmentation and stigmergy-based encoding for real-time human activity recognition in smart homes,” Sensors, vol. 23, no. 4, p. 1969, 2023.
[29] S. Arokiaraj and N. Viswanathan, “Ecg-nets–a novel integration of capsule networks and extreme gated recurrent neural network for iot based human activity recognition,” Journal of Intelligent & Fuzzy Sys- tems, vol. 44, no. 5, pp. 8219–8229, 2023.
[30] Y. Li, R. Yin, Y. Kim, and P. Panda, “Efficient human activity recognition with spatio-temporal spiking neural networks,” Frontiers in Neuroscience, vol. 17, p. 1233037, 2023.
[31] K. Monica and R. Parvathi, “Efficient gait analysis using deep learning techniques.” Computers, Materi- als & Continua, vol. 74, no. 3, 2023.
[32] W. Xie, Q. Huang, C. Li, Y. Wang, and Y. Liu, “Hierarchical multi-scale adaptive conv-lstm network for human action recognition based on wearable sensors,” in Proceedings of the 5th ACM International Conference on Multimedia in Asia, 2023, pp. 1–8.
[33] Q. Huang, W. Xie, C. Li, Y. Wang, and Y. Liu, “Human action recognition based on hierarchical multi- scale adaptive conv-long short-term memory network,” Applied Sciences, vol. 13, no. 19, p. 10560, 2023.
[34] S. Mekruksavanich and A. Jitpattanakul, “Hybrid convolution neural network with channel attention mechanism for sensor-based human activity recognition,” Scientific Reports, vol. 13, no. 1, p. 12067, 2023.
[35] H. Zhang and L. Xu, “Multi-stmt: multi-level network for human activity recognition based on wearable sensors,” IEEE Transactions on Instrumentation and Measurement, 2024.
[36] H. Zou, Z. Chen, C. Zhang, A. Yuan, B. Wang, L. Wang, J. Li, and Y. Pan, “Stfnet: Enhanced and lightweight spatiotemporal fusion network for wearable human activity recognition,” IEEE Sensors Jour- nal, 2024.
[37] F. Zeng, M. Guo, L. Tan, F. Guo, and X. Liu, “Wearable sensor-based residual multifeature fusion shrink- age networks for human activity recognition,” Sensors, vol. 24, no. 3, p. 758, 2024. 306.
[38] S. Mahmud, M. Tanjid Hasan Tonmoy, K. Kumar Bhaumik, A. Mahbubur Rahman, M. Ashraful Amin, M. Shoyaib, M. Asif Hossain Khan, and A. Ahsan Ali, “Human activity recognition from wearable sensor data using self-attention,” in ECAI 2020. IOS Press, 2020, pp. 1332–1339.
[39] J. Wang, Y. Chen, and Y. Gu, “A wearable-har oriented sensory data generation method based on spatio- temporal reinforced conditional gans,” Neurocomputing, vol. 493, pp. 548–567, 2022.
[40] S. Deng, Z. Guo, D. Teng, B. Lin, D. Chen, T. Jia, and H. Wang, “Self-relation attention networks for weakly supervised few-shot activity recognition,” Knowledge-Based Systems, vol. 276, p. 110720, 2023.
[41] D. Liciotti, M. Bernardini, L. Romeo, and E. Frontoni, “A sequential deep learning application for recog- nising human activities in smart homes,” Neurocomputing, vol. 396, pp. 501–513, 2020.
[42] N. T. H. Thu and D. S. Han, “Hihar: A hierarchical hybrid deep learning architecture for wearable sensor-based human activity recognition,” IEEE Access, vol. 9, pp. 145 271–145 281, 2021.
[43] H. Najeh, C. Lohr, and B. Leduc, “Dynamic segmentation of sensor events for real-time human activity recognition in a smart home context,” Sensors, vol. 22, no. 14, p. 5458, 2022.
[44] A. E. Nieto-Vallejo, C. A. Parra-Rodriguez, and O. Ramirez-Perez, “Classification of activities of daily living for older adults using machine learning and fixed time windowing technique,” IEEE Sensors Jour- nal, 2023.
[45] S. Mekruksavanich, N. Hnoohom, and A. Jitpattanakul, “A hybrid deep residual network for efficient transitional activity recognition based on wearable sensors,” Applied Sciences, vol. 12, no. 10, p. 4988, 2022.
[46] Z. Zhang, W. Wang, A. An, Y. Qin, and F. Yang, “A human activity recognition method using wearable sensors based on convtransformer model,” Evolving Systems, vol. 14, no. 6, pp. 939–955, 2023.
[47] J. Wu and Q. Liu, “A novel spatio-temporal network of multi-channel cnn and gcn for human activity recognition based on ban,” Neural Processing Letters, vol. 55, no. 8, pp. 11 489–11 507, 2023.
[48] L. Bai, L. Yao, X. Wang, S. S. Kanhere, B. Guo, and Z. Yu, “Adversarial multi-view networks for activity recognition,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 4, no. 2, pp. 1–22, 2020.
[49] J. Ye, H. Hu, G.-J. Qi, and K. A. Hua, “A temporal order modeling approach to human action recogni- tion from multimodal sensor data,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), vol. 13, no. 2, pp. 1–22, 2017.
[50] A. Sarkar, S. S. Hossain, and R. Sarkar, “Correction to: Human activity recognition from sensor data using spatial attention-aided cnn with genetic algorithm,” Neural Computing and Applications, vol. 35, no. 16, pp. 12 239–12 239, 2023.
[51] S. Reshmi and E. Ramanujam, “An ensemble maximal feature subset selection for smartphone based human activity recognition,” Journal of Network and Computer Applications, vol. 226, p. 103875, 2024.
[52] Y.-H. Tseng and C.-Y. Wen, “Hybrid learning models for imu-based har with feature analysis and data correction,” Sensors, vol. 23, no. 18, 2023. [Online]. Available: https://www.mdpi.com/1424-8220/23/ 18/7802
