Real Time Sign Recognition using YOLOv8 Object Detection Algorithm for Malayalam Sign Language

Esther Daniel¹, V. Kathiresan², Priyadarshini .C¹, Golden Nancy .R ^3,*, P. Sindhu⁴

¹Department of Computer Science and Engineering, Karunya Institute of Technology and Sciences, Coimbatore, India

²GITAM University (Deemed to be University), Karnataka, India

³Department of Artificial Intelligence and Machine Learning, Karunya Institute of Technology and Sciences, Coimbatore, India

⁴Department of Computer Science and Engineering, Rajiv Gandhi College of Engineering, Anna University, Chennai, 602105 , India, India

Emails: estherdaniell@gmail.com; xyzkathir@gmail.com; priyadharsini@karunya.edu; goldennancy@karunya.edu; sindhupandi26@gmail.com

Abstract

Sign language recognition is important for enhancing message and user-friendliness for the community of deaf and hearing-impaired people. This paper proposes a Malayalam Sign Language (MSL) method using sign language that emerged from the state of Kerala. The main factor contributing to this emergence of such regional sign language is the absence of a standardized and consistent approach to the use of Indian Sign Language (ISL) in various states. This is due to the variations in signs, grammar, and syntax used in different regions. The system uses the You Only Look Once v8 (YOLOv8) algorithm-based object detection method which is based on Convolution Neural Network (CNN), a widely accepted deep learning neural network design employed mainly in computer vision. As the dataset for MSL is not publicly available, we used an MSL video from YouTube provided by the National Institute of Speech and Hearing for training a custom model. We pre-processed the video to extract the frames and annotate them with sign labels. Then, we trained the YOLOv8 algorithm on the annotated frames to detect the hand region and recognize signs in real time. The proposed approach achieved an accuracy of 97.21% calculated from the mean Average Precision value on the MSL dataset. The result achieved outperformed other existing approaches even while using less dataset count compared to others.

Keywords: Malayalam sign language; Yolo, Computer vision; Deep learning; Machine learning; Convolution neural network