Real-time Arabic sign language translator Using media pipe and LSTM

Authors

  • Sahar K. Hussin Communication and Computers Engineering Department Alshorouck Academy, Cairo, Egypt
  • Omar Mohamed Communication and Computers Engineering Department Alshorouck Academy, Cairo, Egypt
  • Mustafa Mohamed Communication and Computers Engineering Department Alshorouck Academy, Cairo, Egypt
  • Eslam Ahmed Department of Communication and Computer Engineering, Higher Institute of Engineering, El Shorouk Academy, Cairo 11837, Egypt.
  • Omar Mahmoud Computer Engineering Department, College of Computer Science and Engineering, Hail University, Hail, Saudi Arabia.

Keywords:

sign language translator, deep learning, BiLSTM media pipe

Abstract

People who are dumb and deaf have difficulty communicating daily. Artificial intelligence (AI) developments have allowed the breaking down of this communication barrier. As a result of this work, an Arabic sign language (ASLT) letter recognition system has been created. The ASLT recognition system employs BILSTM with a media pipe structure to interpret depth data and enhance the social interaction of hearing-impaired people. Depending on user input, the suggested approach would automatically detect and identify Arabic and hand-sign alphabet letters. The proposed model should have a 98% accuracy rate in identifying ASLT for letters, 96% for words, and 100% for digits from 0 to 9. We conducted comparative research to evaluate our method, and the results showed that it is more accurate at differentiating between static signs than earlier studies using the same dataset.

References

. M.A. Jalal, R.Chen, R.Moore, et al., “American sign language posture understanding with deep neural networks,” in Proc. of 21st International Conference on Information Fusion (FUSION), pp. 573-579, 10-13 Jul 2018. Article (CrossRef Link)

. S.P. Becky, “Sign Language Recognition and Translation: A Multidisciplined Approach From the Field of Artificial Intelligence,” Journal of Deaf Studies and Deaf Education Advance Access, Vol. 11, no.1, pp.94-101, 2006. Article (CrossRef Link) P. R. Graves and T. A. J.

. M. Mustafa, ''A study on Arabic sign language recognition for differently abled using advanced machine learning classifiers'', J. of Ambient Intelligence and Humanized Computing 12, 4101–4115, Mar. 2020.

. American Sign Language, National Institute on Deafness and Other Communication Disorders. http://www.nidcd.nih.gov/health/hearing/asl.asp

. Rahib H. Abiyev, “Facial Feature Extraction Techniques for Face Recognition,” Journal of Computer Science, Vol.10, no.12, pp.2360-2365, 2014. Article (CrossRef Link)

. C. Jennings, “Robust finger tracking with multiple cameras,” in Proc. of Int. Workshop on Recognition, Analysis, and tracking of Faces and Gestures in Real-Time Systems, 1999.Article (CrossRef Link)

. S. Malassiotis, N. Aifanti, and M. G. Strintzis, “A Gesture Recognition System Using 3D Data,” in Proc. of IEEE 1st International Symposium on 3D Data Processing Visualization and Transmission, June 2002. Article (CrossRef Link)

. B. S.Parton, “Sign Language Recognition and Translation: A Multidisciplined Approach From the Field of Artificial Intelligence,” Journal of Deaf Studies and Deaf Education, Vol.11, no.1, pp.94-101, 2006. Article (CrossRef Link)

. G. Fang, W. Gao, X. Chen, C. Wang, and J. Ma, “Signer independent continuous sign language recognition based on SRN/HMM,” in Proc. of International Gesture Workshop, Gesture and Sign Language in Human-Computer Interaction, pp. 76-85, 2001. Article (CrossRef Link)

. Tharwat, A.; Gaber, T.; Hassanien, A.E.; Shahin, M.K.; Refaat, B. Sift-based arabic sign language recognition system. In Afro-

. European Conference for Industrial Advancement, Proceedings of the First International Afro-European Conference for Industrial Advancement AECIA 2014, Addis Ababa, Ethiopia, 17–19 November 2015; Springer International Publishing: Cham, Switzerland, 2015; pp. 359–370.

. F.N.H. Al-Nuaimy, Proc. 2017 Int. Conf. Eng. Technol. ICET 2017 2018-Janua, 1 (2018).

. B.G. Lee and S.M. Lee, IEEE Sens. J. 18, 1224 (2018).

. H.S. Kala, S. Sushith Rai, S. Pal, K. Uzma Sulthana, and S. Chakma, Proc. - 2018 Int. Conf. Des. Innov. 3Cs Comput. Commun. Control. ICDI3C 2018 97 (2018).

. [Kamruzzaman, M.M. Arabic Sign Language Recognition and Generating Arabic Speech Using Convolutional Neural Network. Wirel. Commun. Mob. Comput. 2020, 2020, 3685614. [CrossRef]

. [Mustafa, M. A study on Arabic sign language recognition for differently abled using advanced machinelearning classifiers. J. Ambient. Intell. Human Comput. 2020, 12, 4101–4115. [CrossRef]

. G. Latif, N. Mohammad, R. AlKhalaf, R. AlKhalaf, J. Alghazo and M. Khan, ''An Automatic Arabic Sig Language Recognition System basedon Deep CNN: An Assistive System for the Deaf and Hard of Hearing'', International Journal of Computing and Digital Systems, Vol.9, No.4, pages 715-724, Jul. 2020.

. Batool Yahya AlKhuraym, Mohamed Maher Ben Ismail and Ouiem Bchir, “Arabic Sign Language Recognition using Lightweight CNN-based Architecture” International Journal of Advanced Computer Science and Applications(IJACSA), 13(4), 2022. http://dx.doi.org/10.14569/IJACSA.2022.0130438

. C. Wang, H. Mark Liao, Y. Wu, P. Chen, J. Hsieh and I. Yeh, ” CSPNet: A New Backbone that can Enhance Learning Capability of CNN” , 2020 IE E E /C V F Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020. Available: 10.1109/cvprw50498.2020.00203 .

. [19] K. Simonyan and A. Zisserman, ”Very Deep Convolutional Networks for Large-Scale Image Recognition” , 2014

. A. Bochkovskiy, C. Wang and H. Mark Liao, ”YOLOv4: Opti mal Speed and Accuracy of Object Detection” , 2020. Available: https://arxiv.org/abs/2004.10934

. Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C. L., & Grundmann, M. 2020. MediaPipe Hands: On-device Real-time Hand Tracking. arXiv preprint arXiv:2006.10214

. M. M. Kamruzzaman, "Arabic Sign Language Recognition and Generating Arabic Speech Using Convolutional Neural Network", Wireless Communications and Mobile Computing, vol. 2020, Article ID 3685614, 9 pages, 2020. https://doi.org/10.1155/2020/368561

. A. A. Alani and G. Cosma, “ArSL-CNN: a convolutional neural network for Arabic sign language gesture recognition,” Indonesian journal of electrical engineering and computer science, vol. 22, 2021.

. Ahmad M. J. AL Moustafa, Mohd Shafry Mohd Rahim, Belgacem Bouallegue, Mahmoud M. Khattab, Amr Mohmed Soliman, Gamal Tharwat, Abdelmoty M. Ahmed, "Integrated Mediapipe with a CNN Model for Arabic Sign Language Recognition", Journal of Electrical and Computer Engineering, vol. 2023, Article ID 8870750, 15 pages, 2023. https://doi.org/10.1155/2023/8870750

. https://developers.google.com/mediapipe/solutions/vision/hand_landmarker

Downloads

Published

2024-05-07

How to Cite

K. Hussin, . . S., Mohamed, O. ., Mohamed, M. ., Ahmed , E. ., & Mahmoud , O. . (2024). Real-time Arabic sign language translator Using media pipe and LSTM. PLOMS AI, 4(1). Retrieved from https://plomscience.com/journals/index.php/PLOMSAI/article/view/21

Issue

Vol. 4 No. 1 (2024): 2024 Issue

Section

Computer

License

Copyright (c) 2024 PLOMS AI

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

PLOMS  Journals Copyright Statement

PLOMS LLC. grants you a non-exclusive, royalty-free, revocable license to: 

  • Academic Journals licenses all works published under the Creative Commons Attribution 4.0 International License. This license grants anybody the right to reproduce, redistribute, remix, transmit, and modify the work, as long as the original work and source are properly cited.
  • PLOMS LLC. grants you no further rights in respect to this website or its content. 

Without the prior consent of PLOMS LLC, this website and its content (in any form or medium) may not be changed or converted in any manner. To avoid doubt, you must not modify, edit, alter, convert, publish, republish, distribute, redistribute, broadcast, rebroadcast, display, or play in public any of the content on this website (in any form or medium) without PLOMS LLC's prior written approval.

Permissions

Permission to use the copyright content on this website may be obtained by emailing to: 

 [email protected].

PLOMS LLC. takes copyright protection very seriously.  If PLOMS LLC. discovers that you have violated the license above by using its copyright materials, PLOMS LLC. may pursue legal action against you, demanding monetary penalties and an injunction to prevent you from using such materials. Additionally, you may be required to pay legal fees.

If you become aware of any unauthorized use of PLOMS LLC. copyright content that violates or may violate the license above, please contact :

[email protected].

Infringing content

If you become aware of any content on the website that you feel violates your or another person's copyright, please notify [email protected].