Abstract
Sign language for communication is efficacious for humans, and vital research is in progress in computer vision systems. The earliest work in Indian Sign Language (ISL) recognition considers the recognition of significant differentiable hand signs and therefore often selecting a few signs from the ISL for recognition. This paper deals with robust modeling of static signs in the context of sign language recognition using deep learning-based convolutional neural networks (CNN). In this research, total 35,000 sign images of 100 static signs are collected from different users. The efficiency of the proposed system is evaluated on approximately 50 CNN models. The results are also evaluated on the basis of different optimizers, and it has been observed that the proposed approach has achieved the highest training accuracy of 99.72% and 99.90% on colored and grayscale images, respectively. The performance of the proposed system has also been evaluated on the basis of precision, recall and F-score. The system also demonstrates its effectiveness over the earlier works in which only a few hand signs are considered for recognition.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Corballis MC (2003) From mouth to hand: gesture, speech and the evolution of right-handedness. Behav Brain Sci 26(2):199–208
Oyedotun OK, Khashman A (2017) Deep learning in vision-based static hand gesture recognition. Neural Comput Appl 28(12):3941–3951
Nagi J, Ducatelle F, Di Caro GA, Cireşan D, Meier U, Giusti A, Gambardella LM (2011) Max-pooling convolutional neural networks for vision-based hand gesture recognition. In: IEEE international conference on signal and image processing applications (ICSIPA), pp 342–347
Rioux-Maldague L, Giguere P (2014) Sign language fingerspelling classification from depth and color images using a deep belief network. In: IEEE Canadian conference on computer and robot vision (CRV), pp 92–97
Huang J, Zhou W, Li H, Li W (2015) Sign language recognition using 3D convolutional neural networks. In: IEEE international conference on multimedia and expo (ICME), pp 1–6
Huang J, Zhou W, Li H, Li W (2015) Sign language recognition using real-sense. In: IEEE China summit and international conference on signal and information processing (ChinaSIP), pp 166–170
Pigou L, Dieleman S, Kindermans PJ, Schrauwen B (2014) Sign language recognition using convolutional neural networks. In: Workshop at the European conference on computer vision. Springer, Cham, pp 572–578
Molchanov P, Gupta S, Kim K, Pulli K (2015) Multi-sensor system for driver’s hand-gesture recognition. In: 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), vol 1, pp 1–8
Tang A, Lu K, Wang Y, Huang J, Li H (2015) A real-time hand posture recognition system using deep neural networks. ACM Trans Intell Syst Technol (TIST) 6(2):21
Yang S, Zhu Q (2017) Video-based Chinese sign language recognition using convolutional neural network. In: IEEE 9th international conference on communication software and networks (ICCSN), pp 929–934
Tushar AK, Ashiquzzaman A, Islam MR (2017) Faster convergence and reduction of overfitting in numerical hand sign recognition using DCNN. In: Humanitarian technology conference (R10-HTC), IEEE Region 10, pp 638–641
Bheda V, Radpour D (2017) Using deep convolutional networks for gesture recognition in American sign language. arXiv preprint arXiv:1710.06836
Rao GA, Syamala K, Kishore PVV, Sastry ASCS (2018) Deep convolutional neural networks for sign language recognition. In: IEEE conference on signal processing and communication engineering systems (SPACES), pp 194–197
Koller O, Zargaran S, Ney H, Bowden R (2018) Deep sign: enabling robust statistical continuous sign language recognition via hybrid CNN-HMMs. Int J Comput Vis 126(12):1311–1325
Kumar EK, Kishore PVV, Kiran Kumar MT (2019) 3D sign language recognition with joint distance and angular coded color topographical descriptor on a 2—stream CNN. Neurocomput 372:40–54
Prabhu R (2018) Understanding of convolutional neural network (CNN) — deep learning. https://medium.com/@RaghavPrabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835f148. Accessed 4 Mar 2018
Rahaman MA, Jasim M, Ali MH, Hasanuzzaman M (2014) Real-time computer vision-based Bengali Sign Language recognition. In: 17th IEEE international conference on computer and information technology (ICCIT), pp 192–197
Uddin MA, Chowdhury SA (2016) Hand sign language recognition for Bangla alphabet using support vector machine. In: IEEE international conference on innovations in science, engineering and technology (ICISET), pp 1–4
Rao GA, Kishore PVV (2017) Selfie video based continuous Indian sign language recognition system. Ain Shams Eng J 9(4):1929–1939
Acknowledgements
This publication is an outcome of R&D work undertaken in the project under the Visvesvaraya PhD Scheme of Ministry of Electronics and Information Technology, Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan XP GPU used for this research.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Grayscale sign samples showing precision, recall and F1 score
S no. | Sign | Precision | Recall | F1-score | S no. | Sign | Precision | Recall | F1-score |
|---|---|---|---|---|---|---|---|---|---|
1 | A | 1.00 | 0.96 | 0.98 | 51 | Leprosy | 1.00 | 1.00 | 1.00 |
2 | Afraid | 0.97 | 0.97 | 0.97 | 52 | M | 0.97 | 0.79 | 0.87 |
3 | Add | 1.00 | 1.00 | 1.00 | 53 | Me | 1.00 | 1.00 | 1.00 |
4 | B | 1.00 | 1.00 | 1.00 | 54 | N | 1.00 | 1.00 | 1.00 |
5 | Bottle | 1.00 | 0.97 | 0.99 | 55 | Nose | 0.98 | 1.00 | 0.99 |
6 | Bud | 1.00 | 1.00 | 1.00 | 56 | Nine | 1.00 | 1.00 | 1.00 |
7 | Bent | 0.97 | 1.00 | 0.99 | 57 | Nurse | 0.99 | 0.98 | 0.98 |
8 | Between | 1.00 | 0.99 | 0.99 | 58 | O | 0.96 | 0.97 | 0.96 |
9 | Blind | 1.00 | 1.00 | 1.00 | 59 | Oath | 1.00 | 1.00 | 1.00 |
10 | Bowl | 0.97 | 1.00 | 0.98 | 60 | One | 1.00 | 0.99 | 0.99 |
11 | Brain | 0.99 | 0.99 | 0.99 | 61 | Open | 1.00 | 0.97 | 0.98 |
12 | C | 1.00 | 1.00 | 1.00 | 62 | Owl | 1.00 | 1.00 | 1.00 |
13 | Coolie | 0.97 | 0.94 | 0.96 | 63 | P | 1.00 | 0.97 | 0.98 |
14 | Cough | 1.00 | 1.00 | 1.00 | 64 | Policy | 0.98 | 1.00 | 0.99 |
15 | Cow | 0.99 | 0.97 | 0.98 | 65 | Pray | 1.00 | 1.00 | 1.00 |
16 | Chest | 1.00 | 1.00 | 1.00 | 66 | Promise | 1.00 | 1.00 | 1.00 |
17 | Claw | 1.00 | 1.00 | 1.00 | 67 | Q | 0.97 | 1.00 | 0.99 |
18 | D | 0.79 | 0.97 | 0.87 | 68 | R | 0.98 | 1.00 | 0.99 |
19 | Devil | 0.98 | 0.94 | 0.96 | 69 | S | 0.95 | 1.00 | 0.97 |
20 | Doctor | 0.98 | 1.00 | 0.99 | 70 | Seven | 0.96 | 1.00 | 0.98 |
21 | E | 1.00 | 1.00 | 1.00 | 71 | Soldier | 1.00 | 1.00 | 1.00 |
22 | East | 1.00 | 1.00 | 1.00 | 72 | Shirt | 1.00 | 1.00 | 1.00 |
23 | Eight | 0.96 | 0.90 | 0.93 | 73 | Six | 1.00 | 0.97 | 0.99 |
24 | Evening | 1.00 | 0.96 | 0.98 | 74 | Sick | 1.00 | 1.00 | 1.00 |
25 | Elbow | 0.95 | 1.00 | 0.98 | 75 | Skin | 1.00 | 1.00 | 1.00 |
26 | Eye | 1.00 | 1.00 | 1.00 | 76 | Shoulder | 1.00 | 0.98 | 0.99 |
27 | F | 0.97 | 0.97 | 0.97 | 77 | Stand | 1.00 | 1.00 | 1.00 |
28 | Fat | 1.00 | 1.00 | 1.00 | 78 | Strong | 0.97 | 1.00 | 0.98 |
29 | Faith | 0.98 | 1.00 | 0.99 | 79 | Sleep | 1.00 | 0.95 | 0.98 |
30 | Fever | 0.95 | 1.00 | 0.97 | 80 | Sunday | 1.00 | 1.00 | 1.00 |
31 | Feel | 0.97 | 1.00 | 0.98 | 81 | T | 0.99 | 1.00 | 0.99 |
32 | Few | 1.00 | 1.00 | 1.00 | 82 | Ten | 1.00 | 0.98 | 0.99 |
33 | Food | 0.98 | 0.86 | 0.92 | 83 | Telephone | 1.00 | 1.00 | 1.00 |
34 | Four | 1.00 | 0.96 | 0.98 | 84 | Tongue | 0.99 | 1.00 | 0.99 |
35 | Fist | 0.97 | 0.98 | 0.97 | 85 | Thorn | 0.96 | 1.00 | 0.98 |
36 | Five | 1.00 | 1.00 | 1.00 | 86 | Three | 0.97 | 1.00 | 0.99 |
37 | G | 0.97 | 0.97 | 0.97 | 87 | Trouble | 1.00 | 0.95 | 0.97 |
38 | Gun | 0.97 | 1.00 | 0.99 | 88 | Two | 1.00 | 1.00 | 1.00 |
39 | Good | 1.00 | 1.00 | 1.00 | 89 | U | 1.00 | 0.99 | 0.99 |
40 | H | 1.00 | 1.00 | 1.00 | 90 | V | 1.00 | 1.00 | 1.00 |
41 | Hair | 1.00 | 1.00 | 1.00 | 91 | W | 1.00 | 1.00 | 1.00 |
42 | Hand | 0.97 | 1.00 | 0.98 | 92 | West | 1.00 | 0.93 | 0.96 |
43 | Head | 0.90 | 0.99 | 0.94 | 93 | Wedding | 0.99 | 0.99 | 0.99 |
44 | Hear | 0.97 | 1.00 | 0.98 | 94 | Water | 0.93 | 0.98 | 0.95 |
45 | I | 1.00 | 1.00 | 1.00 | 95 | White | 1.00 | 0.98 | 0.99 |
46 | Jain | 0.99 | 1.00 | 0.99 | 96 | X | 1.00 | 1.00 | 1.00 |
47 | K | 0.98 | 0.98 | 0.98 | 97 | Y | 1.00 | 1.00 | 1.00 |
48 | King | 1.00 | 0.95 | 0.97 | 98 | You | 0.97 | 1.00 | 0.99 |
49 | L | 1.00 | 1.00 | 1.00 | 99 | Z | 0.99 | 1.00 | 0.99 |
50 | Love | 1.00 | 0.98 | 0.99 | 100 | Zero | 1.00 | 1.00 | 1.00 |
Rights and permissions
About this article
Cite this article
Wadhawan, A., Kumar, P. Deep learning-based sign language recognition system for static signs. Neural Comput & Applic 32, 7957–7968 (2020). https://doi.org/10.1007/s00521-019-04691-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-019-04691-y