Skip to main content
SpringerLink
Log in

Nearest neighbour classification of Indian sign language gestures using kinect camera

  • Published:
Sadhana Aims and scope Submit manuscript

Abstract

People with speech disabilities communicate in sign language and therefore have trouble in mingling with the able-bodied. There is a need for an interpretation system which could act as a bridge between them and those who do not know their sign language. A functional unobtrusive Indian sign language recognition system was implemented and tested on real world data. A vocabulary of 140 symbols was collected using 18 subjects, totalling 5041 images. The vocabulary consisted mostly of two-handed signs which were drawn from a wide repertoire of words of technical and daily-use origins. The system was implemented using Microsoft Kinect which enables surrounding light conditions and object colour to have negligible effect on the efficiency of the system. The system proposes a method for a novel, low-cost and easy-to-use application, for Indian Sign Language recognition, using the Microsoft Kinect camera. In the fingerspelling category of our dataset, we achieved above 90% recognition rates for 13 signs and 100% recognition for 3 signs with overall 16 distinct alphabets (A, B, D, E, F, G, H, K, P, R, T, U, W, X, Y, Z) recognised with an average accuracy rate of 90.68%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 15
Figure 16
Figure 17
Figure 18

Similar content being viewed by others

References

  1. Schmitz M, Endres C and Butz A 2008 A survey of human-computer interaction design in science fiction movies. In: Proceedings of the 2nd international conference on INtelligent TEchnologies for interactive enterTAINment, page 7. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering)

  2. Bauman D 2008 Open your eyes: Deaf studies talking. University of Minnesota Press

  3. Muni B 1951 Natya Shastra. Calcutta: Asiatic Society of Bengal

  4. Bulwer J 1648 Philocopus, or the deaf and dumb man’s friend. London: Humphrey and Moseley

    Google Scholar 

  5. Cornett R O 1967 Cued speech. University of Nebraska Media Center. Captioned Films for the Deaf

  6. Geetha M and Manjusha U 2012 A vision based recognition of indian sign language alphabets and numerals using B-spline approximation. Int. J. Comp. Sci. Eng. (IJCSE)

  7. Pugeault N and Bowden R 2011 Spelling it out: Real-time ASL fingerspelling recognition. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 1114–1119

  8. ISL 2011 Sign dictionaries for indian deaf and dumb population. Sign Language Unit, The Faculty of Disability Management & Special Education (FDMSE) of Ramakrishna Mission Vidyalaya, Perianaickenpalayam, Coimbatore, India. indiansignlanguage.org

  9. Mitra S and Acharya T 2007 Gesture recognition: A survey. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 37 (3): 311–324

    Article  Google Scholar 

  10. Bilal S, Akmeliawati R, El Salami M J and Shafie A A 2011 Vision-based hand posture detection and recognition for sign language–a study. In: 4th International Conference on Mechatronics (ICOM), 2011, pages 1–6

  11. Quek F K and Zhao M 1996 Inductive learning in hand pose recognition. In: Proceedings of the Second International Conference on Automatic Face and Gesture Recognition, 1996, pages 78–83

  12. Starner T, Weaver J and Pentland A 1998 Real-time american sign language recognition using desk and wearable computer based video. IEEE Trans. Pattern Anal. Mach. Intell. 20 (12): 1371–1375

    Article  Google Scholar 

  13. Luis-Pérez F E, Trujillo-Romero F and Martínez-Velazco W 2011 Control of a service robot using the mexican sign language. In: Adv. Soft Comput., pages 419–430. Springer

  14. Rekha J, Bhattacharya J and Majumder S 2011 Shape, texture and local movement hand gesture features for Indian Sign Language recognition. In: 3rd International Conference on Trendz in Information Sciences and Computing (TISC), 2011, pages 30–35

  15. Singha J and Das K 2013 Indian sign language recognition using eigen value weighted euclidean distance based classification technique. arXiv preprint arXiv:1303.0634

  16. Bhuyan M, Kar M K and Neog D R 2011 Hand pose identification from monocular image for sign language recognition. In: 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), pages 378–383

  17. Kenn H, Megen F V and Sugar R 2007 A glove-based gesture interface for wearable computing applications. In: 4th International Forum on Applied Wearable Computing (IFAWC), 2007, pages 1–10

  18. Hernandez-Rebollar J L, Lindeman R W and Kyriakopoulos N 2002 A multi-class pattern recognition system for practical finger spelling translation. In: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, ICMI ’02, pages 185–, Washington, DC, USA. IEEE Computer Society

  19. Liang R -H and Ouhyoung M 1998 A real-time continuous gesture recognition system for sign language. In: Proceedings of the Third IEEE international conference on automatic face and gesture recognition, 1998, pages 558–567

  20. Saengsri S, Niennattrakul V and Ratanamahatana C 2012 Tfrs: Thai finger-spelling sign language recognition system. In: Second International Conference on Digital Information and Communication Technology and it’s Applications (DICTAP), 2012, pages 457–462

  21. Van den Bergh M and Van Gool L 2011 Combining RGB and ToF cameras for real-time 3D hand gesture interaction. In: IEEE Workshop on Applications of Computer Vision (WACV), 2011, pages 66–72

  22. Wang F and Zhang C 2007 Feature extraction by maximizing the average neighbourhood margin. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR’07, pages 1–8

  23. Argyros A and Lourakis M I A 2006 Binocular hand tracking and reconstruction based on 2D shape matching. In: 18th International Conference on pattern recognition, 2006. ICPR 2006. volume 1, pages 207–210

  24. Keskin C, Kıraç F, Kara Y E and Akarun L 2013 Real time hand pose estimation using depth sensors. In: Consumer depth cameras for computer vision, pages 119–137. Springer

  25. Ren Z, Yuan J and Zhang Z 2011 Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera. In: Proceedings of the 19th ACM international conference on Multimedia, MM ’11, pages 1093–1096, New York, NY, USA

  26. Shotton J, Sharp T, Kipman A, Fitzgibbon A, Finocchio M, Blake A, Cook M and Moore R 2013 Real-time human pose recognition in parts from single depth images. Commun. ACM 56 (1): 116–124

    Article  Google Scholar 

  27. Kramer J, Parker M, Herrera D, Burrus N and Echtler F 2012 Hacking the kinect. Apress

  28. ElKoura G and Singh K 2003 Handrix: Animating the human hand. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on computer animation, pages 110–119. Eurographics Association

  29. PrimeSense 2011 OpenNI platform 1.0

  30. Rusu R B and Cousins S 2011 3D is here: Point cloud library (PCL). In: IEEE International Conference on Robotics and Automation (ICRA), 2011, pages 1–4

  31. Buades A, Coll B and Morel J M 2005 A non-local algorithm for image denoising. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, vol. 2, pages 60–65

  32. Tukey J W 1977 Exploratory data analysis. Reading, MA, 231

  33. Shi J and Malik J 2000 Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22 (8): 888–905

    Article  Google Scholar 

  34. Lowe D G 2004 Distinctive image features from scale-invariant keypoints. Int. J. Comp. Vis. 60 (2): 91–110

    Article  Google Scholar 

  35. Lindeberg T 2012 Scale invariant feature transform. Scholarpedia 7 (5): 10491

    Article  Google Scholar 

  36. Lindeberg T 1994 Scale-space theory: A basic tool for analyzing structures at different scales. J. Appl. Stat. 21 (1–2): 225–270

    Article  Google Scholar 

  37. Lindeberg T 1998 Feature detection with automatic scale selection. Int. J. Comp. Vis. 30 (2): 79–116

    Article  Google Scholar 

  38. Bundy A and Wallen L 1984 Difference of gaussians. In: Bundy A and Wallen L (eds) Catalogue of artificial intelligence tools, symbolic computation, page 30. Springer, Berlin Heidelberg

  39. Bhatnagar S 2007 Adaptive newton-based multivariate smoothed functional algorithms for simulation optimization. ACM Trans. Model. Comput. Simul. 18 (1): 2:1–2:35

    Article  Google Scholar 

  40. Lindeberg T 1993 Scale-space theory in computer vision. Springer

  41. Weickert J, Ishikawa S and Imiya A 1999 Linear scale-space has first been proposed in japan. J. Math. Imaging Vis. 10 (3): 237–252

    Article  MathSciNet  MATH  Google Scholar 

  42. Lindeberg T 2013 Generalized axiomatic scale-space theory. Adv. Imaging Electron Phys. 178: 1

    Article  Google Scholar 

  43. Yuille A L and Poggio T A 1986 Scaling theorems for zero crossings. IEEE Trans. Pattern Anal. Mach. Intell. 8 (1): 15–25

    Article  MATH  Google Scholar 

  44. Babaud J, Witkin A P, Baudin M and Duda R O 1986 Uniqueness of the gaussian kernel for scale-space filtering. IEEE Trans. Pattern Anal. Mach. Intell. 8 (1): 26–33

    Article  MATH  Google Scholar 

  45. Allaire S, Kim J J, Breen S L, Jaffray D A and Pekar V 2008 Full orientation invariance and improved feature selectivity of 3d SIFT with application to medical image analysis. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008. CVPRW’08, pages 1–8

  46. Rusu R B, Bradski G, Thibaux R and Hsu J 2010 Fast 3D recognition and pose using the viewpoint feature histogram. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2010, pages 2155–2162

  47. Rusu R B, Blodow N and Beetz M 2009 Fast point feature histograms (FPFH) for 3d registration. In: IEEE International Conference on Robotics and Automation, 2009. ICRA’09, pages 3212–3217

  48. Bay H, Tuytelaars T and Van Gool L 2006 Surf: Speeded up robust features. In: Computer Vision–ECCV 2006, pages 404–417. Springer

  49. McDonnell M 1981 Box-filtering techniques. Comp. Graph. Image Process. 17 (1): 65–70

    Article  Google Scholar 

  50. Viola P and Jones M 2001 Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. CVPR 2001. vol. 1, pages I-511–I-518

  51. Ansari Z 2013a Gesture recognition for Indian Sign Language with Kinect Xbox 360. https://www.youtube.com/watch?v=2oqD-_UCHxQ. Accessed: 2013-06-18

  52. Ansari Z A 2013b Gesture recognition for Indian sign language. Master’s thesis, Indian Institute of Technology Jodhpur, India

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur, Rajasthan, 342011, India

    ZAFAR AHMED ANSARI & GAURAV HARIT

Authors
  1. ZAFAR AHMED ANSARI
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. GAURAV HARIT
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to GAURAV HARIT.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

ANSARI, Z.A., HARIT, G. Nearest neighbour classification of Indian sign language gestures using kinect camera. Sadhana 41, 161–182 (2016). https://doi.org/10.1007/s12046-015-0405-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12046-015-0405-3

Keywords

Search

Navigation