Skip to main content
SpringerLink
Log in

A Survey on Artificial Intelligence in Chinese Sign Language Recognition

  • Review Article-Computer Engineering and Computer Science
  • Published:
Arabian Journal for Science and Engineering Aims and scope Submit manuscript

Abstract

Chinese Sign Language (CSL) offers the main means of communication for the hearing impaired in China. Sign Language Recognition (SLR) can shorten the distance between the hearing-impaired and healthy people and help them integrate into the society. Therefore, SLR has become the focus of sign language application research. Over the years, the continuous development of new technologies provides a source and motivation for SLR. This paper aims to cover the most recent approaches in Chinese Sign Language Recognition (CSLR). With a thorough review of superior methods from 2000 to 2019 in CSLR researches, various techniques and algorithms such as scale-invariant feature transform, histogram of oriented gradients, wavelet entropy, Hu moment invariant, Fourier descriptor, gray-level co-occurrence matrix, dynamic time warping, principal component analysis, autoencoder, hidden Markov model (HMM), support vector machine (SVM), random forest, skin color modeling method, k-NN, artificial neural network, convolutional neural network (CNN), and transfer learning are discussed in detail, which are based on several major stages, that is, data acquisition, preprocessing, feature extraction, and classification. CSLR was summarized from some aspect as follows: methods of classification and feature extraction, accuracy/performance evaluation, and sample size/datasets. The advantages and limitations of different CSLR approaches were compared. It was found that data acquisition is mainly through Kinect and camera, and the feature extraction focuses on hand’s shape and spatiotemporal factors, but ignoring facial expressions. HMM and SVM are used most in the classification. CNN is becoming more and more popular, and a deep neural network-based recognition approach will be the future trend. However, due to the complexity of the contemporary Chinese language, CSLR generally has a lower accuracy than other SLR. It is necessary to establish an appropriate dataset to conduct comparable experiments. The issue of decreasing accuracy as the dataset increases needs to resolve. Overall, our study is hoped to give a comprehensive presentation for those people who are interested in CSLR and SLR and to further contribute to the future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28

Similar content being viewed by others

References

  1. Li, X.: Research on Chinese Sign Language Recognition for Middle and Small Vocabulary Based on Neural Network, pp. 1–10. University of Science and Technology of China, Hefei (2017)

    Google Scholar 

  2. Zhao, X.: A review of Chinese and Foreign Sign Language dictionaries. Lexicogr. Stud. 5, 44–50 (2015)

    Google Scholar 

  3. Wang, H.; Chai, X.; Hong, X.; Zhao, G.; Chen, X.: Isolated Sign Language Recognition with Grassmann covariance matrices. ACM Trans. Access. Comput. 8(4), 14–22 (2016)

    Google Scholar 

  4. Jiang, S.; Gao, Q.; Liu, H.; Shull, P.: A novel, co-located EMG–FMG-sensing wearable armband for hand gesture recognition. Sens. Actuators A 301, 111738 (2020)

    Google Scholar 

  5. Ghanem, S.; Conly, C.; Athitsos, V.: A survey on Sign Language Recognition using smartphones. In: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, pp. 171–176 (2017)

  6. Lu, W.; Tong, Z.; Chu, J.: Dynamic hand gesture recognition with leap motion controller. IEEE Signal Process. Lett. 23(9), 1188–1192 (2016)

    Google Scholar 

  7. Dalawis, R.C.; Olayao, K.D.R.; Ramos, E.G.I.; Samonte, M.J.C.: Kinect-based Sign Language Recognition of static and dynamic hand movements. In: Eighth International Conference on Graphic and Image Processing (ICGIP 2016), vol. 10225, p. 102250I. International Society for Optics and Photonics (2017)

  8. Cheok, M.; Omar, Z.; Jaward, M.: A review of hand gesture and Sign Language Recognition techniques. Int. J. Mach. Learn. Cybern. 10(1), 131–153 (2019)

    Google Scholar 

  9. Rathi, S.; Gawande, U.: Development of full duplex intelligent communication system for deaf and dumb people. In: 2017 IEEE 7th International Conference on Cloud Computing, Data Science and Engineering—Confluence, pp. 733–738 (2017)

  10. Gupta, B.; Shukla, P.; Mittal, A.: K-nearest correlated neighbour classification for Indian Sign Language gesture recognition using feature fusion. In: 2016 International Conference on Computer Communication and Informatics (ICCCI), pp. 1–10 (2015)

  11. Chai, D.; Bouzerdoum, A.: A Bayesian approach to skin color classification in YCbCr color space. In: 2000 TENCON Proceedings. Intelligent Systems and Technologies for the New Millennium (Cat. No. 00CH37119), vol. 2, pp. 421–424. IEEE (2000)

  12. Phung, S.L.; Bouzerdoum, A.; Chai, D.: A novel skin color model in YCbCr color space and its application to human face detection. In: Proceedings. International Conference on Image Processing, vol. 1, p. 1. IEEE (2002)

  13. Kaur, A.; Kranthi, B.: Comparison between YCbCr color space and CIELab color space for skin color segmentation. Int. J. Appl. Inf. Syst. 3(4), 30–33 (2012)

    Google Scholar 

  14. Kawulok, M.: Dynamic skin detection in color images for Sign Language Recognition. In: International Conference on Image and Signal Processing ICISP 2008, pp. 112–119 (2008)

  15. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)

    Google Scholar 

  16. Tharwat, A.; Gaber, T.; Hassanien, A.E.; Shahin, M.K.; Refaat, B.: Sift-based Arabic Sign Language Recognition system. In: Afro-European Conference for Industrial Advancement, pp. 359–370. Springer, Berlin (2015)

  17. Dardas, N.H.; Georganas, N.D.: Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Trans. Instrum. Meas. 60, 3592–3607 (2011)

    Google Scholar 

  18. Pan, T.-Y.; Lo, L.-Y.; Yeh, C.-W.; Li, J.-W.; Liu, H.-T.; Hu, M.-C.: Realtime Sign Language Recognition in complex background scene based on a hierarchical clustering classification method. In: Multimedia Big Data (BigMM), 2016 IEEE Second International Conference, pp. 64–67. IEEE (2016)

  19. Silanon, K.: Thai finger-spelling recognition using a cascaded classifier based on histogram of orientation gradient features. Comput. Intell. Neurosci. 2017, 1–11 (2017)

    Google Scholar 

  20. Zhang, J.; Zhou, W.; Xie, C.; Pu, J.; Li, H.: Chinese Sign Language Recognition with adaptive HMM. In: 2016 IEEE International Conference on Multimedia and Expo (ICME), Seattle, WA, pp. 1–6 (2016)

  21. Li, W.; Lin, Y.; Fu, B.; Sun, M.; Wu, W.: Cascade classifier using combination of histograms of oriented gradients for rapid pedestrian detection. J. Softw. 8(1), 71–77 (2013)

    Google Scholar 

  22. Wang, H.; Chai, X.; Zhou, Y.; Chen, X.: Fast Sign Language Recognition benefited from low rank approximation. In: 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2015, pp. 15–21 (2015)

  23. He, J.; Liu, Z.; Zhang, J.: Chinese Sign Language Recognition based on trajectory and hand shape features. In: Visual Communications and Image Processing (VCIP), Chengdu, 2016, pp. 1–4 (2016)

  24. Chen, Y.; Zhang, W.: Research and implementation of Sign Language Recognition method based on Kinect. In: 2nd IEEE International Conference on Computer and Communications (ICCC), Chengdu, pp. 1947–1951 (2016)

  25. Liu, H.; Xu, T.; Wang, X.; Qian, Y.: Related HOG features for human detection using cascaded adaboost and SVM classifiers. In: Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 7733(2), pp. 345–355 (2013)

  26. Gorriz, J.M.; Ramírez, J.: Wavelet entropy and directed acyclic graph support vector machine for detection of patients with unilateral hearing loss in MRI scanning. Front. Comput. Neurosci. 10, 1–10 (2016), Art. No. 160

  27. Wu, X.: Tea category identification based on optimal wavelet entropy and weighted k-nearest neighbors algorithm. Multimed. Tools Appl. 77(3), 3745–3759 (2018)

    Google Scholar 

  28. Jiang, X.; Zhu, Z.: Chinese Sign Language identification via wavelet entropy and support vector machine. In: Li, J., Wang, S., Qin, S., Li, X., Wang, S. (eds.) Advanced Data Mining and Applications. ADMA 2019. Lecture Notes in Computer Science, vol. 11888, pp. 726–736 (2019)

  29. Zhang, Y.; Wang, S.; Phillipsd, P.; Dong, Z.; Ji, G.; Yang, J.: Detection of Alzheimer’s disease and mild cognitive impairment based on structural volumetric MR images using 3D-DWT and WTA-KSVM trained by PSOTVAC. Biomed. Signal Process. Control 21, 58–73 (2015)

    Google Scholar 

  30. Lichtenauer, J.F.; Hendriks, E.A.; Reinders, M.J.: Sign Language Recognition by combining statistical DTW and independent classification. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2040–2046 (2008)

    Google Scholar 

  31. Rekha, J.; Bhattacharya, J.; Majumder, S.: Shape, texture and local movement hand gesture features for Indian Sign Language Recognition. In: 3rd International Conference on Trends in Information Sciences and Computing (TISC2011), pp. 30–35. IEEE (2011)

  32. Ahmed, W.; Chanda, K.; Mitra, S.: Vision based hand gesture recognition using dynamic time warping for Indian Sign Language. In: 2016 International Conference on Information Science (ICIS), pp. 120–125. IEEE (2016)

  33. Cao, Y.; Rakhilin, N.; Gordon, P.H.; Shen, X.; Kan, E.C.: A real-time spike classification method based on dynamic time warping for extracellular enteric neural recording with large waveform variability. J. Neurosci. Methods 261, 97–109 (2016)

    Google Scholar 

  34. Zhao, J.; Itti, L.: shapedtw: shape dynamic time warping. Pattern Recognit. 74, 171–184 (2018)

    Google Scholar 

  35. Chen, Y.; et al.: Delineating urban functional areas with building-level social media data: a dynamic time warping (DTW) distance based k-medoids method. Landsc. Urban Plan. 160, 48–60 (2017)

    Google Scholar 

  36. Zhang, Y.; Wang, S.; Sun, P.; Phillips, P.: Pathological brain detection based on wavelet entropy and Hu moment invariants. Bio-Med. Mater. Eng. 26, 1283–1290 (2015)

    Google Scholar 

  37. Sokic, E.; Konjicija, S.: Phase preserving Fourier descriptor for shape-based image retrieval. Signal Process. Image Commun. 40, 82–96 (2016)

    Google Scholar 

  38. Huang, X.; Liu, X.; Zhang, L.: A multichannel gray level co-occurrence matrix for multi/hyperspectral image texture representation. Remote Sens. 6, 8424–8445 (2014)

    Google Scholar 

  39. Yang, Q.; Peng, J.: Chinese Sign Language Recognition for a vision-based multi-features classifier. In: 2008 International Symposium on Computer Science and Computational Technology, vol. 2, pp. 194–197. IEEE (2008)

  40. Peng, J.; Li, Y.: Chinese Sign Language Recognition based on gray-level co-occurrence matrix and other multi-features fusion. In: 2009 4th IEEE Conference on Industrial Electronics and Applications, pp. 1569–1572. IEEE (2009)

  41. Arifin, N.A.; Irawan, B.; Setianingsih, C.: Traffic sign recognition application using speeded-up robust features (SURF) and support vector machine (SVM) based on android. In: 2017 IEEE Asia Pacific Conference on Wireless and Mobile (APWiMob), pp. 52–58. IEEE (2017)

  42. Toubia, O.; Iyengar, G.; Bunnell, R.; Lemaire, A.: Extracting features of entertainment products: a guided latent Dirichlet allocation approach informed by the psychology of media consumption. J. Mark. Res. 56(1), 18–36 (2019)

    Google Scholar 

  43. Kumar, G.; Bhatia, P.K.: A detailed review of feature extraction in image processing systems. In: 2014 Fourth International Conference on Advanced Computing and Communication Technologies, pp. 5–12. IEEE (2014)

  44. Huong, T.N.T.; Huu, T.V.; Le Xuan, T.: Static hand gesture recognition for Vietnamese Sign Language (VSL) using principle components analysis. In: 2015 International Conference on Communications, Management and Telecommunications (ComMan-Tel), pp. 138–141. IEEE (2015)

  45. Zaki, M.M.; Shaheen, S.I.: Sign Language Recognition using a combination of new vision based features. Pattern Recognit. Lett. 32, 572–577 (2011)

    Google Scholar 

  46. Liou, C.-Y.; Cheng, W.-C.; Liou, J.-W.; Liou, D.-R.: Autoencoder for words. Neurocomputing 139, 84–96 (2014)

    Google Scholar 

  47. Wang, Y.; Yao, H.; Zhao, S.: Auto-encoder based dimensionality reduction. Neurocomputing 184, 232–242 (2016)

    Google Scholar 

  48. Shi, B.; Livescu, K.: Multitask training with unlabeled data for end-to-end sign language fingerspelling recognition. In: 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Okinawa, pp. 389–396 (2017)

  49. Abd El-Mohsen, A.; Eman, M.; Abou-Chadi, F.E.: Sign language hand gesture recognition using autoencoder and support vector machine classifiers. In: IJSSST, pp. 1–7 (2017)

  50. Parcheta, Z.; Martínez-Hinarejos, C.-D.: Sign language gesture recognition using HMM. In: Iberian Conference on Pattern Recognition and Image Analysis, IbPRIA 2017, vol. 10255, pp. 419–426 (2017)

  51. Koller, O.; Zargaran, O.; Ney, H.; Bowden, R.: Deep sign: hybrid CNN-HMM for continuous Sign Language Recognition. In: The British Machine Vision Conference (BMVC) 2016, pp. 1–10 (2016)

  52. Elmezain, M.; Al-Hamadi, A.; Appenrodt, J.; Michaelis, B.: A hidden Markov model-based isolated and meaningful hand gesture recognition. Int. J. Electr. Comput. Syst. Eng. 3, 156–163 (2009)

    Google Scholar 

  53. Maji, S.; Berg, A.C.; Malik, J.: Efficient classification for additive kernel SVMs. IEEE Trans. Pattern Anal. Mach. Intell. 35(1), 66–77 (2012)

    Google Scholar 

  54. Zhang, Y.; Wu, L.: Classification of fruits using computer vision and a multiclass support vector machine. Sensors 12, 12489–12505 (2012)

    Google Scholar 

  55. Zhang, Y.; Wang, S.; Dong, Z.: Classification of Alzheimer disease based on structural magnetic resonance imaging by kernel support vector machine decision tree. Prog. Electromagn. Res. 144, 171–184 (2014)

    Google Scholar 

  56. Kumar, P.; Saini, R.; Roy, P.P.; Dogra, D.P.: A position and rotation invariant framework for Sign Language Recognition (SLR) using Kinect. Multimed. Tools Appl. 77(7), 8823–8846 (2018)

    Google Scholar 

  57. Neiva, D.H.; Zanchettin, C.: Gesture recognition: a review focusing on sign language in a mobile context. Expert Syst. Appl. 103, 159–183 (2018)

    Google Scholar 

  58. Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    MATH  Google Scholar 

  59. Chavan, P.; Ghorpade, T.; Padiya, P.: Indian Sign Language to forecast text using leap motion sensor and RF classifier. In: 2016 Symposium on Colossal Data Analysis and Networking (CDAN), Indore, pp. 1–5 (2016)

  60. Su, R.; Chen, X.; Cao, S.; Zhang, X.: Random forest-based recognition of isolated sign language subwords using data from accelerometers and surface electromyographic sensors. Sensors 16(1), 100–105 (2016)

    Google Scholar 

  61. Hassanat, A.B.; Abbadi, M.A.; Altarawneh, G.A.; Alhasanat, A.A.: Solving the problem of the K parameter in the KNN classifier using an ensemble learning approach. arXiv preprint arXiv:1409.0919 (2014)

  62. Pattanaworapan, K.; Chamnongthai, K.; Guo, J.M.: Signer-independence finger alphabet recognition using discrete wavelet transform and area level run lengths. J. Vis. Commun. Image Represent. 38, 658–677 (2016)

    Google Scholar 

  63. Wong, S.-F.; Cipolla, R.: Real-time adaptive hand motion recognition using a sparse Bayesian classifier. In: International Workshop on Human–Computer Interaction, pp. 170–179. Springer, Berlin (2005)

  64. Pramunanto, E.; Sumpeno, S.; Legowo, R.S.: Classification of hand gesture in Indonesian Sign Language system using Naive Bayes. In: 2017 International Seminar on Sensors, Instrumentation, Measurement and Metrology (ISSIMM), pp. 187–191 (2017)

  65. Rao, G.A.; Kishore, P.V.V.: Selfie Sign Language Recognition with multiple features on adaboost multilabel multiclass classifier. J. Eng. Sci. Technol. 13(8), 2352–2368 (2018)

    Google Scholar 

  66. Zhao, Y.; Wang, L.: The application of convolution neural networks in Sign Language Recognition. In: 2018 Ninth International Conference on Intelligent Control and Information Processing (ICICIP), pp. 269–272. IEEE (2018)

  67. Rastegari, M.; Ordonez, V.; Redmon, J.; Farhadi, A.: Xnor-net: imagenet classification using binary convolutional neural networks. In: European Conference on Computer Vision, pp. 525–542. Springer, Berlin (2016)

  68. Jin, K.H.; McCann, M.T.; Froustey, E.; Unser, M.: Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process. 26(9), 4509–4522 (2017)

    MathSciNet  MATH  Google Scholar 

  69. Shi, W.; et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)

  70. Jiang, X.; Lu, M.; Wang, S.-H.: An eight-layer convolutional neural network with stochastic pooling, batch normalization and dropout for fingerspelling recognition of Chinese Sign Language. Multimed. Tools Appl. 79, 15697–15715 (2019)

    Google Scholar 

  71. Jiang, X.; Zhang, Y.-D.: Chinese Sign Language fingerspelling via six-layer convolutional neural network with leaky rectified linear units for therapy and rehabilitation. J. Med. Imaging Health Inform. 9(9), 2031–2090 (2019)

    Google Scholar 

  72. Li, Y.; Wang, N.; Shi, J.; Hou, X.; Liu, J.: Adaptive batch normalization for practical domain adaptation. Pattern Recognit. 80, 109–117 (2018)

    Google Scholar 

  73. Yang, G.; Pennington, J.; Rao, V.; Sohl-Dickstein, J.; Schoenholz, S.S.: A mean field theory of batch normalization. arXiv preprint arXiv:1902.08129 (2019)

  74. Krizhevsky, A.; Sutskever, I.; Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 84–90 (2017)

    Google Scholar 

  75. Gal, Y.; Ghahramani, Z.: A theoretically grounded application of dropout in recurrent neural networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 1019–1027. Barcelona, Spain (2016)

  76. Cai, S.; Gao, J.; Zhang, M.; Wang, W.; Chen, G.; Ooi, B.C.: Effective and efficient dropout for deep convolutional neural networks. arXiv preprint arXiv:1904.03392 (2019)

  77. Schmidt-Hieber, J.: Nonparametric regression using deep neural networks with ReLU activation function. arXiv preprint arXiv:1708.06633 (2017)

  78. Lin, G.; Shen, W.: Research on convolutional neural network based on improved Relu piecewise activation function. Procedia Comput. Sci. 131, 977–984 (2018)

    Google Scholar 

  79. Salamon, J.; Bello, J.P.: Deep convolutional neural networks and data augmentation for environmental sound classification. IEEE Signal Process. Lett. 24(3), 279–283 (2017)

    Google Scholar 

  80. Peng, X.; Tang, Z.; Yang, F.; Feris, R.S.; Metaxas, D.: Jointly optimize data augmentation and network training: adversarial data augmentation in human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2226–2234 (2018)

  81. Hershey, S.; et al.: CNN architectures for large-scale audio classification. In: 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 131–135. IEEE (2017)

  82. Weiss, K.; Khoshgoftaar, T.M.; Wang, D.: A survey of transfer learning. J. Big Data 3(1), 9 (2016)

    Google Scholar 

  83. Jiang, X.; Hu, B.; Chandra Satapathy, S.; Wang, S.-H.; Zhang, Y.-D.: Fingerspelling identification for Chinese Sign Language via AlexNet-based transfer learning and Adam optimizer. Sci. Program. 2020, 1–10 (2020)

    Google Scholar 

  84. Zhang, Y.-D.; Govindaraj, V.V.; Tang, C.; Zhu, W.; Sun, J.: High performance multiple sclerosis classification by data augmentation and AlexNet transfer learning model. J. Med. Imaging Health Inform. 9(9), 2012–2021 (2019)

    Google Scholar 

  85. Krizhevsky, A.; Sutskever, I.; Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc., Red Hook (2012)

    Google Scholar 

  86. Alom, M.Z.; et al.: The history began from AlexNet: a comprehensive survey on deep learning approaches. arXiv preprint arXiv:1803.01164 (2018)

  87. Lecun, Y.; Bottou, L.; Bengio, Y.; Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)

    Google Scholar 

  88. Samir, S.; Emary, E.; El-Sayed, K.; Onsi, H.: Optimization of a pre-trained AlexNet model for detecting and localizing image forgeries. Information 11(5), 275 (2020)

    Google Scholar 

  89. Han, X.; Zhong, Y.; Cao, L.; Zhang, L.: Pre-trained alexnet architecture with pyramid pooling and supervision for high spatial resolution remote sensing image scene classification. Remote Sens. 9(8), 848 (2017)

    Google Scholar 

  90. Iandola, F.N.; Han, S.; Moskewicz, M.W.; Ashraf, K.; Dally, W.J.; Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50× fewer parameters and < 0.5 MB model size. arXiv preprint arXiv:1602.07360 (2016)

  91. LeCun, Y.; Bengio, Y.; Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Google Scholar 

  92. Ngiam, J.; Khosla, A.; Kim, M.; Nam, J.; Lee, H.; Ng, A.Y.: Multimodal deep learning. In: Proceedings of the 28th International Conference on International Conference on Machine Learning (ICML), Washington, USA, pp. 689–696 (2011)

  93. Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)

    Google Scholar 

  94. He, K.; Zhang, X.; Ren, S.; Sun, J.: Identity mappings in deep residual networks. In: European Conference on Computer Vision, pp. 630–645. Springer, Berlin (2016)

  95. He, K.; Zhang, X.; Ren, S.; Sun, J.: Deep Residual Learning for Image Recognition, pp. 1–10. arXiv:1512.03385 [cs] (2015)

  96. Dai, J.; He, K.; Sun, J.: Instance-aware semantic segmentation via multi-task network cascades. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3150–3158 (2016)

  97. Xie, S.; Girshick, R.; Dollár, P.; Tu, Z.; He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492–1500 (2017)

  98. Simonyan, K.; Zisserman, A.: Very deep convolutional networks for large-scale image recognition, pp. 1–10. arXiv:1409.1556 [cs] (2014)

  99. Szegedy, C.; et al.: Going deeper with convolutions. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–10 (2015)

  100. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

  101. Song, N.; Yang, H.; Wu, P.: A gesture-to-emotional speech conversion by combining gesture recognition and facial expression recognition. In: First Asian Conference on Affective Computing and Intelligent Interaction (ACII Asia), Beijing, pp. 1–6 (2018)

  102. Yang, Q.; Peng, J.-Y.: Chinese Sign Language Recognition method based on depth image information and SURF-BoW. Pattern Recognit. Artif. Intell. 27(8), 741–749 (2014) (in Chinese)

    Google Scholar 

  103. Zhang, J.; Zhou, W.; Li, H.: A new system for Chinese Sign Language Recognition. In: 2015 IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP), Chengdu, pp. 534–538 (2015)

  104. Wang, C.; Chen, X.; Gao, W.: Expanding training set for Chinese Sign Language Recognition. In: 7th International Conference on Automatic Face and Gesture Recognition (FGR06), pp. 323–328. IEEE (2006)

  105. Fang, G.; Gao, W.; Zhao, D.: Large-vocabulary continuous Sign Language Recognition based on transition-movement models. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 37(1), 1–9 (2006)

    Google Scholar 

  106. Wang, C.; Gao, W.; Ma, J.: A real-time large vocabulary recognition system for Chinese Sign Language. In: International Gesture Workshop, pp. 86–95. Springer, Berlin (2001)

  107. Fang, G.; Gao, W.; Chen, X.; Wang, C.; Ma, J.: Signer-independent continuous Sign Language Recognition based on SRN/HMM. In: International Gesture Workshop, pp. 76–85. Springer, Berlin (2001)

  108. Yuan, Q.; Geo, W.; Yao, H.; Wang, C.: Recognition of strong and weak connection models in continuous sign language. In: Object Recognition Supported by User Interaction for Service Robots, vol. 1, pp. 75–78. IEEE (2002)

  109. Zhang, L.-G.; Chen, Y.; Fang, G.; Chen, X.; Gao, W.: A vision-based Sign Language Recognition system using tied-mixture density HMM. In: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 198–204 (2004)

  110. Gao, W.; Fang, G.; Zhao, D.; Chen, Y.: A Chinese Sign Language Recognition system based on SOFM/SRN/HMM. Pattern Recognit. 37(12), 2389–2402 (2004)

    MATH  Google Scholar 

  111. Zhang, L.-G.; Chen, X.; Wang, C.; Chen, Y.; Gao, W.: Recognition of sign language subwords based on boosted hidden Markov models. In: Proceedings of the 7th International Conference on Multimodal Interfaces, pp. 282–287 (2005)

  112. Li, Y.; Chen, X.; Tian, J.; Zhang, X.; Wang, K.; Yang, J.: Automatic recognition of sign language subwords based on portable accelerometer and EMG sensors. In: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction, pp. 1–7 (2010)

  113. Zhang, X.; Chen, X.; Li, Y.; Lantz, V.; Wang, K.; Yang, J.: A framework for hand gesture recognition based on accelerometer and EMG sensors. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 41(6), 1064–1076 (2011)

    Google Scholar 

  114. Li, Y.; Chen, X.; Zhang, X.; Wang, K.; Wang, Z.: A sign-component-based framework for Chinese Sign Language Recognition using accelerometer and sEMG data. IEEE Trans. Biomed. Eng. 59(10), 2695–2704 (2012)

    Google Scholar 

  115. Liao, Y.; Xiong, P.; Min, W.; Min, W.; Lu, J.: Dynamic Sign Language Recognition based on video sequence with BLSTM-3D residual networks. IEEE Access 7, 38044–38054 (2019)

    Google Scholar 

  116. Zhang, Z.; Su, Z.; Yang, G.: Real-time Chinese Sign Language Recognition based on artificial neural networks. In: 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), Dali, China, pp. 1413–1417 (2019)

  117. Yang, S.; Zhu, Q.: Video-based Chinese Sign Language Recognition using convolutional neural network. In: IEEE 9th International Conference on Communication Software and Networks (ICCSN), Guangzhou, pp. 929–934 (2017)

  118. Huang, J.; Zhou, W.; Li, H.; Li, W.: Attention-based 3D-CNNs for large-vocabulary Sign Language Recognition. IEEE Trans. Circuits Syst. Video Technol. 29(9), 2822–2832 (2019)

    Google Scholar 

  119. Ma, J.; Gao, W.; Wu, J.; Wang, C.: A continuous Chinese Sign Language Recognition system. In: Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580), pp. 428–433. IEEE (2000)

  120. Fang, G.; Gao, W.: A SRN/HMM system for signer-independent continuous Sign Language Recognition. In: Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition, pp. 312–317. IEEE (2002)

  121. Fang, G.; Gao, X.; Gao, W.; Chen, Y.: A novel approach to automatically extracting basic units from Chinese Sign Language. In: Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004, vol. 4, pp. 454–457. IEEE (2004)

  122. Wang, C.; Chen, X.; Gao, W.: A comparison between etymon-and word-based Chinese Sign Language Recognition systems. In: International Gesture Workshop, pp. 84–87. Springer, Berlin (2005)

  123. Chai, X.; et al.: Sign Language Recognition and translation with Kinect. In: IEEE Conference on AFGR, vol. 655, p. 4 (2013)

  124. Geng, L.; Ma, X.; Wang, H.; Gu, J.; Li, Y.: Chinese Sign Language Recognition with 3D hand motion trajectories and depth images. In: Proceeding of the 11th World Congress on Intelligent Control and Automation, pp. 1457–1461. IEEE (2014)

  125. Jiang, Y.; Tao, J.; Ye, W.; Wang, W.; Ye, Z.: An isolated Sign Language Recognition system using RGB-D sensor with sparse coding. In: 2014 IEEE 17th International Conference on Computational Science and Engineering, pp. 21–26. IEEE (2014)

  126. Li, L.; Dashun, Q.: Design of data-glove and Chinese Sign Language Recognition system based on ARM9. In: 12th IEEE International Conference on Electronic Measurement and Instruments (ICEMI), Qingdao, pp. 1130–1134 (2015)

  127. Liu, T.; Zhou, W.; Li, H.: Sign Language Recognition with long short-term memory. In: 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, pp. 2871–2875 (2016)

  128. Yang, X.; Chen, X.; Cao, X.; Wei, S.; Zhang, X.: Chinese Sign Language Recognition based on an optimized tree-structure framework. IEEE J. Biomed. Health Inform. 2(4), 994–1004 (2017)

    Google Scholar 

  129. Mao, C.; Huang, S.; Li, X.; Ye, Z.: Chinese Sign Language Recognition with sequence to sequence learning. In: CCF Chinese Conference on Computer Vision, pp. 180–191. Springer, Singapore (2017)

  130. Huang, S.; Mao, C.; Tao, J.; Ye, Z.: A novel Chinese Sign Language Recognition method based on keyframe-centered clips. IEEE Signal Process. Lett. 25(3), 442–446 (2018)

    Google Scholar 

  131. Liang, Z.-J.; Liao, S.-B.; Hu, B.-Z.: 3D convolutional neural networks for dynamic Sign Language Recognition. Comput. J. 61(11), 1724–1736 (2018)

    Google Scholar 

  132. Huang, J.; Zhou, W.; Zhang, Q.; Li, H.; Li, W.: Video-based Sign Language Recognition without temporal segmentation. In: Thirty-Second AAAI Conference on Artificial Intelligence, pp. 1–10 (2018)

  133. Yuan, T.; et al.: Large scale sign language interpretation. In: 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019), pp. 1–5. IEEE (2019)

  134. Pu, J.; Zhou, W.; Li, H.: Iterative alignment network for continuous Sign Language Recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4165–4174 (2019)

  135. Sidig, A.A.I.; Luqman, H.; Mahmoud, S.A.: Arabic Sign Language Recognition using vision and hand tracking features with HMM. Int. J. Intell. Syst. Technol. Appl. 18(5), 430–447 (2019)

    Google Scholar 

  136. Fatmi, R.; Rashad, S.; Integlia, R.: Comparing ANN, SVM, and HMM based machine learning methods for American Sign Language Recognition using wearable motion sensors. In: IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC), Las Vegas, NV, USA, pp. 0290–0297 (2019)

  137. Sajanraj, T.D.; Beena, M.: Indian Sign Language numeral recognition using region of interest convolutional neural network. In: Second International Conference on Inventive Communication and Computational Technologies (ICICCT), Coimbatore, pp. 636–640 (2018)

  138. Suri, K.; Gupta, R.: Convolutional neural network array for Sign Language Recognition using wearable IMUs. In: 2019 6th International Conference on Signal Processing and Integrated Networks (SPIN), Noida, India, pp. 483–488 (2019)

  139. Soodtoetong, N.; Gedkhaw, E.: The efficiency of Sign Language Recognition using 3D convolutional neural networks. In: 15th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), Chiang Rai, Thailand, pp. 70–73 (2018)

  140. Kumar, E.K.; Kishore, P.V.V.; Sastry, A.S.C.S.; Kumar, M.T.K.; Kumar, D.A.: Training CNNs for 3-D sign language recognition with color texture coded joint angular displacement maps. IEEE Signal Process. Lett. 25(5), 645–649 (2018)

    Google Scholar 

  141. Farooq, U.; Asmat, A.; Rahim, M.S.B.M.; Khan, N.S.; Abid, A.: A comparison of hardware based approaches for sign language gesture recognition systems. In: 2019 International Conference on Innovative Computing (ICIC), Lahore, Pakistan, pp. 1–6 (2019)

  142. Yang, L.; Zhu, Y.; Li, T.: Towards computer-aided Sign Language Recognition technique: a directional review. In: 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chengdu, China, pp. 721–725 (2019)

  143. Kishore, P.V.V.; Prasad, M.V.D.; Prasad, C.R.; Rahul, R.: 4-Camera model for Sign Language Recognition using elliptical fourier descriptors and ANN. In: 2015 International Conference on Signal Processing and Communication Engineering Systems, Guntur, pp. 34–38 (2015)

  144. Dinh, D.; Lee, S.; Kim, T.: Hand number gesture recognition using recognized hand parts in depth images. Multimed. Tools Appl. 75, 1333–1348 (2016)

    Google Scholar 

  145. Xie, M.; Ma, X.: End-to-end residual neural network with data augmentation for Sign Language Recognition. In: 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chengdu, China, pp. 1629–1633 (2019)

  146. Huang, J.; Zhou, W.; Li, H.; Li, W.: Sign Language Recognition using 3D convolutional neural networks. In: 2015 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2015)

  147. Ye, J.; Yao, H.; Jiang, F.: Based on HMM and SVM multilayer architecture classifier for Chinese Sign Language Recognition with large vocabulary. In: Third International Conference on Image and Graphics (ICIG’04), pp. 377–380. IEEE (2004)

  148. Gao, W.; et al.: HandTalker II: a Chinese Sign Language Recognition and synthesis system. In: ICARCV 2004 8th Control, Automation, Robotics and Vision Conference, 2004, vol. 1, pp. 759–764. IEEE (2004)

  149. Abreu, J.G.; Teixeira, J.M.; Figueiredo, L.S.; Teichrieb, V.: Evaluating Sign Language Recognition using the Myo Armband. In: 2016 XVIII Symposium on Virtual and Augmented Reality (SVR), pp. 64–70. IEEE (2016)

  150. Tubaiz, N.; Shanableh, T.; Assaleh, K.: Glove-based continuous Arabic Sign Language Recognition in user-dependent mode. IEEE Trans. Hum. Mach. Syst. 45(4), 526–533 (2015)

    Google Scholar 

  151. Al-Rousan, M.; Assaleh, K.; Tala’a, A.: Video-based signer-independent Arabic Sign Language Recognition using hidden Markov models. Appl. Soft Comput. 9(3), 990–999 (2009)

    Google Scholar 

  152. Mohandes, M.; Aliyu, S.; Deriche, M.: Arabic Sign Language Recognition using the leap motion controller. In: 2014 IEEE 23rd International Symposium on Industrial Electronics (ISIE), pp. 960–965. IEEE (2014)

  153. Plouffe, G.; Cretu, A.-M.: Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Trans. Instrum. Meas. 65(2), 305–316 (2015)

    Google Scholar 

  154. Abhishek, K.S.; Qubeley, L.C.K.; Ho, D.: Glove-based hand gesture recognition sign language translator using capacitive touch sensor. In: 2016 IEEE International Conference on Electron Devices and Solid-State Circuits (EDSSC), Hong Kong, pp. 334–337 (2016)

  155. Raheja, J.; Mishra, A.; Chaudhary, A.: Indian Sign Language Recognition using SVM. Pattern Recognit. Image Anal. 26(2), 434–441 (2016)

    Google Scholar 

  156. Karami, A.; Zanj, B.; Sarkaleh, A.K.: Persian Sign Language (PSL) Recognition using wavelet transform and neural networks. Expert Syst. Appl. 38(3), 2661–2667 (2011)

    Google Scholar 

  157. Li, Z.; Lin, Y.; Elofsson, A.; Yao, Y.: Protein contact map prediction based on ResNet and DenseNet. Biomed. Res. Int. 2020, 1–10 (2020)

    Google Scholar 

  158. Tenbrink, L.; Feldotto, B.; Röhrbein, F.; Knoll, A.: Motion prediction of virtual patterns, human hand motions and a simplified hand manipulation task with hierarchical temporal memory. In: Proceedings IEEE International Conference on Cyborgs and Bionic Systems, pp. 1–10 (2019)

Download references

Acknowledgements

This work was supported from Jiangsu Overseas Visiting Scholar Program for University Prominent Young & Middle-aged Teachers and Presidents of China; The Natural Science Foundation of Jiangsu Higher Education Institutions of China (19KJA310002); Guangxi Key Laboratory of Trusted Software (kx201901); Royal Society International Exchanges Cost Share Award, UK (RP202G0230); Medical Research Council Confidence in Concept Award, UK (MC_PC_17171); Hope Foundation for Cancer Research, UK (RM60G0680); British Heart Foundation Accelerator Award, UK; Fundamental Research Funds for the Central Universities (CDLS-2020-03); and Key Laboratory of Child Development and Learning Science (Southeast University), Ministry of Education.

Author information

Authors and Affiliations

  1. Nanjing Normal University of Special Education, Nanjing, 210038, China

    Xianwei Jiang & Longxiang Yang

  2. Joint Accessibility Key Laboratory, China Disabled Persons’ Federation, Nanjing, 210038, China

    Xianwei Jiang

  3. School of Computer Engineering, KIIT Deemed to University, Bhubaneswar, India

    Suresh Chandra Satapathy

  4. School of Architecture Building and Civil Engineering, Loughborough University, Loughborough, LE11 3TU, UK

    Shui-Hua Wang

  5. School of Mathematics and Actuarial Science, University of Leicester, Leicester, LE1 7RH, UK

    Shui-Hua Wang

  6. School of Informatics, University of Leicester, Leicester, LE1 7RH, UK

    Yu-Dong Zhang

  7. Department of Information Systems, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, 21589, Saudi Arabia

    Yu-Dong Zhang

Authors
  1. Xianwei Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Suresh Chandra Satapathy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Longxiang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shui-Hua Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yu-Dong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Shui-Hua Wang or Yu-Dong Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, X., Satapathy, S.C., Yang, L. et al. A Survey on Artificial Intelligence in Chinese Sign Language Recognition. Arab J Sci Eng 45, 9859–9894 (2020). https://doi.org/10.1007/s13369-020-04758-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-020-04758-2

Keywords

Search

Navigation