Skip to main content
SpringerLink
Log in

Sign Language Recognition Using Sub-units

  • Chapter
  • First Online:
Gesture Recognition

Part of the book series: The Springer Series on Challenges in Machine Learning ((SSCML))

Abstract

This chapter discusses sign language recognition using linguistic sub-units. It presents three types of sub-units for consideration; those learnt from appearance data as well as those inferred from both 2D or 3D tracking data. These sub-units are then combined using a sign level classifier; here, two options are presented. The first uses Markov Models to encode the temporal changes between sub-units. The second makes use of Sequential Pattern Boosting to apply discriminative feature selection at the same time as encoding temporal information. This approach is more robust to noise and performs well in signer independent tests, improving results from the 54% achieved by the Markov Chains to 76%.

Editors: Isabelle Guyon and Vassilis Athitsos.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Note that conversion between the two forms is possible. However while HamNoSys is usually presented as a font for linguistic use, SiGML is more suited to automatic processing.

References

  • Y. Amit, D. Geman, Shape quantization and recognition with randomized trees. Neural Comput. 9, 1545–1588 (1997)

    Article  Google Scholar 

  • L. Breiman, Random forests. Mach. Learn 45, 5–32 (2001)

    Article  MATH  Google Scholar 

  • British Deaf Association, Dictionary of British Sign Language/English, (Faber and Faber Ltd, London, 1992)

    Google Scholar 

  • P. Buehler, M. Everingham, A. Zisserman, Learning sign language by watching TV (using weakly aligned subtitles), in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–26 June 2009, pp. 2961–2968

    Google Scholar 

  • H. Cooper, R. Bowden, Large lexicon detection of sign language, in Proceedings of the IEEE International Conference on Computer Vision: Workshop Human Computer Interaction, Rio de Janario, Brazil, 16–19 October 2007, pp. 88–97 doi: 10.1007/978-3-540-75773-3_10

  • H. Cooper, R. Bowden, Sign language recognition using linguistically derived sub-units, in Proceedings of the Language Resources and Evaluation Conference Workshop on the Representation and Processing of Sign Languages : Corpora and Sign Languages Technologies, Valetta, Malta, 17–23 May 2010

    Google Scholar 

  • R. Elliott, J. Glauert, J. Kennaway, K. Parsons, D5-2: SiGML Definition. ViSiCAST Project working document, 2001

    Google Scholar 

  • H. Ershaed, I. Al-Alali, N. Khasawneh, M. Fraiwan, An arabic sign language computer interface using the xbox kinect, in Annual Undergraduate Research Conference on Applied Computing, May 2011

    Google Scholar 

  • Y. Freund, R.E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, in Proceedings of the European Conference on Computational Learning Theory, Springer, Barcelona, 13–15 March 1995, pp. 23–37. ISBN 3-540-59119-2

    Google Scholar 

  • J.W. Han, G. Awad, A. Sutherland, Modelling and segmenting subunits for sign language recognition based on hand motion analysis. Pattern Recognit. Lett. 30(6), 623–633 (2009)

    Article  Google Scholar 

  • T. Hanke, C. Schmaling. Sign Language Notation System, (Institute of German Sign Language and Communication of the Deaf, Hamburg, Germany, January 2004), http://www.sign-lang.uni-hamburg.de/projects/hamnosys.html

  • M.K. Hu, Visual pattern recognition by moment invariants. IRE Transactions on Information Theory, IT-8, pp. 179–187, February 1962

    Google Scholar 

  • T. Kadir, R. Bowden, E.J. Ong, A. Zisserman, Minimal training, large lexicon, unconstrained sign language recognition, in Proceedings of the BMVA British Machine Vision Conference, vol. 2, Kingston, UK, 7–9 September 2004, pp. 939–948

    Google Scholar 

  • S. Kim, M.B. Waldron, Adaptation of self organizing network for ASL recognition, in Proceedings of the Annual International Conference of the IEEE Engineering in Engineering in Medicine and Biology Society, San Diego, California, USA, 28–31 October 1993, pp. 254–254

    Google Scholar 

  • W.W. Kong, S. Ranganath, Automatic hand trajectory segmentation and phoneme transcription for sign language, in Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, Amsterdam, The Netherlands, 17–19, September 2008, pp. 1–6. doi: 10.1109/AFGR.2008.4813462

  • S.K. Liddell, R.E. Johnson, American sign language: the phonological base. Sign Lang. Stud. 64, 195–278 (1989)

    Article  Google Scholar 

  • K. Lyons, H. Brashear, T.L. Westeyn, J.S. Kim, T. Starner, Gart: The gesture and activity recognition toolkit, in Proceedings of the International Conference HCI, July 2007, pp. 718–727

    Google Scholar 

  • E.J. Ong, R. Bowden, Learning sequential patterns for lipreading, in Proceedings of the BMVA British Machine Vision Conference (Dundee, UK, August 29–September 10 2011)

    Google Scholar 

  • OpenNI User Guide. OpenNI organization, November 2010. Accessed 20 Apr 2011 18:15

    Google Scholar 

  • V. Pitsikalis, S. Theodorakis, C. Vogler, P. Maragos, Advances in phonetics-based sub-unit modeling for transcription alignment and sign language recognition, in Proceedings of the International Conference IEEE Computer Society Conference on Computer Vision and Pattern RecognitionWorkshop : Gesture Recognition, Colorado Springs, CO, USA, 21–23 June 2011

    Google Scholar 

  • Prime Sensor™NITE 1.3 Algorithms notes. PrimeSense Inc., 2010. Accesesed 20 Apr 2011 18:15

    Google Scholar 

  • A. Roussos, S. Theodorakis, V. Pitsikalis, P. Maragos, Hand tracking and affine shape-appearance handshape sub-units in continuous sign language recognition, in Proceedings of the International Conference European Conference on Computer VisionWorkshop : SGA, Heraklion, Crete, 5–11 September 2010

    Google Scholar 

  • J.E. Shoup, Phonological aspects of speech recognition. In Wayne A. Lea, editor, Trends in Speech Recognition, Prentice-Hall, Englewood Cliffs, NJ, 1980, pp. 125–138

    Google Scholar 

  • T. Starner, A. Pentland, Real-time American sign language recognition from video using hidden Markov models. Comput. Imaging Vis. 9, 227–244 (1997)

    Article  Google Scholar 

  • W.C. Stokoe, Sign language structure: an outline of the visual communication systems of the American Deaf. Stud. Linguist. 8, 3–37 (1960)

    Google Scholar 

  • P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple features, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, Kauai, HI, USA, 2001, pp. 511–518

    Google Scholar 

  • C. Vogler, D. Metaxas, Adapting hidden Markov models for ASL recognition by using three-dimensional computer vision methods, in Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, vol. 1, Orlando, FL, USA, 12–15 October 1997, pp. 156–161

    Google Scholar 

  • C. Vogler, D. Metaxas, Parallel hidden Markov models for American sign language recognition, in Proceedings of the IEEE International Conference on Computer Vision, vol. 1, Corfu, Greece, 21–24 September 1999, pp. 116–122

    Google Scholar 

  • M.B. Waldron, S. Kim, Increasing manual sign recognition vocabulary through relabelling, in Proceedings of the IEEE International Conference on Neural Networks IEEE World Congress on Computational Intelligence, vol. 5, Orlando, Florida, USA, June 27–July 2 1994, pp. 2885–2889. doi: 10.1109/ICNN.1994.374689

  • M.B. Waldron, S. Kim, Isolated ASL sign recognition system for deaf persons. IEEE Trans. Rehab. Eng. 3(3), 261–271 (1995). doi:10.1109/86.413199

    Article  Google Scholar 

  • M.B. Waldron, D. Simon, Parsing method for signed telecommunication, in Proceedings of the Annual International Conference of the IEEE Engineering in Engineering in Medicine and Biology Society: Images of the Twenty-First Century, vol. 6, Seattle, Washington, USA, November 1989, pp. 1798–1799. doi: 10.1109/IEMBS.1989.96461

  • H. Wassner. kinect + reseau de neurone = reconnaissance de gestes, http://tinyurl.com/5wbteug, May 2011

  • P. Yin, T. Starner, H. Hamilton, I. Essa, J.M. Rehg, Learning the basic units in American sign language using discriminative segmental feature selection, in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Taipei, Taiwan, 19–24 April 2009, pp. 4757–4760. doi: 10.1109/ICASSP.2009.4960694

  • Z. Zafrulla, H. Brashear, P. Presti, H. Hamilton, T. Starner, Copycat-center for accessible technology in sign. http://tinyurl.com/3tksn6s, December 2010, http://www.youtube.com/watch?v=qFH5rSzmgFE&feature=related

  • Z. Zafrulla, H. Brashear, T. Starner, H. Hamilton, P. Presti, American sign language recognition with the kinect, in Proceedings of the 13th International Conference on Multimodal Interfaces, ICMI ’11 (New York, NY, USA, 2011. ACM. ISBN 978-1-4503-0641-6), pp. 279–286. doi: 10.1145/2070481.2070532

Download references

Acknowledgements

The research leading to these results has received funding from the European Community’s Seventh Framework Programme (FP7/2007–2013) under grant agreement number 231135 Dicta-Sign. The Dicta-Sign data sets used and additional SL resources are available via http://www.sign-lang.uni-hamburg.de/dicta-sign/portal/.

Author information

Authors and Affiliations

  1. Centre for Vision Speech and Signal Processing, University of Surrey, Guildford, GU2 9PY, UK

    Helen Cooper, Eng-Jon Ong, Nicolas Pugeault & Richard Bowden

Authors
  1. Helen Cooper
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eng-Jon Ong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicolas Pugeault
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard Bowden
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard Bowden .

Editor information

Editors and Affiliations

  1. University of Barcelona, Barcelona, Spain

    Sergio Escalera

  2. ChaLearn, Berkeley, California, USA

    Isabelle Guyon

  3. Department of Computer Science and Engineering, University of Texas at Arlington, Arlington, Texas, USA

    Vassilis Athitsos

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Cooper, H., Ong, EJ., Pugeault, N., Bowden, R. (2017). Sign Language Recognition Using Sub-units. In: Escalera, S., Guyon, I., Athitsos, V. (eds) Gesture Recognition. The Springer Series on Challenges in Machine Learning. Springer, Cham. https://doi.org/10.1007/978-3-319-57021-1_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-57021-1_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-57020-4

  • Online ISBN: 978-3-319-57021-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation