Skip to main content
SpringerLink
Log in
Book cover

Gesture Recognition pp 203–230Cite as

Finding Recurrent Patterns from Continuous Sign Language Sentences for Automated Extraction of Signs

  • Chapter
  • First Online:

Part of the book series: The Springer Series on Challenges in Machine Learning ((SSCML))

Abstract

We present a probabilistic framework to automatically learn models of recurring signs from multiple sign language video sequences containing the vocabulary of interest. We extract the parts of the signs that are present in most occurrences of the sign in context and are robust to the variations produced by adjacent signs. Each sentence video is first transformed into a multidimensional time series representation, capturing the motion and shape aspects of the sign. Skin color blobs are extracted from frames of color video sequences, and a probabilistic relational distribution is formed for each frame using the contour and edge pixels from the skin blobs. Each sentence is represented as a trajectory in a low dimensional space called the space of relational distributions. Given these time series trajectories, we extract signemes from multiple sentences concurrently using iterated conditional modes (ICM). We show results by learning single signs from a collection of sentences with one common pervading sign, multiple signs from a collection of sentences with more than one common sign, and single signs from a mixed collection of sentences. The extracted signemes demonstrate that our approach is robust to some extent to the variations produced within a sign due to different contexts. We also show results whereby these learned sign models are used for spotting signs in test sequences.

Editors: Isabelle Guyon

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  • O. Al-Jarrah, A. Halawani, Recognition of gestures in Arabic Sign Language using neuro-fuzzy systems. Artif. Intell. 133, 117–138 (2001)

    Article  MATH  Google Scholar 

  • V. Athitsos, J. Alon, S. Sclaroff, G. Kollios, Boostmap: a method for efficient approximate similarity rankings, in IEEE Conference on Computer Vision and Pattern Recognition, 2004, pp. 268–275

    Google Scholar 

  • T. Bailey, C. Elkan, Unsupervised learning of multiple motifs in biopolymers using expectation maximization. Mach. Learn. 21, 51–80 (1995)

    Google Scholar 

  • B. Bauer, H. Hienz, Relevant features for video-based continuous sign language recognition, in Automatic Face and Gesture Recognition, 2000, pp. 440–445

    Google Scholar 

  • B. Bauer, K.F. Kraiss, Video-based sign recognition using self-organizing subunits. Int. Conf. Pattern Recognit. 2, 434–437 (2002)

    Article  Google Scholar 

  • J. Besag, On the statistical analysis of dirty pictures. J. R. Stat. Soc. 48, 259–302 (1986)

    MathSciNet  MATH  Google Scholar 

  • R. Bowden, D. Windridge, T. Kadir, A. Zisserman, M. Brady, A linguistic feature vector for the visual interpretation of sign language. Eur. Conf. Comput. Vis. 1, 390–401 (2004)

    MATH  Google Scholar 

  • P. Buehler, A. Zisserman, M. Everingham, Learning sign language by watching tv (using weakly aligned subtitles), in IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp. 2961–2968

    Google Scholar 

  • G. Casella, E.I. George, Explaining the Gibbs sampler. Am. Stat. 46, 167–174 (1992)

    MathSciNet  Google Scholar 

  • S. Chib, E. Greenberg, Understanding the Metropolis-Hastings algorithm. Am. Stat. 49, 327–335 (1995)

    Google Scholar 

  • B. Chiu, E. Keogh, S. Lonardi, Probabilistic discovery of time series motifs, in ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2003, pp. 493–498

    Google Scholar 

  • H. Cooper, R. Bowden, Learning signs from subtitles: a weakly supervised approach to sign language recognition, in IEEE Conference on Computer Vision and Pattern Recognition, June 2009, pp. 2568–2574

    Google Scholar 

  • Y. Cui, J. Weng, Appearance-based hand sign recognition from intensity image sequences. Comput. Vis. Image Underst. 78, 157–176 (2000)

    Article  Google Scholar 

  • A. Denton, Kernel-density-based clustering of time series subsequences using a continuous random-walk noise model, in International Conference on Data Mining, 2005

    Google Scholar 

  • K.G. Derpanis, R.R. Wildes, J.K. Tsotsos, Hand gesture recognition within a linguistics-based framework, in European Conference on Computer Vision, 2004, pp. 282–296

    Google Scholar 

  • F. Duchene, C. Garbay, V. Rialle, Learning recurrent behaviors from heterogeneous multivariate time-series. Artif. Intell. Med. 39(1), 25–47 (2007)

    Article  Google Scholar 

  • G. Fang, X. Gao, W. Gao, Y. Chen, A novel approach to automatically extracting basic units from Chinese Sign Language. Int. Conf. Pattern Recognit. 4, 454–457 (2004)

    Google Scholar 

  • A. Farhadi, D.A. Forsyth, R. White, Transfer learning in sign language, in Computer Vision and, Pattern Recognition, 2007, pp. 1–8

    Google Scholar 

  • W.R. Gilks, S. Richardson, D.J. Spiegelhalter, Markov chain Monte Carlo in practice. Chapman and Hall, 1998

    Google Scholar 

  • J. Han, G. Awad, A. Sutherland, Modelling and segmenting subunits for sign language recognition based on hand motion analysis. Pattern Recognit. Lett. 30(6), 623–633 (2009)

    Article  Google Scholar 

  • C.E. Lawrence, S.F. Altschul, M.S. Boguski, J.S. Liu, A.F. Neuwald, J.C. Wootton, Detecting subtle sequence signals: a Gibbs sampling strategy for multiple alignment. Science 262, 208–214 (1993)

    Article  Google Scholar 

  • S.K. Liddell, R.E. Johnson, American Sign Language: the phonological base. Sign Lang. Stud. 195–277 (1989)

    Google Scholar 

  • J. Ma, W. Gao, C. Wang, J. Wu, A continuous Chinese Sign Language recognition system, in International Conference on Automatic Face and Gesture Recognition, 2000, pp. 428–433

    Google Scholar 

  • D. Minnen, C.L. Isbell, I. Essa, T. Starner, Discovering multivariate motifs using subsequence density estimation and greedy mixture learning, in Conference on Artificial Intelligence, 2007

    Google Scholar 

  • S. Nayak, S. Sarkar, B. Loeding, Unsupervised modeling of signs embedded in continuous sentences, in IEEE Workshop on Vision for Human-Computer Interaction, 2005

    Google Scholar 

  • S. Nayak, S. Sarkar, B. Loeding, Automated extraction of signs from continuous sign language sentences using iterated conditional modes, in IEEE Conference on Computer Vision and Pattern Recognition, June 2009a, pp. 2583–2590

    Google Scholar 

  • S. Nayak, S. Sarkar, B. Loeding, Distribution-based dimensionality reduction applied to articulated motion recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(5), 795–810 (2009b)

    Article  Google Scholar 

  • T. Oates, PERUSE: an unsupervised algorithm for finding recurring patterns in time series, in International Conference on Data Mining, 2002, pp. 330–337

    Google Scholar 

  • S.C.W. Ong, S. Ranganath, Automatic sign language analysis: a survey and the future beyond lexical meaning. IEEE Trans. Pattern Anal. Mach. Intell. 27, 873–891 (2005)

    Article  Google Scholar 

  • M. Oszust, M. Wysocki, Determining subunits for sign language recognition by evolutionary cluster-based segmentation of time series, in Artifical Intelligence and Soft Computing, vol. 6114 of Lecture Notes in Computer Science (Springer, Berlin/Heidelberg, 2010), pp. 189–196

    Google Scholar 

  • P.A. Pevzner, S.H. Sze, Combinatorial approaches to finding subtle signals in DNA sequences, in International Conference on Intelligent Systems for Molecular Biology, 2000, pp. 269–278

    Google Scholar 

  • S.L. Phung, A. Bouzerdoum, D. Chai, Skin segmentation using color pixel classification: analysis and comparison. IEEE Trans. Pattern Anal. Mach. Intell. 27, 148–154 (2005)

    Article  Google Scholar 

  • I. Rigoutsos, A. Floratos, Combinatorial pattern discovery in biological sequences: the Teiresias algorithm. Bioinformatics 14, 55–67 (1998)

    Article  Google Scholar 

  • C.P. Robert, G. Casella, Monte Carlo Statistical Methods (Springer, New York, 2004)

    Book  MATH  Google Scholar 

  • T. Starner, A. Pentland, Real-time American Sign Language recognition from video using hidden Markov Models. Comput. Imaging Vis. 9, 227–244 (1997)

    Article  Google Scholar 

  • T. Starner, J. Weaver, A. Pentland, Real-time American Sign Language recognition using desk and wearable computer based video. IEEE Trans. Pattern Anal. Mach. Intell. 20(12), 1371–1375 (1998)

    Article  Google Scholar 

  • Y. Tanaka, K. Iwamoto, K. Uehara, Discovery of time-series motif from multidimensional data based on MDL principle. Mach. Learn. 58(2–3), 269–300 (2005)

    Article  MATH  Google Scholar 

  • I.R. Vega, S. Sarkar, Statistical motion model based on the change of feature relationships: human gait-based recognition. IEEE Trans. Pattern Anal. Mach. Intell. 25, 1323–1328 (2003)

    Article  Google Scholar 

  • C. Vogler, D. Metaxas, Parallel hidden Markov models for American sign language recognition. Int. Conf. Comput. Vis. 1, 116–122 (1999)

    Google Scholar 

  • C. Vogler, D. Metaxas, A framework of recognizing the simultaneous aspects of American Sign Language. Comput. Vis. Image Underst. 81, 358–384 (2001)

    Article  MATH  Google Scholar 

  • C. Wang, W. Gao, S. Shan, An approach based on phonemes to large vocabulary Chinese Sign Language recognition, in International Conference on Automatic Face and Gesture Recognition, 2002, pp. 393–398

    Google Scholar 

  • Q. Wang, X. Chen, L.G. Zhang, C. Wang, W. Gao, Viewpoint invariant sign language recognition. Comput. Vis. Image Underst. 108, 87–97 (2007)

    Article  Google Scholar 

  • M.H. Yang, N. Ahuja, M. Tabb, Extraction of 2d motion trajectories and its application to hand gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell. 24, 1061–1074 (2002)

    Article  Google Scholar 

  • R. Yang, S. Sarkar, B. Loeding, Handling movement epenthesis and hand segmentation ambiguities in continuous sign language recognition using nested dynamic programming. IEEE Trans. Pattern Anal. Mach. Intell. 32(3), 462–477 (2010)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by funds from University of South Florida’s College of Engineering Interdisciplinary Scholarship Program and the National Science Foundation under ITR Grant IIS 0312993.

Author information

Authors and Affiliations

  1. Taaz Inc., 4250 Executive Square, Suite 420, La Jolla, CA, 92037, USA

    Sunita Nayak

  2. Department of Computer Science & Engineering, University of South Florida, Tampa, FL, 33620, USA

    Kester Duncan & Sudeep Sarkar

  3. Department of Special Education, University of South Florida, Lakeland, FL, 33803, USA

    Barbara Loeding

Authors
  1. Sunita Nayak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kester Duncan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sudeep Sarkar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Barbara Loeding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sunita Nayak .

Editor information

Editors and Affiliations

  1. University of Barcelona, Barcelona, Spain

    Sergio Escalera

  2. ChaLearn, Berkeley, California, USA

    Isabelle Guyon

  3. Department of Computer Science and Engineering, University of Texas at Arlington, Arlington, Texas, USA

    Vassilis Athitsos

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Nayak, S., Duncan, K., Sarkar, S., Loeding, B. (2017). Finding Recurrent Patterns from Continuous Sign Language Sentences for Automated Extraction of Signs. In: Escalera, S., Guyon, I., Athitsos, V. (eds) Gesture Recognition. The Springer Series on Challenges in Machine Learning. Springer, Cham. https://doi.org/10.1007/978-3-319-57021-1_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-57021-1_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-57020-4

  • Online ISBN: 978-3-319-57021-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation