Skip to main content
SpringerLink
Log in

Learning to Gesticulate by Observation Using a Deep Generative Approach

  • Conference paper
  • First Online:
Social Robotics (ICSR 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11876))

Included in the following conference series:

Abstract

The goal of the system presented in this paper is to develop a natural talking gesture generation behavior for a humanoid robot, by feeding a Generative Adversarial Network (GAN) with human talking gestures recorded by a Kinect. A direct kinematic approach is used to translate from human poses to robot joint positions. The provided videos show that the robot is able to use a wide variety of gestures, offering a non-dreary, natural expression level.

This work has been partially supported by the Basque Government (IT900-16 and Elkartek 2018/00114) and the Spanish Ministry of Economy and Competitiveness MINECO/FEDER (RTI 2018-093337-B-100, MINECO/FEDER, EU). We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.ald.softbankrobotics.com/en/robots/pepper.

  2. 2.

    http://wiki.ros.org/naoqi_driver.

  3. 3.

    http://doc.aldebaran.com/2-5/naoqi/index.html.

  4. 4.

    http://www.ros.org.

  5. 5.

    https://www.vicon.com/.

  6. 6.

    http://wiki.ros.org/skeleton_markers.

  7. 7.

    http://doc.aldebaran.com/2-8/family/pepper_technical/joints_pep.html.

  8. 8.

    https://www.youtube.com/watch?v=iW1566ozbdg.

  9. 9.

    https://www.youtube.com/watch?v=1It_Y_AEnts.

  10. 10.

    https://wrnch.ai/.

References

  1. Alibeigi, M., Rabiee, S., Ahmadabadi, M.N.: Inverse kinematics based human mimicking system using skeletal tracking technology. J. Intell. Robotic Syst. 85(1), 27–45 (2017)

    Article  Google Scholar 

  2. Augello, A., Cipolla, E., Infantino, I., Manfrè, A., Pilato, G., Vella, F.: Creative robot dance with variational encoder. CoRR abs/1707.01489 (2017)

    Google Scholar 

  3. Beck, A., Yumak, Z., Magnenat-Thalmann, N.: Body movements generation for virtual characters and social robots. In: Judee, K.B., Nadia, M.-T., Maja, P., Alessandro, V. (eds.) Social Signal Processing, pp. 273–286. Cambridge University Press, Cambridge (2017)

    Chapter  Google Scholar 

  4. Breazeal, C.: Designing sociable robots. In: Intelligent Robotics and Autonomous Agents. MIT Press, Cambridge (2004)

    Google Scholar 

  5. Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: OpenPose: realtime multi-person 2D pose estimation using part affinity fields. arXiv preprint arXiv:1812.08008 (2018)

  6. Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: CVPR (2017)

    Google Scholar 

  7. Castillo, E., Gutiérrez, J.M., Hadi, A.S.: Learning Bayesian networks. In: Expert Systems and Probabilistic Network Models. Monographs in computer science. Springer-Verlag, New York (1997). https://doi.org/10.1007/978-1-4612-2270-5_11

    Chapter  Google Scholar 

  8. Everitt, B., Hand, D.: Finite Mixture Distributions. Chapman and Hall, New York (1981)

    Book  Google Scholar 

  9. Fadli, H., Machbub, C., Hidayat, E.: Human gesture imitation on NAO humanoid robot using kinect based on inverse kinematics method. In: International Conference on Advanced Mechatronics, Intelligent Manufacture, and Industrial Automation (ICAMIMIA). IEEE (2015)

    Google Scholar 

  10. Goodfellow, I.: NIPS tutorial: generative adversarial networks. ArXiv e-prints, December 2017

    Google Scholar 

  11. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  12. Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., Alahi, A.: Social GAN: socially acceptable trajectories with generative adversarial networks. CoRR abs/1803.10892 (2018). http://arxiv.org/abs/1803.10892

  13. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  14. Kwon, J., Park, F.C.: Using hidden markov models to generate natural humanoid movement. In: International Conference on Intelligent Robots and Systems (IROS). IEEE/RSJ (2006)

    Google Scholar 

  15. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  16. MacCormick, J.: How does the kinect work?. http://pages.cs.wisc.edu/ahmad/kinect.pdf. Accessed 3 June 2019

  17. Manfrè, A., Infantino, I., Vella, F., Gaglio, S.: An automatic system for humanoid dance creation. Biologically Inspired Cogn. Architect. 15, 1–9 (2016)

    Article  Google Scholar 

  18. McNeill, D.: Hand and Mind: What Gestures Reveal About Thought. University of Chicago press (1992)

    Google Scholar 

  19. Mehta, D., et al.: VNect: real-time 3D human pose estimation with a single RGB camera. ACM Trans. Graph. 36(4), 44:1–44:14 (2017)

    Article  Google Scholar 

  20. Mukherjee, S., Paramkusam, D., Dwivedy, S.K.: Inverse kinematics of a NAO humanoid robot using Kinect to track and imitate human motion. In: International Conference on Robotics, Automation, Control and Embedded Systems (RACE). IEEE (2015)

    Google Scholar 

  21. Okamoto, T., Shiratori, T., Kudoh, S., Nakaoka, S., Ikeuchi, K.: Toward a dancing robot with listening capability: keypose-based integration of lower-, middle-, and upper-body motions for varying music tempos. IEEE Trans. Robot. 30, 771–778 (2014). https://doi.org/10.1109/TRO.2014.2300212

    Article  Google Scholar 

  22. Poubel, L.P.: Whole-body online human motion imitation by a humanoid robot using task specification. Master’s thesis, Ecole Centrale de Nantes-Warsaw University of Technology (2013)

    Google Scholar 

  23. Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE. 77, 257–286 (1989)

    Article  Google Scholar 

  24. Rodriguez, I., Astigarraga, A., Ruiz, T., Lazkano, E.: Singing minstrel robots, a means for improving social behaviors. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 2902–2907 (2016)

    Google Scholar 

  25. Rodriguez, I., Astigarraga, A., Jauregi, E., Ruiz, T., Lazkano, E.: Humanizing NAO robot teleoperation using ROS. In: International Conference on Humanoid Robots (Humanoids) (2014)

    Google Scholar 

  26. Rodriguez, I., Martínez-Otzeta, J.M., Irigoien, I., Lazkano, E.: Spontaneous talking gestures using generative adversarial networks. Robot. Auton. Syst. 114, 57–65 (2019)

    Article  Google Scholar 

  27. Schubert, T., Eggensperger, K., Gkogkidis, A., Hutter, F., Ball, T., Burgard, W.: Automatic bone parameter estimation for skeleton tracking in optical motion capture. In: International Conference on Robotics and Automation (ICRA). IEEE (2016)

    Google Scholar 

  28. Tanwani, A.K.: Generative models for learning robot manipulation. Ph.D. thesis, École Polytechnique Fédéral de Laussane (EPFL) (2018)

    Google Scholar 

  29. Tits, M., Tilmanne, J., Dutoit, T.: Robust and automatic motion-capture data recovery using soft skeleton constraints and model averaging. PLOS One 13(7), 1–21 (2018)

    Article  Google Scholar 

  30. Wold, S., Esbensen, K., Geladi, P.: Principal component analysis. Chemometr. Intell. Lab. Syst. 2(1–3), 37–52 (1987)

    Article  Google Scholar 

  31. Zhang, Z., Niu, Y., Yan, Z., Lin, S.: Real-time whole-body imitation by humanoid robots and task-oriented teleoperation using an analytical mapping method and quantitative evaluation. Appl. Sci. 8(10), 2005 (2018). https://www.mdpi.com/2076-3417/8/10/2005

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science and Artificial Intelligence, Faculty of Informatics, UPV/EHU, Manuel Lardizabal 1, 20018, Donostia, Spain

    Unai Zabala, Igor Rodriguez, José María Martínez-Otzeta & Elena Lazkano

Authors
  1. Unai Zabala
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Igor Rodriguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. José María Martínez-Otzeta
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Elena Lazkano
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Igor Rodriguez .

Editor information

Editors and Affiliations

  1. Robotics Lab, Universidad Carlos III de Madrid, Leganés, Madrid, Spain

    Miguel A. Salichs

  2. The National University of Singapore, Singapore, Singapore

    Shuzhi Sam Ge

  3. Faculty of Industrial Design, Eindhoven University of Technology, Eindhoven, Noord-Brabant, The Netherlands

    Emilia Ivanova Barakova

  4. Mechanical & Industrial, Qatar University, Doha, Qatar

    John-John Cabibihan

  5. Department of Aerospace Engineering, The Pennsylvania State University, University Park, PA, USA

    Alan R. Wagner

  6. Robotics Lab - Department of Systems Engineering and Automation, Universidad Carlos III de Madrid, Leganés, Madrid, Spain

    Álvaro Castro-González

  7. Wichita State University, Wichita, KS, USA

    Hongsheng He

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zabala, U., Rodriguez, I., Martínez-Otzeta, J.M., Lazkano, E. (2019). Learning to Gesticulate by Observation Using a Deep Generative Approach. In: Salichs, M., et al. Social Robotics. ICSR 2019. Lecture Notes in Computer Science(), vol 11876. Springer, Cham. https://doi.org/10.1007/978-3-030-35888-4_62

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-35888-4_62

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-35887-7

  • Online ISBN: 978-3-030-35888-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation