Matthew Kyan
Skip Abstract Section
Abstract
This article proposes a novel framework for the real-time capture, assessment, and visualization of ballet dance movements as performed by a student in an instructional, virtual reality (VR) setting. The acquisition of human movement data is facilitated by skeletal joint tracking captured using the popular Microsoft (MS) Kinect camera system, while instruction and performance evaluation are provided in the form of 3D visualizations and feedback through a CAVE virtual environment, in which the student is fully immersed. The proposed framework is based on the unsupervised parsing of ballet dance movement into a structured posture space using the spherical self-organizing map (SSOM). A unique feature descriptor is proposed to more appropriately reflect the subtleties of ballet dance movements, which are represented as gesture trajectories through posture space on the SSOM. This recognition subsystem is used to identify the category of movement the student is attempting when prompted (by a virtual instructor) to perform a particular dance sequence. The dance sequence is then segmented and cross-referenced against a library of gestural components performed by the teacher. This facilitates alignment and score-based assessment of individual movements within the context of the dance sequence. An immersive interface enables the student to review his or her performance from a number of vantage points, each providing a unique perspective and spatial context suggestive of how the student might make improvements in training. An evaluation of the recognition and virtual feedback systems is presented.
- D. Alexiadis, P. Daras, P. Kelly, N. E. O’Connor, T. Boubekeur, and M. B. Moussa. 2011. Evaluating a dancer's performance using Kinect--based skeleton tracking. In ACM Multimedia. 659--662. Google Scholar
Digital Library
- O. Arikan and E. Forsyth. 2002. Interactive motion generation from examples. ACM Transactions on Graphics 21, 3 (2002), 483--490. Google ScholarDigital Library
- C. W. Armstrong and S. J. Hoffman. 1979. Effects of teaching experience, knowledge of performer competence, and knowledge of performance outcome on performance error identification. Research Quarterly 50, 3 (1979), 318--327.Google Scholar
- J. Barbic, A. Safonova, J. Y. Pan, C. Faloutsos, J. K. Hodgins, and N. S. Pollard. 2004. Segmenting motion capture data into distinct behaviors. In Proceedings of Graphics Interface. 185--194. Google ScholarDigital Library
- M. A. Barnes, D. Krasnow, S. J. Tupling, and M. Thomas. 2000. Knee rotation in classical dancers during the grand plié. Medical Problems of Performing Artists 15, 4 (2000), 140--147.Google Scholar
- D. A. Becker and A. Pentland. 1996. Using a virtual environment to teach cancer patients T’ai Chi, relaxation, and self-imagery. In Proceedings of the International Conference on Automatic Face and Gesture Recognition.Google Scholar
- M. Bertucco and P. Cesari. 2010. Does movement planning follow Fitts’ law? Scaling anticipatory postural adjustments with movement speed and accuracy. Neuroscience 171, 1 (2010), 205--213.Google ScholarCross Ref
- A. F. Bobick and J. W. Davis. 2001. The recognition of human movement using temporal templates. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 3 (2001), 257--267. Google ScholarDigital Library
- D. Brennan and M. M. Van Hulle. 2007. Comparison of flat SOM with spherical SOM: A case study. In The Self-Organizing Maps and the Development -- From Medicine and Biology to the Sociological Field. 31--41. Springer, Tokyo, Japan, (2007).Google Scholar
- S. Bronner and S. Ojofeitimi. 2006. Gender and limb differences in healthy elite dancers: Passé kinematics. Journal of Motor Behavior 38, 1 (2006), 71--79.Google ScholarCross Ref
- S. Bronner and S. Ojofeitimi. 2011. Pelvis and hip three-dimensional kinematics in grand battement movements. Journal of Dance Medicine and Science 15, 1 (2011), 23--30.Google Scholar
- J. C. Chan, H. Leung, J. K. Tang, and T. Komura. 2011. A virtual reality dance training system using motion capture technology. IEEE Transactions on Learning Technologies 4, 2 (2011), 187--195. Google ScholarDigital Library
- P. T. Chua, R. Crivella, B. Daly, N. Hu, R. Schaaf, T. Ventura, J. Camill, J. Hodgins, and R. Pausch. 2003. Tai chi: Training for physical tasks in virtual environments. In Proceedings of IEEE Virtual Reality. 87--94. Google ScholarDigital Library
- A. Clay, N. Couture, and L. Nigay. 2009. Towards an architecture model for emotion recognition in interactive systems: Application to a ballet dance show. In Proceedings of the World Conference on Innovative Virtual Reality. 19--24.Google Scholar
- A. Couillandres, P. Lewton-Brain, and P. Portero. 2008. Exploring the effects of kinesiological awareness and mental imagery on movement intention in the performance of demi-plie. Journal of Dance Medicine and Science 12, 3 (2008), 91--98.Google Scholar
- J. L. Deckert, S. M. Barry, and T. M. Welsh. 2007. Analysis of pelvic alignment in university ballet majors. Journal of Dance Medicine and Science 11, 4 (2007), 110--117.Google Scholar
- L. Deng, H. Leung, N. Gu, and Y. Yang. 2011. Real time mocap dance recognition for an interactive dancing game. Computer Animation and Virtual Worlds 22 (2011), 229--237. Google ScholarDigital Library
- K. A. Ericsson, R. T. Krampe, and C. Tesch-Roemer. 1993. The role of deliberate practice in the acquisition of expert performance. Psychological Review 100, 3 (1993), 363--406.Google ScholarCross Ref
- N. Gamboian, S. J. Chatfield, and M. H. Woollacott. 2000. Further effects of somatic training on pelvic tilt and lumbar lordosis alignment during quiet stance and dynamic dance movement. Journal of Dance Medicine and Science 4, 3 (2000), 90--98.Google Scholar
- E. Golomer, R. M. Gravenhorst, and Y. Toussaint. 2009. Influence of vision and motor imagery styles on equilibrium control during whole-body rotations. Somatosensory and Motor Research 26, 4 (2009), 105--110.Google ScholarCross Ref
- A. O. Gonsales and M. Kyan. 2012. Trajectory analysis on spherical self-organizing maps with application to gesture recognition. In Proceedings of the 9th International Workshop (WSOM’12). 125--134.Google Scholar
- K. Hachimura, H. Kato, and H. Tamura. 2004. A prototype dance training support system with motion capture and mixed reality technologies. In Proceedings of the IEEE International Workshop on Robot and Human Interactive Communication. 217--222.Google Scholar
- E. Ho, J. Chan, T. Komura, and H. Leung. 2013. Interactive partner control in close interactions for real-time applications. ACM Transactions on Multimedia Computing, Communications, and Applications 9, 3 (2013), 21. Google ScholarDigital Library
- K. M. Holt, T. M. Welsh, and J. Speights. 2011. A within-subject analysis of the effects of remote cueing on pelvic alignment in dancers. Journal of Dance Medicine and Science 15, 1 (2011), 15--22.Google Scholar
- A. Imura, Y. Iino, and T. Kojima. 2008. Biomechanics of the continuity and speed change during one revolution of the fouette turn. Human Movement Science 27, 6 (2008), 903--913.Google ScholarCross Ref
- A. Imura, Y. Iino, and T. Kojima. 2010. Kinematic and kinetic analysis of the fouette turn in classical ballet. Journal of Applied Biomechanics 26, 4 (2010), 484--492.Google ScholarCross Ref
- G. Kassing and M. J. Danielle. 1998. Teaching Beginning Ballet Technique. Human Kinetics.Google Scholar
- E. Kavakli, S. Bakogianni, A. Damianakis, M. Loumou, and D. Tsatsos. 2004. Traditional dance and e-learning: The WEBDANCE learning environment. In Proceedings of the International Conference on Theory and Applications of Mathematics and Informatics. 272--281.Google Scholar
- M. Kawashima, A. Shimada, and R.-I. Taniguchi. 2009. Early recognition of gesture patterns using sparse code of self-organising map. In Advances in Self-Organizing Maps. Springer, Berlin, 116--123. Google ScholarDigital Library
- E. Keogh. 2002. Exact indexing of dynamic time warping. In Proceedings of the International Conference on Very Large Data Bases. 406--417. Google ScholarDigital Library
- E. Keogh, T. Palpanas, V. B. Zordan, E. Gunopulos, and M. Cardle. 2004. Indexing large human-motion databases. In Proceedings of the International Conference on Very Large Data Bases. 780--791. Google ScholarDigital Library
- T. Komura, B. Lam, R. W. H. Lau, and H. Leung. 2006. e-Learning martial arts. Lecture Notes in Computer Science 4181 (2006), 239--248. Google ScholarDigital Library
- D. H. Krasnow and S. J. Chatfield. 1997. Imagery and conditioning practices for dancers. Dance Research Journal 29, 1 (1997), 43--64.Google ScholarCross Ref
- K. Kulig, A. L. Fietzer, and J. M. Popovich. 2011. Ground reaction forces and knee mechanics in the weight acceptance phase of a dance leap take-off and landing. Journal of Sports Sciences 29, 2 (2011), 125--131.Google ScholarCross Ref
- C. F. Lin, F. C. Su, and H. W. Wu. 2005. Ankle biomechanics of ballet dancers in relevé en pointé dance. Research in Sports Medicine 13, 1 (2005), 23--35.Google ScholarCross Ref
- F. Lv, R. Nevatia, and M. W. Lee. 2005. 3D human action recognition using spatio-temporal motion templates. In Computer Vision in Human-Computer Interaction. Springer, Berlin, 120--130. Google ScholarDigital Library
- N. Masso, A. Germain, F. Rey, L. Costa, D. Romero, and S. Guitart. 2004. Study of muscle activity during releve in first position and sixth positions. Journal of Dance Medicine and Science 8, 4 (2004), 101--107.Google Scholar
- L. Mayers, S. Bronner, S. Agraharasamakulam, and S. Ojofeitimi. 2010. Lower extremity kinetics in tap dance. Journal of Dance Medicine and Science 14, 1 (2010), 3--10.Google Scholar
- J. D. McAuley and M. R. Jones. 2003. Modeling effects of rhythmic context on perceived duration: A comparison of interval and entrainment approaches to short-interval timing. Journal of Experimental Psychology: Human Perception and Performance 29 (2003), 1102--1125.Google ScholarCross Ref
- M. Naemura and M. Suzuki. 2006. A method for estimating dance action based on motion analysis. In Computer Vision and Graphics. Springer Netherlands, 695--702.Google Scholar
- J. G. Noverre. 1760. Letters on Dancing and Ballet, translated by Cyril W. Beaumont. London 1830, Reproduced, New York Dance Horizons.Google Scholar
- M. Raptis, D. Kirovski, and H. Hoppe. 2011. Real-time classification of dance gestures from skeleton animation. In Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 147--156. Google ScholarDigital Library
- A. P. Sangole and A. Leontitis. 2006. Spherical self-organizing feature map: An introductory review. International Journal of Bifurcation and Chaos 16, 11 (2006), 3195--3206.Google ScholarCross Ref
- C. Schuldt, I. Laptev, and B. Caputo. 2004. Recognizing human actions: A local SVM approach. In Proceedings of the IEEE International Conference on Pattern Recognition. 32--36. Google ScholarDigital Library
- G. Shan. 2005. Comparison of repetitive movements between ballet dancers and martial artists: Risk assessment of muscle overuse injuries and prevention strategies. Research in Sports Medicine 13, 1 (2005), 63--76.Google ScholarCross Ref
- J. M. Shippen and B. May. 2010. Calculation of muscle loading and joint contact forces during the rock step in Irish dance. Journal of Dance Medicine and Science 14, 1 (2010), 11--18.Google Scholar
- R. W. Simmons. 2005. Sensory organization determinants of postural stability in trained ballet dancers. International Journal of Neuroscience 115, 1 (2005), 87--97.Google ScholarCross Ref
- J. K. T. Tang, H. Leung, T. Komura, and H. P. H. Shum. 2008. Emulating human perception of motion similarity. Computer Animation and Virtual Worlds 19, 3--4 (2008), 211--221. Google ScholarDigital Library
- M. Uejou, H.-H. Huang, J.-H. Lee, and K. Kawagoe. 2011. Toward a conversational virtual instructor of ballroom dance. In Intelligent Virtual Agents. Springer, Berlin, 477--478. Google ScholarDigital Library
- R. E. Ward. 2012. Biomechanical Perspectives on Classical Ballet Technique and Implications for Teaching Practice. PhD Thesis, University of New South Wales.Google Scholar
- M. Wilson, J.-H. Ryu, and Y.-H. Kwon. 2007. Contribution of the pelvis to gesture leg range of motion in a complex ballet movement grand rond de jambe en l'air en dehors. Journal of Dance Medicine and Science 11, 4 (2007), 118--123.Google Scholar
- U. Yang and G. J. Kim. 2002. Implementation and evaluation of “just follow me”: An immersive, VR-based, motion-training system. Presence: Teleoperators and Virtual Environments 11, 3 (2002), 304--323. Google ScholarDigital Library
- Y. Yang, H. Leung, L. Yue, and L. Deng. 2012. Automatic dance lesson generation. IEEE Transactions on Learning Technologies 3, 5 (2012), 191--198. Google ScholarDigital Library
- Y. Yang, H. Leung, L. Yue, and L. Deng. 2013. Generating a two-phase lesson for guiding beginners to learn basic dance movements. Computers and Education 61, 1 (2013), 1--20. Google ScholarDigital Library
Index Terms
- An Approach to Ballet Dance Training through MS Kinect and Visualization in a CAVE Virtual Reality Environment
Recommendations
An Advanced Computational Intelligence System for Training of Ballet Dance in a Cave Virtual Reality Environment
ISM '14: Proceedings of the 2014 IEEE International Symposium on MultimediaThis paper presents a computer-based system for assessment and training of ballet dance in a CAVE virtual reality environment. The system utilizes Kinect sensor to capture student's dance and extracts features from skeleton joints. This system depends ...
CAVE and Fishtank Virtual-Reality Displays: A Qualitative and Quantitative Comparison
We present the results from a qualitative and quantitative user study comparing fishtank virtual-reality (VR) and CAVE displays. The results of the qualitative study show that users preferred the fishtank VR display to the CAVE system for our scientific ...
Thinning trainer based on forest-growth model, virtual reality and computer-aided virtual environment
Immersive virtual reality is applied in many human activities. This virtual reality can be used as a training tool for thinning operations in forests. The aim of this study is to describe the complex solution of Thinning Trainer, that we developed. This ...
Comments