Abstract
Concurrent augmented feedback has been shown to be less effective for learning simple motor tasks than for complex tasks. However, as mostly artificial tasks have been investigated, transfer of results to tasks in sports and rehabilitation remains unknown. Therefore, in this study, the effect of different concurrent feedback was evaluated in trunk-arm rowing. It was then investigated whether multimodal audiovisual and visuohaptic feedback are more effective for learning than visual feedback only. Naïve subjects (N = 24) trained in three groups on a highly realistic virtual reality-based rowing simulator. In the visual feedback group, the subject’s oar was superimposed to the target oar, which continuously became more transparent when the deviation between the oars decreased. Moreover, a trace of the subject’s trajectory emerged if deviations exceeded a threshold. The audiovisual feedback group trained with oar movement sonification in addition to visual feedback to facilitate learning of the velocity profile. In the visuohaptic group, the oar movement was inhibited by path deviation-dependent braking forces to enhance learning of spatial aspects. All groups significantly decreased the spatial error (tendency in visual group) and velocity error from baseline to the retention tests. Audiovisual feedback fostered learning of the velocity profile significantly more than visuohaptic feedback. The study revealed that well-designed concurrent feedback fosters complex task learning, especially if the advantages of different modalities are exploited. Further studies should analyze the impact of within-feedback design parameters and the transferability of the results to other tasks in sports and rehabilitation.
Similar content being viewed by others
References
Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262
Blandin Y, Toussaint L, Shea CH (2008) Specificity of practice: interaction between concurrent sensory information and terminal feedback. J Exp Psychol Learn Mem Cogn 34(4):994–1000
Braun DA, Mehring C, Wolpert DM (2010) Structure learning in action. Behav Brain Res 206(2):157–165
Brown R, Palmer C (2012) Auditory–motor learning influences auditory memory for music. Mem Cognit 40(4):567–578
Buchanan J, Wang C (2012) Overcoming the guidance effect in motor skill learning: feedback all the time can be beneficial. Exp Brain Res 219(2):305–320
Burke JL, Prewett MS, Gray AA, Yang L, Stilson FRB, Coovert MD, Elliot LR, Redden E (2006) Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis. In: Proceedings of the 8th international conference on multimodal interfaces, New York, NY, USA, pp 108–117
Carson RG, Kelso JAS (2004) Governing coordination: behavioural principles and neural correlates. Exp Brain Res 154(3):267–274
Cesqui B, Aliboni S, Mazzoleni S, Carrozza M, Posteraro F, Micera S (2008) On the use of divergent force fields in robot-mediated neurorehabilitation. In: 2nd IEEE RAS EMBS international conference on biomedical robotics and biomechatronics 2008 BioRob 2008, pp 854–861
Chen X, Agrawal S (2013) Assisting versus repelling force-feedback for learning of a line following task in a wheelchair. IEEE Trans Neural Syst Rehabil Eng 21(6):959–968
Chen JL, Penhune VB, Zatorre RJ (2008) Listening to musical rhythms recruits motor regions of the brain. Cereb Cortex 18(12):2844–2854
Chiviacowsky S, Wulf G (2007) Feedback after good trials enhances learning. Res Q Exerc Sport 78:40–47
Chollet D, Madani M, Micallef JP (1992) Effects of two types of biomechanical bio-feedback on crawl performance. In: MacLaren D, Reilly T, Lees A (eds) Biomechanics and medicine in swimming. E & FN Spon, London, pp 57–62
Effenberg AO (2005) Movement sonification: effects on perception and action. IEEE Multimedia 12(2):53–59
Emken J, Reinkensmeyer DJ (2005) Robot-enhanced motor learning: accelerating internal model formation during locomotion by transient dynamic amplification. IEEE Trans Neural Syst Rehabil Eng 13(1):33–39
Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433
Freides D (1974) Human information processing and sensory modality: cross-modal functions information complexity memory and deficit. Psychol Bull 81(5):284–310
Giese MA, Poggio T (2000) Morphable models for the analysis and synthesis of complex motion patterns. Int J Comput Vis 38(1):59–73
Guadagnoli M, Kohl R (2001) Knowledge of results for motor learning: relationship between error estimation and knowledge of results frequency. J Mot Behav 33(2):217–224
Guadagnoli MA, Lee TD (2004) Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning. J Mot Behav 36(2):212–224
Hale K, Stanney K (2004) Deriving haptic design guidelines from human physiological psychophysical and neurological foundations. IEEE Comput Graph Appl 24(2):33–39
Holden MK (2005) Virtual environments for motor rehabilitation: review. Cyberpsychol Behav 8(3):187–211
Huang F, Patton J (2013) Augmented dynamics and motor exploration as training for stroke. IEEE Trans Biomed Eng 60(3):838–844
Huang H, Ingalls T, Olson L, Ganley K, Rikakis T, He J (2005) Interactive multimodal biofeedback for task-oriented neural rehabilitation. In: 27th annual international conference of the engineering in medicine and biology society 2005 IEEE-EMBS 2005, Shanghai, pp 2547–2550
Hubbard T (2013) Auditory imagery contains more than audition. In: Lacey S, Lawson R (eds) Multisensory imagery. Springer, New York, pp 221–247
Huegel J, O’Malley MK (2010) Progressive haptic and visual guidance for training in a virtual dynamic task. In: haptics symposium 2010 IEEE, pp 343–350
Israel J, Campbell D, Kahn J, Hornby T (2006) Metabolic costs and muscle activity patterns during robotic-and therapist-assisted treadmill walking in individuals with incomplete spinal cord injury. Phys Ther 86(11):1466–1478
Kapur A, Tzanetakis G, Virji-Babul N, Wang G, Cook PR (2005) A framework for sonification of vicon motion capture data. In: Proceedings of the 8th conference on digital audio effects, Madrid, Spain
Kennedy D, Boyle J, Shea C (2013) The role of auditory and visual models in the production of bimanual tapping patterns. Exp Brain Res 224(4):507–518
Keysers C, Gazzola V (2010) Social neuroscience: mirror neurons recorded in humans. Curr Biol 20(8):R353–R354
Kim RS, Seitz AR, Shams L (2008) Benefits of stimulus congruency for multisensory facilitation of visual learning. PLoS One 3(1):e1532
Kovacs AJ, Shea CH (2011) The learning of 90° continuous relative phase with and without lissajous feedback: external and internally generated bimanual coordination. Acta Psychol 136(3):311–320
Krakauer J, Mazzoni P (2011) Human sensorimotor learning: adaptation skill and beyond. Curr Opin Neurobiol 21(4):636–644
Kramer G (1994) Auditory display: sonification audification and auditory interfaces. Addison-Wesley, Reading MA
Krebs HI, Palazzolo JJ, Dipietro L, Ferraro M, Krol J, Rannekleiv K, Volpe BT, Hogan N (2003) Rehabilitation robotics: performance-based progressive robot-assisted therapy. Auton Robots 15(1):7–20
Lahav A, Saltzman E, Schlaug G (2007) Action representation of sound: audiomotor recognition network while listening to newly acquired actions. J Neurosci 27(2):308–314
Liebermann DG, Katz L, Hughes MD, Bartlett RM, McClements J, Franks IM (2002) Advances in the application of information technology to sport performance. J Sports Sci 20(10):755–769
Liu D, Todorov E (2007) Evidence for the flexible sensorimotor strategies predicted by optimal feedback control. J Neurosci 27(35):9354–9368
Liu J, Wrisberg CA (1997) The effect of knowledge of results delay and the subjective estimation of movement form on the acquisition and retention of a motor skill. Res Q Exerc Sport 68(2):145–151
Lüttgen J, Heuer H (2012a) The influence of haptic guidance on the production of spatio-temporal patterns. Hum Mov Sci 31(3):519–528
Lüttgen J, Heuer H (2012b) Robotic guidance benefits the learning of dynamic but not of spatial movement characteristics. Exp Brain Res 222(1–2):1–9
Lüttgen J, Heuer H (2013) The influence of robotic guidance on different types of motor timing. J Mot Behav 45(3):249–258
Marchal-Crespo L, Reinkensmeyer DJ (2008a) Effect of robotic guidance on motor learning of a timing task. In: 2nd IEEE RAS EMBS international conference on biomedical robotics and biomechatronics 2008 BioRob 2008, pp 199–204
Marchal-Crespo L, Reinkensmeyer DJ (2008b) Haptic guidance can enhance motor learning of a steering task. J Mot Behav 40(6):545–557
Marchal-Crespo L, Reinkensmeyer DJ (2009) Review of control strategies for robotic movement training after neurologic injury. J Neuroeng Rehabil 6(1):20
Marchal-Crespo L, Furumasu J, Reinkensmeyer DJ (2010) A robotic wheelchair trainer: design overview and a feasibility study. J Neuroeng Rehabil 7(1):40–51
Marchal-Crespo L, Raai M, Rauter G, Wolf P, Riener R (2013) The effect of haptic guidance and visual feedback on learning a complex tennis task. Exp Brain Res 231(3):277–291
Marschall F, Bund A, Wiemeyer J (2007) Does frequent feedback really degrade learning? A meta analysis. E-Journal Bewegung und Training 1:75–86
Milot MH, Marchal-Crespo L, Green CS, Cramer SC, Reinkensmeyer DJ (2010) Comparison of error-amplification and haptic-guidance training techniques for learning of a timing-based motor task by healthy individuals. Exp Brain Res 201(2):119–131
Minogue J, Jones MG (2006) Haptics in Education: exploring an untapped sensory modality. Rev Educ Res 76(3):3–17
Patton JL, Stoykov M, Kovic M, Mussa-Ivaldi F (2006) Evaluation of robotic training forces that either enhance or reduce error in chronic hemiparetic stroke survivors. Exp Brain Res 168(3):368–383
Patton JL, Wei YJ, Bajaj P, Scheidt RA (2013) Visuomotor learning enhanced by augmenting instantaneous trajectory error feedback during reaching. PLoS One 8(1):e46466
Proteau L (1992) On the specificity of learning and the role of visual information for movement control. In: Proteau L, Elliott D (eds) Vision and motor control, vol 85. North-Holland, Amsterdam, pp 67–103
Rauter G, von Zitzewitz J, Duschau-Wicke A, Vallery H, Riener R (2010) A tendon based parallel robot applied to motor learning in sports. In: 3rd IEEE RAS and EMBS international conference on biomedical robotics and biomechatronics (BioRob) 2010, Tokyo, Japan, pp 82–87
Rauter G, Sigrist R, Marchal-Crespo L, Vallery H, Riener R, Wolf P (2011) Assistance or challenge? Filling a gap in user-cooperative control. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 3068–3073
Rauter G, Sigrist R, Koch C, Crivelli F, van Raai M, Riener R, Wolf P (2013) Transfer of complex skill learning from virtual to real rowing. PLoS One 8(12):1–18
Reinkensmeyer DJ, Akoner O, Ferris D, Gordon K (2009) Slacking by the human motor system: computational models and implications for robotic orthoses. In: Engineering in medicine and biology society 2009 EMBC 2009 annual international conference of the IEEE, pp 2129–2132
Ribeiro DC, Sole G, Abbott JH, Milosavljevic S (2011) Extrinsic feedback and management of low back pain: a critical review of the literature. Man Ther 16(3):231–239
Robin C, Toussaint L, Blandin Y, Proteau L (2005) Specificity of learning in a video-aiming task: modifying the salience of dynamic visual cues. J Mot Behav 37(5):367–376
Ronsse R, Puttemans V, Coxon JP, Goble DJ, Wagemans J, Wenderoth N, Swinnen SP (2011) Motor learning with augmented feedback: modality-dependent behavioral and neural consequences. Cereb Cortex 21(6):1283–1294
Salmoni S (1984) Knowledge of results and motor learning A review and critical reappraisal. Psychol Bull 95(3):355–386
Schaffert N, Mattes K, Effenberg AO (2011) An investigation of online acoustic information for elite rowers in on-water training conditions. J Hum Sport Exerc 6(2):392–405
Schmidt RA (1991) Frequent augmented feedback can degrade learning: evidence and interpretations. Tutor Motor Neurosci 62:59–75
Schmidt RA, Wrisberg C (2008) Motor learning and performance: a situation-based learning approach. Human Kinetics, Champaign, IL
Schmidt RA, Wulf G (1997) Continuous concurrent feedback degrades skill learning: implications for training and simulation. Hum Factors 39(4):509–525
Schmidt RA, Young DE, Swinnen S, Shapiro DC (1989) Summary knowledge of results for skill acquisition: support for the guidance hypothesis. J Exp Psychol Learn Mem Cogn 15(2):352–359
Schmitz G, Mohammadi B, Hammer A, Heldmann M, Samii A, Munte T, Effenberg A (2013) Observation of sonified movements engages a basal ganglia frontocortical network. BMC Neurosci 14(1):1–11
Secoli R, Milot M, Rosati G, Reinkensmeyer DJ (2011) Effect of visual distraction and auditory feedback on patient effort during robot-assisted movement training after stroke. J Neuroeng Rehabil 8(1):1–10
Seitz AR, Dinse HR (2007) A common framework for perceptual learning. Curr Opin Neurobiol 17(2):148–153
Seitz AR, Kim R, Shams R (2006) Sound facilitates visual learning. Curr Biol 16(14):1422–1427
Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12(11):411–417
Sigrist R, Schellenberg J, Rauter G, Broggi S, Riener R, Wolf P (2011) Visual and auditory augmented concurrent feedback in a complex motor task. Presence Teleop Virt 20(1):15–32
Sigrist R, Rauter G, Riener R, Wolf P (2013a) Augmented visual auditory haptic and multimodal feedback in motor learning: a review. Psychon Bull Rev 20:21–53
Sigrist R, Rauter G, Riener R, Wolf P (2013b) Terminal feedback outperforms concurrent visual auditory and haptic feedback in learning a complex rowing-type task. J Mot Behav 45(6):455–472
Snodgrass SJ, Rivett DA, Robertson VJ, Stojanovski E (2010) Real-time feedback improves accuracy of manually applied forces during cervical spine mobilization. Man Ther 15:19–25
Swinnen SP, Schmidt RA, Nicholson DE, Shapiro DC (1990) Information feedback for skill acquisition: instantaneous knowledge of results degrades learning. J Exp Psychol Learn Mem Cogn 16(4):706–716
Swinnen SP, Lee TD, Verschueren S, Serrien DJ, Bogaerds H (1997) Interlimb coordination: learning and transfer under different feedback conditions. Hum Mov Sci 16(6):749–785
Thoroughman K, Shadmehr R (2000) Learning of action through adaptive combination of motor primitives. Nature 407(6805):742
Timmermans AAA, Seelen HAM, Willmann RD, Kingma H (2009) Technology-assisted training of arm-hand skills in stroke: concepts on reacquisition of motor control and therapist guidelines for rehabilitation technology design. J Neuroeng Rehabil 6:1
Todorov E (2004) Optimality principles in sensorimotor control. Nat Neurosci 7:907–915
Todorov E, Jordan MI (2002) Optimal feedback control as a theory of motor coordination. Nat Neurosci 5:1226–1235
Todorov E, Shadmehr R, Bizzi E (1997) Augmented feedback presented in a virtual environment accelerates learning of a difficult motor task. J Mot Behav 29(2):147–158
van Beers RJ (2009) Motor learning is optimally tuned to the properties of motor noise. Neuron 63(3):406–417
van Beers RJ, Sittig AC, Gon JJ (1999) Integration of proprioceptive and visual position-information: an experimentally supported model. J Neurophysiol 81(3):1355
van der Linden DW, Cauraugh JH, Greene TA (1993) The effect of frequency of kinetic feedback on learning an isometric force production task in nondisabled subjects. Phys Ther 73(2):79–87
Vlachos M, Hadjieleftheriou M, Gunopulos D, Keogh E (2003) Indexing multi-dimensional time-series with support for multiple distance measures. In: proceedings of the ninth acm sigkdd international conference on knowledge discovery and data mining KDD’03, New York, NY, USA, pp 216–225
von Zitzewitz J, Wolf P, Novakovic V, Wellner M, Rauter G, Brunschweiler A, Riener R (2008) Real-time rowing simulator with multimodal feedback. Sports Technol 1(6):257–266
Wei K, Körding K (2009) Relevance of error: what drives motor adaptation? J Neurophysiol 101(2):655–664
Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88(3):638–667
Wickens CD (2002) Multiple resources and performance prediction. Theor Issues Ergon Sci 3(2):159–177
Winstein CJ (1991) Knowledge of results and motor learning—implications for physical therapy. Phys Ther 71(2):140–149
Winstein CJ, Pohl PS, Cardinale C, Green A, Scholtz L, Waters CS (1996) Learning a partial-weight-bearing skill: effectiveness of two forms of feedback. Phys Ther 76(9):985–993
Wishart LR, Lee TD, Cunningham SJ, Murdoch JE (2002) Age-related differences and the role of augmented visual feedback in learning a bimanual coordination pattern. Acta Psychol (Amst) 110(2–3):247–263
Wolpert D, Flanagan J (2010) Motor learning. Curr Biol 20(11):R467–R472
Wolpert DM, Diedrichsen J, Flanagan JR (2011) Principles of sensorimotor learning. Nat Rev Neurosci 12:739–749
Wulf G (2007) Self-controlled practice enhances motor learning: implications for physiotherapy. Physiotherapy 93(2):96–101
Wulf G, Shea CH (2002) Principles derived from the study of simple skills do not generalize to complex skill learning. Psychon Bull Rev 9(2):185–211
Wulf G, Shea CH, Matschiner S (1998) Frequent feedback enhances complex motor skill learning. J Mot Behav 30(2):180–192
Wulf G, Hörger M, Shea CH (1999) Benefits of blocked over serial feedback on complex motor skill learning. J Mot Behav 31(1):95–103
Zatorre RJ, Chen JL, Penhune VB (2007) When the brain plays music: auditory–motor interactions in music perception and production. Nat Rev Neurosci 8(7):547–558
Acknowledgments
We want to thank Prof. Nicole Wenderoth for her enriching comments. We thank Alessandro Rotta, Pascal Wespe, and Marco Bader for their indispensable technical contributions to the design and setup of the simulator. Moreover, thanks go to Michael Herold-Nadig for his support on technical and safety issues and Mark van Raai for designing the visual feedback and rowing scenario. We want to thank Stefan Seiterle, Luca Lörtscher, Florian Kübler, and Nicolas Gerig for their help in conducting the measurements, and the subjects for participating. We thank Andrew Pennycott for proofreading the manuscript. This work was supported by ETH Zurich, the SNF-Grant “Impact of Different Feedback Modalities on Complex Skill Learning,” CR22I2 135101/1, and the SNF-Grant “Acceleration of complex motor learning by skill level-dependent feedback design and automatic selection”, CR23I2_152817.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sigrist, R., Rauter, G., Marchal-Crespo, L. et al. Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning. Exp Brain Res 233, 909–925 (2015). https://doi.org/10.1007/s00221-014-4167-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-014-4167-7