Elsevier

Gait & Posture

Volume 81, Supplement 1, September 2020, Pages 261-262
Gait & Posture

Deep learning for automated pose estimation of infants at home from smart phone videos

https://doi.org/10.1016/j.gaitpost.2020.08.026Get rights and content

Introduction

Timely access to health services are essential for infants at risk of adverse neurodevelopmental outcomes, such as cerebral palsy. To address this need the Baby Moves App was developed [1]. This application allows parents/caregivers to record their baby’s movements at specified timepoints and securely upload to clinicians. The General Movements Assessment (GMA), a good predictor of neurodevelopment, is performed using these at-home videos, without a visit to specialised clinic or hospitals. However, access to GMA trained clinicians remains a limiting factor. Advances in video-based pose estimation techniques allow the mapping of infant body position and limb movement over time using machine learning and offer an alternative to identify infants at risk using at-home videos.

Section snippets

Research Question

Can machine learning techniques reliably estimate the pose of infants from at-home videos recorded on a smart phone?

Methods

Videos from 510 infants (n = 236 preterm; 274 term) were acquired at 12(n = 199) and 14(n = 311) weeks post-term through the BabyMoves app between April-2016 and March-2017. Each video was 3 minutes in length (5142 ± 457 frames). We used Deep LabCut toolbox [2] to train a deep learning model to detect 18 key-points on each video frame (Fig. 1). For model training, we selected 100 videos (50 at 12weeks and 14weeks post-term). From each video, 5 frames were randomly selected, and 18 key-points

Results

Accurate automated labelling was achieved for all key-points using deep learning. Cross-validated root mean squared error was small at 4.5pixels. On average 90% of key-points were identified, accounting for both missing labels and occlusion of key-points due to body movements, particularly of the heel and wrist. Twenty-one videos were excluded due to <70% of keypoints being labelled. This was related to occlusion due to clothing or camera angle.

Discussion

Deep learning models allow the estimation of pose from at-home videos of infant movement. The use of smart phone videos taken by parents/caregivers carries additional data quality and processing challenges compared to controlled lab-based videos. Further research will focus on whether these key-point trajectories can predict the GMA.

References (2)

  • A.J. Spittle et al.

    The Baby Moves prospective cohort study protocol: Using a smartphone application with the General Movements Assessment to predict neurodevelopmental outcomes at age 2 years for extremely preterm or extremely low birthweight infants

    BMJ Open

    (2016)
  • A. Mathis et al.

    DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

    Nat Neurosci

    (2018)

Cited by (0)

View full text