Introduction

Recent estimates indicate that approximately 5.5 million people in the United States use a wheelchair to facilitate performance of daily mobility [1]. For wheelchair users, transfers, which can be performed from a wheelchair to a bed, chair, bathtub, car or vice-versa, are one of the most important and common activities performed in a daily basis. Fliess-Douer et al. estimated that wheelchair users with spinal cord injury (SCI) transfer between 15 to 20 times per day [2]. In the context of these high frequencies, improper transfer techniques may place wheelchair users at a high risk of upper extremity overuse injuries [3, 4]. Improper transfer techniques may also lead to increased fear of falling (FOF) [5]. FOF has been associated with a higher risk of falling, activity curtailment, and reduced independence to perform daily living activities [6, 7]. Therefore, an objective evaluation of the quality of transfer techniques may help clinicians to develop targeted interventions and guidelines on transfers to avoid upper extremity injuries, reduce risk of falls, and improve general quality of life.

The Transfer Assessment Instrument (TAI) is a valid and reliable outcome measure that has been developed to evaluate transfer quality among wheelchair users [8]. The assessment was originally developed in 2011 to assess performance of transfers in a clinical setting [9]. The importance of the TAI in clinical studies has also been demonstrated [10, 11]. For example, Hogaboom et al. [11] found that individuals with higher TAI scores, indicating higher transfer quality, demonstrated decreased loading on the upper extremity and fewer signs of shoulder and wrist pathology [11].

In 2020, a self-assessment, questionnaire version of the TAI (TAI-Q) was developed to increase the accessibility of the outcome measure in non-clinical settings and provide an opportunity for wheelchair users to rate the quality of their own transfers [12]. In addition, the measurement properties of the TAI were examined to determine whether a clinician could use the TAI remotely to assess transfer quality [13]. Results indicate that the TAI demonstrated good to excellent reliability when used to assess transfer quality in a laboratory setting by remote investigators [13].

Remote assessment of transfer quality can help to continue the provision of health care to wheelchair users when access to in-person care is limited [14, 15]. Such barriers to in-person care can include, but are not limited to, difficulties with transportation, living in rural areas, and financial burdens [16, 17]. Hoenig et al. found that community dwelling wheelchair users who experienced barriers visiting locations outside of their home reported less participation in medical related activities [18]. In addition, during the lockdown of the recent COVID-19 pandemic, wheelchair users with SCI were advised to stay home due to underlying health conditions. Consequently, the importance of home-based telehealth and telerehabilitation became increasingly evident [19].

To address these barriers and facilitate the assessment of transfer quality through telehealth, it is essential to investigate the reliability of assessing transfer quality remotely in a home environment. Reliability refers to the degree to which a measurement is free of measurement error [20]. A reliable assessment tool that can remotely evaluate the quality of a transfer in a home environment will be an important step to increase access to essential healthcare services to a population that faces many barriers accessing care. Therefore, the purpose of this study was to evaluate the interrater and intrarater reliability of a remote assessment of transfer quality in a home environment using the TAI among wheelchair users with SCI. Specifically, the interrater reliability included the comparisons of the live remote score with the TAI-Q and the average score of two asynchronous raters. The intrarater reliability compared the live remote score with the asynchronous score of the same rater. We hypothesized that the reliability of home-based total TAI score would meet at least moderate levels (intraclass coefficient correlation - ICC > 0.5). This hypothesis was based on the findings from previous studies on the reliability of remote in-clinic evaluations of the TAI and TAI-Q [12, 13].

Methods

Participants

This study is part of another study that aimed to investigate factors associated with falls among wheelchair users with SCI [21]. A convenience sample of participants was recruited remotely from the community during the COVID-19 pandemic between January and July 2021. Volunteers were invited to participate in the study if they met the following inclusion criteria: (1) older than 18 years old; (2) self-report use of a wheelchair as a main form of mobility (>40 h/wk); (3) motor complete SCI classified as American Spinal Injury Association Impairment Scale (AIS) A, or B, or motor incomplete injury AIS C who are full-time wheelchair users; (4) able to communicate with the research team through smartphone or laptop video conferencing software; (5) self-report ability to independently transfer to/from a wheelchair; and (6) ability to read and understand English. Participants were excluded if they presented with any additional medical conditions that might affect their ability to perform the tests. The procedures of the study were reviewed and approved by the Office for the Protection of Research Subjects at the University of Illinois at Urbana-Champaign and the University of Pittsburgh.

Study protocol

All study procedures were conducted remotely. Following screening for eligibility criteria, all participants provided web-based informed consent via the Research Electronic Data Capture (REDCap) survey platform. Then, participants completed a demographic survey also through REDCap. After completion of the online survey, participants met with a research team member over a video call using Zoom (San Jose, CA). They were all joined by a family member, care partner, or a friend to help with participant’s safety as well as camera adjustment.

At the start of the call, participants were asked to show the surface, such as a sofa, bed, or bench, where they would transfer to from their wheelchair. According to the position of the bed, sofa, or bench, general instructions were given to the participants on how to position their video camera for the remote assessment to maximize the rater’s view of the transfer and the environment. Next, participants were asked to position themselves to transfer from their wheelchair to the target surface. A paper ruler and a paper goniometer were sent to the participants through the mail before the assessment day (See Appendix A). With guidance from the rater on the placement of the paper ruler, the participant or their assistant measured the distance from the front corner of their wheelchair to the surface to which they planned to transfer to (item #1 on the TAI). Similarly, the paper goniometer was used by the participant or their assistant to measure the angle between their wheelchair and the surface they planned to transfer to (item #2 on the TAI). Participants were asked to read the angle and distance aloud.

Next, participants were asked to transfer as they normally would in their daily routines. Transfer boards were permitted, and use was scored according to the TAI scoring instructions. If the live rater could not observe an assessment item directly, the participant was asked to provide feedback. For example, if the rater could not observe the foot placement of the participant based on the camera position, the participant was asked to report the location of their feet during the transfer.

One researcher (rater 1) scored the TAI remotely during the live study visit (assessment #1). After the participant transferred to the target surface, a link to the TAI self-assessment questionnaire (TAI-Q) was sent to the participants through REDCap to self-assess the quality of their transfer. Participants could ask questions about the items of the TAI-Q and further clarifications were provided by the remote live rater. Video and audio of the assessments were recorded through the video conference software. Rater 1, who completed the live remote TAI assessment (assessment #1), reviewed the video of the live session approximately four weeks later and evaluated the transfers again using the TAI (assessment #2) to assess intrarater reliability. Two additional clinicians (raters 2 & 3) asynchronously reviewed the recorded video and completed the TAI to assess interrater reliability. The scores of raters 2 and 3 were averaged together (assessment #3). For assessment #3, the raters were able to review the videos of the participant’s transfer as many times as desired. The video assessments were completed at the rater’s convenience, spread over several sessions. Figure 1 shows a summary of the timing of the assessments.

Fig. 1: Summary of the assessments for the home-based remote TAI reliability.
figure 1

Assessment #1: Live remoteassessment from rater 1 and self-assessment by study participants. Assessment #2: Asynchronous rating from rater 1.Assessment #3: Asynchronous ratings from raters 2 and 3.

Full size image

Transfer assessment

The TAI 4.0 is an objective measure of transfer quality with established in-person reliability (ICC = 0.550–0.850) and validity [8]. The concurrent validity (ICC = 0.554–0.740) and reliability (ICC = 0.627–0.705) of the TAI-Q [12], a self-assessment version of the TAI 4.0 in which item scores are not shown, as well as the reliability (ICC = 0.687–0.962) of remote assessment of the TAI in a controlled environment have also been established [13]. The TAI 4.0 is an 18-item instrument comprised of 3 phases: wheelchair set-up (items 1–6), body set-up (items 7–12), and flight/landing (items 13–16). The last 2 items (items 17 and 18) relate to usage of assistive technology (transfer board). The full description and scoring of the TAI is available elsewhere [8, 13]. Briefly, each item receives a score of 1 or 0 indicating high or low quality, respectively. Partial credit (0.5) or not applicable answer options are allowed for some items. Not applicable responses are not included in the total score. All item scores are summed together, multiplied by 10, and averaged, resulting in a score varying from 0 to 10 points [13]. Subscores are calculated in a similar manner for the 3 phases of a transfer.

$${{{\mathrm{TAI}}}}\,{{{\mathrm{score}}}} = \frac{{Sum\,of\,Items\,Scores \ast 10}}{{No\,Applicable\,Items}}$$

Statistical analysis

Data were found to be normally distributed using the Shapiro–Wilk test. Descriptive statistics (mean and SD) were calculated for TAI total and subscores (wheelchair set-up, body set-up, and flight/landing). ICCs were used to assess interrater reliability: [1] assessment #1 (live, rater 1) vs assessment #3 (asynchronous, average raters 2 & 3), [2] assessment #1 vs TAI-Q, and [3] assessment #2 (asynchronous, rater 1) and assessment #3 (asynchronous, average raters 2 & 3). ICCs were also used to assess intrarater reliability between assessment #1 (live, rater 1) vs assessment #2 (asynchronous, rater 1). ICCs were classified as excellent (ICC > 0.9), good (ICC = 0.75–0.9), moderate (ICC = 0.5–0.75), and poor (ICC < 0.5)[22]. Paired sample t-tests with a Holm-Bonferroni correction for multiple comparisons were conducted to examine differences in TAI scores between: [1] assessment #1 (live, rater 1) and assessment #2 (asynchronous rater 1), [2] assessment #1 (live, rater 1) and assessment #3 (asynchronous, average raters 2 & 3), [3] assessment #1 (live, rater 1) and TAI-Q (self-assessment), and [4] assessment #2 (asynchronous, rater 1) and assessment #3 (asynchronous, average raters 2 & 3). Levels of agreement between the TAI evaluations were assessed using the Bland–Altman plots. Good agreement level was indicated by an even spread of points within the 95% limits of agreement and a mean difference close to zero. Linear regression analyses were conducted to assess the relationship between bias and the magnitude of measurements considering the mean of two comparisons as the independent variable and the difference between both comparisons as the dependent variable.

Standard error measurements (SEMs) were calculated using the equation: SEM = \(SD\sqrt{\left(\,\right.}1 - ICC\left. \right)\)where SD indicates the standard deviation of the data set and ICC is the interrater reliability coefficient [23]. The minimal detectable change (MDC) was estimated using the equation: MDC = 1.96*SEM\(\sqrt{2}\) [23]. All data analysis was performed using IBM-SPSS Statistics for Macintosh version 25 (SPSS Inc., Chicago, IL, USA).

Results

Participants and raters

A total of 18 full-time manual wheelchair users with SCI were assessed. Participants’ mean age was 41.1 ± 14.2 years and most participants were male (66.7%). Time since injury ranged from 1 to 53 years. Details of participants’ demographics are presented in Table 1.

Table 1 Participants demographics presented as mean (SD) for continuous variables and count (percentage) for gender.
Full size table

Rater 1 was a physical therapist with 7 years of experience working with individuals with SCI and approximately 1 year of experience using the TAI. The video raters 2 and 3 included physical therapists with 5–12 years of experience providing transfer training to wheelchair users and using the TAI.

Reliability

Table 2 depicts ICCs for interrater and intrarater reliability for total and subscores of the remote home-based TAI. Moderate to good interrater reliability was found for TAI total score. Regarding TAI subscores, good, moderate, and poor interrater reliability were found for wheelchair setup, body setup, and flight/landing, respectively. Bland–Altman plots indicate good interrater agreements with only one outlier within the 95% CI (Fig. 2a, b). The means for the interrater agreements between live rater 1 and asynchronous raters 2 and 3 and between live rater 1 and TAI-Q were higher than 0 (mean differences = 0.5 and 0.9, respectively) though not statistically significant, p > 0.05.

Table 2 ICCs for interrater and intrarater reliability of the remote home-based TAI.
Full size table
Fig. 2: Bland–Altman plot for the home-based remote TAI reliability.
figure 2

a Home-based interrater reliability (Assessment #1 – Assessment #3); b home-based interrater reliability (Assessment #1 – TAI-Q); and c home-based intrarater reliability (Assessment #1 – Assessment #2).

Full size image

Good intrarater reliability was also found for remote home-based TAI total scores. Regarding TAI subscores, wheelchair setup and body setup showed good intrarater reliability while flight/landing showed moderate intrarater reliability. Bland–Altman plot indicate a good intrarater agreement with no outlier within the 95% CI (Fig. 2c).

Overall, the linear regression analyses indicate no significant trends in proportional error or error related to the measurement of error in the reliability analyses. The means of differences were closer to zero in the intrarater reliability analysis compared to the interrater reliability indicating less systematic bias in intrarater reliability than interrater reliability.

TAI scores

For the total TAI score, assessment #1 (live, rater 1) score was significantly higher compared to assessment #3 (asynchronous, average raters 2 & 3) and TAI-Q score (Table 3). Also, assessment #2 (asynchronous, rater 1) was significantly higher than assessment #3 (asynchronous, average raters 2 & 3) for total TAI score. Upon examination of sub-scores, Assessment #1 (live, rater 1) was significantly higher than the TAI-Q for body setup subscores. Also, Assessment #2 (asynchronous rater 1) score was significantly higher than Assessment #1 (live, rater 1) for flight/landing subscores. The SEM and MDC for the remote home-based TAI total score range from 0.38 to 0.79 and 1.04 to 2.20, respectively. The SEM and MDC for the TAI subscores range from 0.28 to 1.03 and 0.78 to 2.85, respectively (Table 3).

Table 3 Differences between average video raters, remote rater, and self-assessment of the TAI.
Full size table

Discussion

This study aimed to evaluate the reliability of a remote assessment of transfer quality in a home setting using the TAI and TAI-Q among wheelchair users with SCI. Findings indicate moderate to good interrater and intrarater reliability of the total TAI score (ICC = 0.57–0.90). Moderate to good interrater and intrarater reliability were also found for all subscores of the TAI (ICC = 0.60–0.94) except for the interrater reliability of the flight/landing which was poor (ICC = 0.20). Our results highlight the potential of assessing transfer quality within the client/patient’s home environment. Such assessments are important to the advancement of telehealth and telerehabilitation to continue serving individuals who may not have access to in-person health care for a variety of reasons including, but not limited to, outbreaks of infections (i.e., COVID-19 pandemic), long travel distances to clinics, or transportation difficulties.

Several studies have analyzed the assessment of functional movement remotely including transfer quality [13], balance [24], and wheelchair skills [25] among wheelchair users. Worobey et al. reported good to excellent reliability of the remote TAI assessments (ICCs = 0.687–0.962) in a laboratory-based assessment that was videotaped [13]. Similarly, Kirby et al. reported that remote video assessment of wheelchair skills by occupational therapists, able-bodied individuals, and wheelchair users presented with good to excellent reliability (ICC = 0.904–0.968) in a laboratory-based environment [25]. Abou et al. indicated that remote, home-based, balance assessment among wheelchair users is feasible and presented with good to excellent concurrent reliability (ICCs = 0.880–0.982) [24]. Our analysis of remote, home-based, transfer quality assessment, using the TAI, is comparable to the other outcome measures of functional movement among wheelchair users. The usefulness of remote assessments to decrease barriers to care heightens the importance of remote functional movement assessments in rehabilitation and demonstrates the necessity for reliable outcome measures to use for remote assessments.

Remote, live scoring (assessment #1) resulted in significantly higher TAI scores compared to asynchronous scoring by different raters (assessment #3). Viewing the transfer asynchronously allowed the raters to view the assessments as many times as needed and pause the videos to better visualize an item. This more detailed viewing may have resulted in a more accurate rating of the quality of the transfer. Remote, live scoring also resulted in significantly higher TAI total scores and body setup subscore compared to the self-assessment scores. The results indicate that the live rater may have overestimated participants transfer quality. It is also arguable that participants tended to underestimate the quality of their transfer, reporting more deficient items when compared to the rating of a clinician. Similar findings have been reported in a previous study where the authors also found lower self-assessment scores when compared to the TAI score of a clinician with 13 years of experience [12]. The differences in the TAI scores indicate that the best approach to evaluate home based wheelchair transfer quality might be in addition to scoring live, to also record videos of participants’ transfers and review them when possible. Indeed, the validation process of the TAI-Q included participants reviewing videos of themselves completing a transfer as many times as possible [12]. It is also critical to provide end-users with detailed instructions when a self-assessment is being performed.

Differences were also observed between TAI subscores across assessments. For flight/landing subscores, the asynchronous video assessment performed by rater 1 (assessment #2) was found to be higher than the remote, live scoring of the same rater (assessment #1). In addition, the self-assessment flight/landing subscore was found to be higher than the average asynchronous subscores from raters 2 and 3. The inconsistencies in the ratings of the flight/landing subscores support that this phase is the most challenging phase of the TAI to score remotely. Indeed, our study also found a poor interrater reliability of the flight/landing subscore (ICC = 0.20). Significant differences in remote and in-person ratings for this phase were also reported by Worobey et al. in a previous study [13]. The authors argued that the significant difference between assessments of the flight/landing phase may be because this phase happens quickly and may be more difficult to correctly score the items of this phase from viewing it once [13]. The video recorded assessment provides the opportunity to view the items several times and adjust the scoring. The protocol of the laboratory-based remote TAI previously conducted by Worobey et al. included two camera angles for the rater to view and therefore, a clear view of the entire person and wheelchair was possible [13]. In real life, smaller confined spaces do not always allow raters to have a clear view of the entire person and wheelchair. Therefore, difficulties associated with limited camera angles while rating the TAI remotely in a real life may hinder the scoring of the flight/landing phase.

Despite the differences noted in both TAI total scores and subscores, the reliability analysis indicated good interrater and intrarater reliability of the remote home-based total TAI scores. The agreement analysis with the Bland–Altman indicated no evidence of trends in proportional error or error related to the measurement of error was found. However, caution should be taken when interpreting results as the small sample size in our study limits generalizability. The MDC indicates that through a remote home-based assessment using the total TAI score, a change of at least 1.04 (2 items) is needed to detect significant differences in transfer quality. An MDC of 1.30 (2 items) was reported for in-person TAI assessment [8]. Our findings are also comparable with the results presented in another study that reported an MDC of 1.23 (2 items) and good to excellent reliability of remote TAI assessment in a laboratory setting [13].

The findings of this study provide clinicians and researchers with a reliable option to assess real world transfer quality in home setting and the ability to reach clients/patients who do not have the means to travel to a healthcare setting in which an in-person assessment of transfer quality can be performed. The transfer assessments were performed in participant’s home environments reflecting their daily living transfer activities. In addition, participants and/or caregivers set up the cameras for the assessments and measured themselves the distance, angle, and level between transfers surfaces (TAI items 1, 2, and 6, respectively). The evidence presented in this study increases the potential for home-based telehealth and telerehabilitation of transfer quality among wheelchair users with SCI.

Study limitations

The study presents with several limitations. First, we did not perform a sample size calculation and the number of participants was limited to a convenience sample of 18 individuals with SCI. The small sample may have influenced the reliability analysis. Specifically, Bland–Altman plots with a bigger sample size are needed to further confirm the no significant trends in proportional error or error related to the measurement of error as reported in the current study. We were not able to present characteristics of the specific level of injury and the AIS presented by study participants was only used as an eligibility criterion of study participation. Also, we did not include wheelchair users with other health conditions such as multiple sclerosis, cerebral palsy, lower limb amputation, or others. Future studies with a larger sample size as recommended by the COSMIN [26], including a diverse populations of wheelchair users with various health conditions and evaluating the differences in home-based reliability of the TAI between level and severity of SCI, are necessary to increase the scope of the current findings.

The limited camera angles may make the scoring of some items of body setup and flight/landing difficult during a real-life home-based assessment. The placement of the cameras varied greatly among participants and was dependent on a variety of factors including space available for the assessment, location of assessment (bedroom versus living room), and device used (laptop versus cellphone). Future studies should investigate different camera placement options in an effort to standardize the camera set up and provide more detailed guidance to participants on ways to improve the video capture during the remote assessment. Such efforts will increase the reliability of the TAI assessment in all home environments. In addition, rater 1 had fewer years of experience using the TAI compared to raters 2 and 3, who had extensive experience with the instrument. This may have resulted in discrepancies with the assessments. Future studies should compare the evaluations of clinicians/researchers with similar experience using the TAI as well as the effect of experience/training on TAI scoring.

Conclusion

Our findings indicate that the TAI is a reliable outcome measure to remotely evaluate the quality of the wheelchair and body setup phases of a transfer and through self-assessment among wheelchair users with SCI in their home environments. Our findings also indicate that the remote flight/landing phase may be more challenging to score and camera angles and placements in wheelchair users’ homes are important aspects that need to be addressed in further studies.