Introduction

The onset of the Covid-19 pandemic precipitated a dramatic shift in the way mental healthcare services are delivered. The need for social distancing to mitigate the spread of Covid-19 has required health care organizations to swiftly develop and implement strategies to provide continuity of care while maintaining the safety of patients and providers. This is critical in mental healthcare where stress related to the virus and mitigation efforts may heighten symptoms among those with mental health concerns (Asmundson et al., 2020; Druss, 2020; Zhou et al., 2020). To address these concerns, telemental health (TMH), which includes the synchronous delivery of mental health services through electronic technologies including video and telephone, has been widely adopted by healthcare organizations (Baum et al., 2021; Connolly et al., 2020; Hoffnung et al., 2021; Mishkind et al., 2020; Shore et al., 2020; Tse et al., 2020). TMH is an effective means for delivering mental health services to diverse populations presenting with a variety of conditions (Bashshur et al., 2016; Fiacco et al., 2021). Furthermore, TMH removes some of the barriers to service access, thus improving accessibility among patients in low-access areas and populations with special needs (Bashshur et al., 2016).

Despite the demonstrated benefits of TMH, insurance coverage barriers have typically made it unfeasible to implement TMH into routine practice in many states (Wilson et al., 2017). Prior to the pandemic, state-level restrictions on TMH allowed fewer than half of states to provide TMH services at parity. Furthermore, organizations experienced barriers to implementing TMH including costs associated with deploying new technology and training providers (Wilson et al., 2017). However, at the onset of Covid-19 in early 2020, many state and federal governmental bodies lifted insurance and licensure barriers to telehealth, paving the way for its widespread adoption. In Massachusetts, the site of the current research, an executive order required local state insurance companies to permit in-network providers to deliver covered services via telehealth (“Order Expanding Access to Telehealth Services and to Protect Health Care Providers,” 2020). Critically, this order eliminated inter-insurance differences in authorizations. Similar orders went into effect in 40 states nationwide, and the Centers for Medicare and Medicaid Services (CMS) expanded coverage to visits conducted via telemedicine (CMS.gov, 2020; Haque, 2021).

In response to the rapid expansion of TMH, a burgeoning body of research has begun to examine the impact of Covid-19 related TMH implementations on patients, providers, and healthcare organizations. The findings of this research have generally been favorable; the majority of patients report feeling as connected to providers in the TMH model and many would prefer TMH or a combination of TMH and in-person in the long term (Nicholas et al., 2021; Tse et al., 2020). Others have described short-term improvements in visit completions and no show rates following the transition to TMH (Mishkind et al., 2020).

The current study builds upon this research to examine the impact of the transition to TMH on the Behavioral Health (BH) Department at Reliant Medical Group (RMG) in central Massachusetts. In the ten months following the transition from exclusively in-person only service delivery to TMH, we noted a 10% improvement in completed visits. This study uses an interrupted time series (ITS) design to examine longitudinal changes in the trends of the two factors that drive visit completion rates, cancellations and no shows, over the 14 months prior to and ten months following the transition to TMH.

Methods

Setting

[Name redacted] Reliant Medical Group is a multi-specialty medical group located in central Massachusetts serving over 300,000 patients in adult and pediatric primary care. Behavioral health services provided by the department include both treatment for mental health concerns and substance use, as well as behaviorally oriented treatments for physical health problems. This study focuses on visit data for 12,245 patients who received mental health services from the adult, pediatric and substance use services sub-clinics. The BH department operates an integrated model in adult and pediatric primary care, in which BH clinicians are tightly allied with primary care practitioners and available in real-time for consultation, triage, and intervention. The BH Provider is the primary internal treatment force, offering relatively brief, goal-oriented, evidence-based psychotherapy (8–10 sessions) to patients. The BH Consulting Prescriber is an MD or psychiatric advanced practitioner who supports primary care with prescribing advice and brief interventions to evaluate and adjust medications (usually not longer than 6–8 weeks) before passing the case back to primary care for ongoing management. In addition to these roles, there are specialty BH sub-clinics for substance use disorders, bariatrics, and a newly established dialectical behavior therapy (DBT) center. Combining these services, over 40,000 BH visits and consultations were completed in 2020.

On March 17th, 2020 all BH providers and consulting prescribers began to work remotely, conducting visits via telephone until all providers had been trained to use the secure video platform. Prior to this transition, all visits were conducted in-person. At that time nearly all visits transitioned to video. The BH chief trained each provider on the new video software via a one-on-one call. Additionally, all staff participated in a one-hour video training provided by a telemedicine start-up with expertise in providing BH video treatments.

Study Design

We conducted an interrupted time series (ITS) analysis of routine monthly encounter data spanning from January of 2019 to January of 2021, treating the March, 2020 transition to TMH as the intervention period. Considered to be the strongest quasi-experimental design, ITS is ideal for measuring the population-level impact of healthcare interventions and quality improvement initiatives where random assignment is not feasible (Penfold & Zhang, 2013). By treating the pre-intervention period as the counterfactual had the intervention not occurred, ITS can be used to identify changes in the slope of the data over time (Linden, 2015; Penfold & Zhang, 2013).

Data

Visit data includes monthly scheduled visits, completed visits, cancellations, and no-shows routinely captured in the Electronic Medical Record (EMR). We used visit data for all psychotherapy and psychiatry visits scheduled in the adult, pediatric and substance use service sub-clinics between January, 2019 and January, 2021. This resulted in 14 months of pre-intervention data and ten months of post-intervention data, which included a total of 102,072 scheduled visits during the study period. This covered 10,515 patients seen prior to the TMH-transition, and 12,245 patients seen after. Patient-level characteristics across the three sub-clinics and intervention periods are displayed in Appendix. This study was ruled exempt quality improvement by the Reliant Medical Group IRB.

Outcome Variables

The primary outcome variables were visit cancellations (including 24-hour cancellations) and no shows (visits where the patient did not cancel or show up to their appointment) as a proportion of total scheduled visits (all visits scheduled with BH providers and prescribers). The analyses focus on adult, pediatric, and substance use service sub-clinics, which cover 89 percent of all services provided in the BH department. We excluded visit data for DBT and bariatrics because both of these programs were initiated after the pandemic started.

Statistical Analysis

Prior to running the analyses, we examined the pre-TMH data for linearity. We assessed autocorrelation at different lags by plotting residuals and partial autocorrelation functions and implementing the Cumby-Huizinga general test of autocorrelated errors for each model to determine the optimal lag (Baum & Schaffer, 2013). The models were run in Stata 15 using the program its a, which estimates ordinary least squares (OLS) regression with Newey West standard errors (Linden, 2015). March 2020, the month TMH was introduced, was excluded from analyses as the intervention took place mid-month (Linden, 2015; Penfold & Zhang, 2013).

First, we estimated the change in the slope and level change of the percentage of cancellations and no-shows across the adult, pediatric, and substance use service sub-clinics combined. We then examined the changes in slopes and level changes for each sub-clinic individually. We also assessed potential seasonal effects and possible confounding due to changes in initial versus follow-up encounters, and psychiatric visits versus therapy visits. We did not identify any differences in the coefficients or standard errors for the primary variables of interest and these variables were omitted from the models.

Results

Visit Characteristics

In the 14 months prior to the transition to TMH, approximately 62.0% (95% CI 60.7, 63.4) of all visits were completed (i.e. patient did not cancel or no-show), a trend that was relatively consistent across all months, with the exception of December, 2019 when completions fell to 55%. With the onset of Covid-19 and the transition to TMH, there was an immediate 10% increase in the proportion of completed visits in April. Overall completions have remained around 72% (95% CI 71.8, 73.25) over the past 10 months. Pre and post-transition visits, cancellations, and no show rates across all sub-clinics, and disaggregated by each unique sub-clinic are displayed in Table 1. Prior to the transition, 27.8% of adult visits were cancelled, compared to only 17.3% following the transition. For pediatric visits, 30.3% were cancelled prior to the transition compared to 18.5% after. In the substance use sub-clinic, cancellations remained constant across the transition. In the postintervention period 98% of all visits were conducted via TMH; the remaining 2% face-to-face visits primarily included patients from the substance use sub-clinic who were not considered appropriate candidates for TMH. In the preintervention period, 68% of all scheduled visits were in the adult sub-clinic, 25% were in the pediatric sub-clinic, and 7% were in the substance use sub-clinic. In the postintervention period, approximately 63% of all scheduled visits were in the adult sub-clinic and 12% were in the substance use sub-clinic. The proportion of pediatric scheduled visits remained constant across the intervention. Patient demographic and psychiatric characteristics across the pre-TMH and post-TMH periods are displayed in the Appendix.

Table 1 Average monthly scheduled visits, cancellations, and no shows
Full size table

Aggregate Changes in Cancellations and No Show Rates

Table 2 and Fig. 1 display changes in the trend and level of the percentage of cancelled visits over time following the transition to TMH. Overall, cancellations decreased by 13.5% (95% CI  − 17.9, − 9.0) in April, immediately following the transition. In the preintervention period, the slope for the percentage of cancellations was not statistically different from zero (B = 0.2, 95% CI − 0.3, 0.8). Although the trend change in the slope of cancellations (B = 0.2, 95% CI − 0.4, 0.9) was not statistically significant, analysis of the postintervention linear trend indicated a significant postintervention slope period (B = 0.4, 95% CI 0.2, 0.7, p < 0.005), indicating a gradual increase in cancellations following the initial drop in the post-transition period.

Table 2 Level and trend changes across TMH-transition across combined sub-clinics
Full size table
Fig. 1
figure 1

Time series of % Cancellations for all Sub-clinics Combined. Lines linear trend. Vertical dashed line indicates the transition to TMH

Full size image

Table 2 and Fig. 2 display changes in the trend and level of the percentage of no shows following the transition to TMH. The percentage of no shows increased significantly by 1.4% (95% CI 0.1, 2.7, p < 0.05) in April, immediately following the transition. After this initial jump, the percentage of no shows declined at a rate of 0.6% per month (95% CI − 0.8, − 0.4, p < 0.001), a pattern that was not observed in the preintervention period where no shows were rising overtime. The monthly decline from May, 2020 to January, 2021 is depicted in Fig. 2. Due to the combined effects of the initial jump followed by the gradual decline over time, the overall no show rate remained at 9.0% in the pre- and post-TMH periods.

Fig. 2
figure 2

Time series of % No Shows for all Sub-clinics Combined. Lines linear trend. Vertical dashed line indicates the transition to TMH

Full size image

Subgroup Analyses of Cancellation and No Show Rates

Cancellation Rates

Table 3 displays the results of the subgroup ITS analysis including the baseline trends and level and trend changes for adult, pediatric and substance use service sub-clinics, respectively. Both the adult and pediatric service sub-clinics experienced significant level decreases in the percentage of cancellations immediately following the transition. Among adult service users, cancellations dropped by 12.9% in April (95% CI − 17.0, − 8.7, p < 0.001). Among pediatric service users, cancellations fell by 12.0% (95% CI − 16.2, − 7.8, p < 0.001). However, following initial level drops, both adult and pediatric sub-clinics showed significant gradual increases in the rate of cancellations. Among adults, cancellations increased by 0.6% per month (95% CI 0.1, 1.1, p < 0.05) and in the pediatric sub-clinic, cancellations increased by 0.7% per month (95% CI 0.1, 1.3, p < 0.05) in the post-intervention period. We did not observe significant changes in the level or trend of cancellations in the substance use sub-clinic.

Table 3 Level and trend changes across TMH-transition by sub-clinic
Full size table

No Show Rates

In stratified ITS analyses, both adult and pediatric sub-clinics experienced gradual declines in the percentage of no shows across the TMH-transition period. In the adult sub-clinic, no shows decreased at a rate of 0.7% (95% CI − 0.9%, − 0.5%, p < 0.001) per month after the transition. In the pediatric sub-clinic, no shows decreased at a rate 0.5% (95% CI − 0.8%, − 0.3%, p < 0.001). In the pediatric and adult sub-clinic ITS analyses, the level change that was observed in aggregate analysis of no shows did not reach statistical significance, although there were marginal trends in significance. In the adult service sub-clinic, there was a level change of 1. 2% (95% CI − 0.01, 2.4, p = 0.052) and in the pediatric sub-clinic there was a level change of 1.7 (95% CI − 0.1, 3.5, p = 0.069). In the substance use sub-clinic, neither the level nor trend of no shows changed significantly across the transition to TMH.

Discussion

By using an ITS design to examine trends in mental health encounter data from 14 months prior to the transition to TMH and ten months following the transition, we were able to examine the impact of TMH on visit cancellation and no show rates in order to better understand the overall improvement in visit completion. The patterns in cancellation and no show rates differed from each other. In the analyses aggregated across service sub-clinics, there was a sharp decline in cancellations in April, 2020 immediately following the transition to TMH, followed by a gradual increase in cancellations in the months after. This pattern was also observed among the adult and pediatric service sub-clinics in stratified analyses. However, the overall level and trend in substance use service cancellations was not impacted by the transition. Level and trend changes in no shows followed a different pattern. Directly after the transition to TMH, the overall level of no shows increased slightly, followed by a gradual decline over time. Similar patterns were observed in adult and pediatric service sub-clinics. Again, no clear changes in no shows emerged in the substance use services sub-clinic.

It is likely that the initial increase in no shows followed by a decrease over time in post-TMH no shows reflects a learning curve for the new virtual technology (Wootton et al., 2020). As providers and patients adapted to telehealth technology, they likely became more efficient engaging in the TMH model of care leading to incremental improvements in no show rates among pediatric and adult service users over time. Furthermore, systematic learning by the department, reflected in improvements in workflows, patient-facing materials, and support mechanisms, also likely contributed to the gradual reduction in no-shows over time.

One possible explanation for the sharp decrease in cancellations is that TMH improves visit adherence by removing some of the typical barriers patients encounter keeping face-to-face visits such as transportation, childcare, and other logistical barriers. It is also critical to consider that the decline in cancellations may have been related to the enhanced need for behavioral health services during the Covid-19 pandemic. Increases in mental health symptomology related to the pandemic and resulting social isolation may motivate patients to keep their behavioral health appointments. However, in one study of failure to attend TMH services using data collected prior to the pandemic, the authors reported that patients missed TMH visits at nearly half the rate that they missed in-person visits (Snoswell & Comans, 2021). This suggests that the reduction in cancellations may indeed be the product of improved convenience due TMH rather than an increased need for services due to the pandemic. Further, the gradual increase in cancellations several months into the pandemic may reflect state-level loosening of restrictions that allowed some workers to return to work sites and gradual reopening of local economies. These changes may have reintroduced logistical barriers to visit adherence as patients had more reasons to leave their homes. It is also possible that patients became more likely to cancel visits as they developed strategies for coping with stress related to the pandemic and relied less on BH services.

Differences Among Sub-clinics

It is promising that TMH appeared to positively impact both the adult and pediatric service sub-clinics. The literature on TMH among pediatric populations has identified mixed support for TMH. One study reported that adults preferred TMH visits and children were twice as likely to return to face-to-face visits following the lock down, compared to adults (Hoffnung et al., 2021). Yet, another study found that young people aged 12–25 reported that TMH was effective and that it made them less likely to cancel visits. Unlike pediatric and adult service users, patients using substance use services did not show changes in the rate of cancellations or no shows across the transition. This suggests that patients seeking substance use services may face specific barriers to engaging in TMH compared to other behavioral health patients. Prior to the onset of Covid-19, TMH was leveraged for patients seeking substance use treatment at twenty-percent the rate of use for other behavioral health patient populations (Huskamp et al., 2018). While research on TMH in behavioral health practice demonstrates broad clinical efficacy of TMH for behavioral health, relatively less is understood about the efficacy of TMH in substance use treatment and this is an important area for future inquiry (Huskamp et al., 2018; Lin et al., 2019; Lin et al., 2020).

Limitations

Despite promising evidence that the transition to TMH may improve visit adherence, this study has several important limitations. First, the rapid transition to TMH occurred in response to the onset of the Covid-19 pandemic and the effects of TMH cannot be separated from concurrent developments in the pandemic. Nonetheless, these findings are important as they show that in a public health emergency, TMH is a viable means for engaging patients in behavioral health care and may improve adherence. Furthermore, under non-emergency circumstances TMH may still benefit visit completion by removing logistical barriers. Second, although ITS is a powerful approach for examining trends in population-level data over time, individual level effects of TMH are washed out in aggregate analyses. It remains vital to understand how individual-level factors such as diagnosis and severity of illness impact adherence and outcomes related to TMH interventions. This will be especially important in determining the future of TMH and making decisions about the circumstances in which TMH models should be used. Third, we used a single group ITS design which lacked a control group, and it is possible that a concurrent, unobserved changes may have impacted no shows and cancellation rates, potentially confounding these findings (Linden, 2017). Finally, Reliant Medical Group operates under an integrated behavioral health model in which patients undergo relatively brief, targeted treatments for mental health concerns. Therefore, these findings may not be generalizable to more intensive, longer term behavioral health services.

This study has several notable strengths. By focusing on group-level outcomes, ITS is particularly robust to potential confounding by individual level factors (Penfold & Zhang, 2013). ITS allowed us to test both the immediate changes in outcomes as well as the trends in in these outcomes over time. While a mean comparison (e.g. t-test) of no shows would not reveal any change before and after the transition, by examining the time trends, we detected improvements in no-show rates over time following the TMH-transition. While we include data from 10 months following the transition to TMH, it remains vital to examine the longer-term impact of TMH on no shows on overall visit completion as more data becomes available. Finally, the ITS method controls for secular trends that existed in the data prior to the implementation of TMH.

Conclusion

This investigation suggests that TMH may improve visit adherence through significant immediate reductions in cancellations and gradual decreases no show rates over time. However, these improvements may not be universal for all patients or types of services. Future research is needed to understand which services can be delivered effectively via TMH and which benefit from in-person care or hybrid models. As mitigation efforts and vaccine distribution begin to slow the spread of Covid-19, it is important to understand the long-term role that telehealth may play in healthcare. Some states are beginning to pass legislation to require parity for services delivered via telehealth after it becomes safe to resume in person visits, paving the way for its continued use after the pandemic (An Act promoting a resilient health care system that puts patients first., 2020). The expansion of TMH following the Covid-19 presents an opportunity to increase access to behavioral health services, potentially reducing costs to healthcare organizations and payers. However, more work is needed to understand the optimal circumstances for telehealth services, as well as potential barriers that some patients may have to accessing necessary technologies.