Preparation–resources and capacity needed for implementation.

For the routine CHMT supportive supervision infrequent implementation was reported and observed several times (13/23, observational data). Main reasons for this were lack of time (human resources) due to competing tasks (11/23, observational data) as well as insufficient and untimely financial resources because of cumbersome administration processes at council level and delayed or inadequate disbursement of money from the central government (10/23, observational data). This was illustrated by a CHMT member as follows:

“Doing it [supportive supervision] has its challenges. We have… competing tasks. You do a schedule [which shows] that the whole week we dedicate to go to the health facilities for supportive supervision but in between some CHMT members are called for a certain seminar… Or we get visitors from the ministry, different organisations, NGOs that we work with. Thus, some of us need to go there, join them to do some work. Hence, you come to realize this week is lost and… maybe you managed to just visit one facility… these have been our big challenges…” (CHMT member, Iringa MC)

In contrast, for the e-TIQH supportive supervision approach financial resources were readily accessible through project funds. Also, several respondents stated that e-TIQH supportive supervision required less time at the health facility (10/23) and for reporting (4/23), supporting findings of the cost assessment below. This was clearly attributed to the electronic nature of the assessment and the more concise list of indicators compared to routine CHMT supportive supervision.

Additionally, some respondents (3/23) elaborated from a provider perspective how the e-TIQH assessment not only reduced time required, but also the time burden of supportive supervision as explained by a CHMT member:

“The e-TIQH assessment often doesn’t involve all the staff… Sometimes it’s not necessary [to directly work with] the facility in-charge… an experienced person can show us all the places… Therefore, the rest of the work continues as normal. Also, because you use the tablets it doesn’t take a lot of time… But for the one [supportive supervision] of the CHMT… it means the service stands still (…) But for e-TIQH you go and people go on with the work. (…) Then, to interview someone it doesn’t take a lot of time because once s/he answered, you just enter it [into the tablet].” (CHMT member, Rufiji DC)

If the e-TIQH approach was to be implemented by the CHMT alone, the main concerns regarding resources were the affordability of tablets (8/23), of the dissemination meeting (4/23) and of the platform running cost (observational data) as well as the availability and affordability of non-CHMT assessors (2/23, observational data). However, as presented in findings of the cost assessment below, these concerns could not be confirmed by the costing data apart from the cost of the dissemination meeting.

In terms of human capacity, for both approaches respondents emphasized the importance of well-trained assessors with the required contextual knowledge and professional skills to conduct the assessment (7/23), but also the organisational skills to ensure smooth implementation of the approach (11/23, observational data). Both issues were seen to be lacking to some extent during routine CHMT supervision, but not during e-TIQH supportive supervision. In addition, for the e-TIQH approach basic IT skills for managing the electronic devices were perceived as necessary (5/23), but less analytical skills were reported to be required due to automated data analysis (4/23). However, interviews and observations also revealed some inability to fully use the results generated at aggregated level by those in a position to do so (3/23).

Implementation.

Respondents said that a main advantage of the e-TIQH assessment tool was its design with the emphasis on key issues of primary healthcare (8/23), the wider range of topics assessed (e.g. staff motivation, patient satisfaction) (18/23) and the assessment type, which focused on adequacy and processes of care (clinical observations) (16/23) [38]. A facility in-charge summarized this as follows:

“There are a lot of supervisions being done, but they [e-TIQH assessors] want to observe [the service delivered]. It’s not like we sit and you ask [if] a certain thing [process] is being done. He [the e-TIQH assessor] wants to see if you are really doing it. If you say the guidelines are there, ‘Where are they?’ It’s not [possible] to say they are there, but they aren’t. He looks exactly where they are. (…) However, when they [CHMT] come, they look how you fill [the register book]. Thus, they don’t look how you did the diagnosis of this patient… but they look how you filled [the record books]. (…) But he [e-TIQH assessors] wanted to see how the clients are being attended, and how at the same time the data was entered [into the record books]. (…) But they [CHMT] do supervision by simply asking questions independent of whether or not there is a client.” (Facility in-charge, Iringa MC)

Moreover, it was seen as less biased (12/23) due to a more diverse and skilled assessment team (8/12) and the usage of an electronic tool (8/12). This was illustrated by a CHSB member:

“The nice thing about the [e-TIQH] approach was [that] it used a mixed group of assessors, it took people form the private [sector] to go and do supervision even at public [facilities]. So, this takes away biases.” (CHSB member, Rufiji DC)

As a result of improved assessment design and reduced bias, most people perceived the overall e-TIQH assessment as more accurate than routine CHMT supportive supervision (15/23). Observations and interviews with CHMT members revealed that checklists of routine CHMT supportive supervision were often more extensive and covered more types of medical services (8/23, observational data). However, mainly health facility record books and availability of care were investigated, whereas adequacy was only assessed to some extent and processes hardly ever (11/23; observational data). Thus, the increased perceived accuracy of the e-TIQH assessment tool was not necessarily due to a higher number of indicators. However, the perception of increased accuracy led to higher acceptance of the assessments and their results amongst stakeholders involved (18/23).

Reliability and completeness of routine CHMT supportive supervision data was reported to be strongly affected by inconsistent data collection (14/23, observational data). This was seen to be due to insufficient human resources, which resulted in a constantly changing composition of the assessment team (4/14) and lack of time to go through an extensive supervision checklist (13/14).

In contrast, owing to a clearly defined, fixed and more concise set of indicators as well as the electronic nature of the tool, e-TIQH assessment reliability was perceived higher and data completeness was not an issue (9/23, observational data). The electronic tool was also seen as more user-friendly compared to the paper-based assessment (9/23). In terms of feedback given, at health facilities interviewees stated that it was more adequate and constructive (20/23). This was said to be due to the more supportive attitude and language of the assessors (15/20) as well as the immediate availability of initial analyses thanks to the electronic format of the assessment (timeliness and accessibility of data) (11/20).

Thus, according to the respondents, acceptance of the feedback (16/23), ownership of the actions to be taken at facility level (12/23) and staff motivation (10/23) increased. This was elaborated by a CHMT member as follows:

“In the past you really only [pointed out] the problems… only problems. There were no congratulations to them [the healthcare provider]. There was no [thing like] telling them that at least they reached some percentages. But with e-TIQH… it shows you ‘Here you did well, here there’s a problem’. …you can see the area in which you have improved, and the area [where] you still have a lot of work. But the old one [routine CHMT supportive supervision] only showed problems. It didn’t show an area where you put efforts in. […] This one [e-TIQH supportive supervision],… it doesn’t discourage you… it shows you the weaknesses and where you did well. So you know it’s possible. At least you are activated [motivated] to continue working.” (CHMT member, Mvomero DC)

Ownership was further increased by the feedback summary form left at the health facility (6/23, observational data). Lastly, although it was intended in the e-TIQH approach to involve the HFGC chair during the feedback at the health facility, we could not find respondents confirming this. However, most interviewees generally supported this idea and saw it as an additional option to further increase feedback acceptance and ownership at health facility level.

Reporting.

Automated data entry with instant and continuous access to more detailed reports after uploading the surveys ensured timeliness and accessibility of data (18/23). According to respondents this was unlike routine CHMT supportive supervision where data was hardly ever systematically analysed (2/23, observational data), feedbacks delayed (8/23) and reports difficult to access (8/23, observational data). Further benefits of the electronic tool were increased legibility (3/23) and security (3/23) of the data compared to routine CHMT supportive supervision. Importantly, due to the overall improved data quality, the e-TIQH approach also led to more meaningful and actionable data, which could be aggregated and compared at health facility and council level (19/23) as illustrated by a CHSB member:

“We use those [the results of the e-TIQH assessment] because they were being compiled and they show that our facilities had the issues 1, 2, 3. In case of the routine supportive supervision [CHMT] I actually haven’t seen its results [showing that] we visited all facilities [and] we saw that the main problem is this… They produce [results] for individual facilities. But if they were to do it like e-TIQH to compile results [showing] that in all our health facilities it appears as this is the problem… Then this obtains weight during the planning [and] if it’s common [to all facilities], it is necessary to plan for this. (…) Thus, during implementation it gets priority. (…) Thus, it [e-TIQH] gives you an overview of the whole district [council] showing the problem is this, but the other one of the CHMT it was like individual [data]” (CHSB member, Rufiji DC)

In contrast, lower data quality of routine CHMT supportive supervision reduced its usefulness (15/23). Consequently, respondents reported that it was difficult to keep track of what needed to be addressed (8/15), do follow-ups (12/15) and monitor changes (10/15), which led to untimely and/or inadequate actions (8/15) and ultimately to no or only slow improvements. A CHMT and a facility in-charge stated their point of view as follows:

“When coming back [to the facility] for another supervision you may or may not find the [previous supportive supervision] report. Thus, you might not know anymore where the problem was. This is different from now… once uploaded, even at the office you have the file… Thus, it’s easy, even when going back another time you exactly know ‘There I left with this particular problem at that time. Now let me follow up and see how far they’ve come.’”(CHMT member, Mvomero DC)

Dissemination.

Having access to comparable health facility results (as it was the case during the annual dissemination meeting) contributed to increased result acceptance (18/23), ownership of quality improvement initiatives (16/23) and motivation (20/23) amongst all stakeholders. This was summarized by a facility in-charge:

“In the past this [dissemination meeting] was not done… They [CHMT] came, did supervision and left to do their [work] (…) Completely different from e-TIQH, because when they came [for the dissemination meeting] they transparently displayed for the whole district [council] how we deliver our services and where the weaknesses are […] I used to believe that maybe I was the only one with challenges, but when I arrived there, [I saw] there are colleagues of mine, whose conditions were very bad… So, at least I got motivated [that]… I had to work hard in order to reach another level… I was very pleased because I realized that I already reached a certain position. Thus, [I asked myself] what should I do in order to move further?” (Facility in-charge, Mvomero DC)

The annual dissemination meeting with all relevant stakeholders was seen as a crucial forum for mutual learning and understanding, where best practices, lessons learned, success and failures of quality improvement initiatives could be shared (8/23).

Financial and human resources.

Both qualitative findings and costing results demonstrated that e-TIQH supportive supervision reduced time and cost spent, despite a higher number of assessors needed per team. This allowed saving precious time of overburdened council and health facility staff as well as reducing the need for financial resources. Additionally, owing to the mixed assessment team in the e-TIQH approach only four and not five CHMT members per team were required, thereby further reducing the staff demands on the CHMT’s side. Thus, the more efficient use of human and financial resources could make supportive supervision implementation more feasible and therewith more likely to happen.

Data quality.

Importantly, our findings showed that e-TIQH supportive supervision also improved availability of evidence through the better quality of collected data. The electronic format of the tool in particular increased completeness, legibility, timeliness, accessibility, security and meaningfulness of the data, as well as the user-friendliness of the assessment. Automated data entry and analysis facilitated simple and immediate access to aggregated and comparable data and eliminated the problem of manual data entry errors. This was found to be a major improvement compared to routine CHMT supportive supervision, where data entry and systematic analysis was hardly ever done.

Other features of the e-TIQH approach also contributed to improved data quality. For example the improved assessment design, in particular the multi-dimensional quality concept and the assessment’s focus on processes and structural adequacy, increased perceived accuracy and acceptance of the assessment [38]. This was the case although the e-TIQH assessment tool did not have a higher number of indicators. In fact, the clearly defined and more concise indicator set improved reliability of the assessment. Thus, fewer, but more accurate indicators that are consistently followed up might lead to more substantial improvements than a more comprehensive indicator set, which is not consistently followed-up [38].

Also, a more diverse assessment team, involving CHMT core and co-opted members as well as community and private sector representatives, reduced bias and further increased perceived accuracy and acceptance of the assessment. Interestingly, the national supportive supervision guidelines already stipulate the need of such mixed teams, but it has not been implemented so far [20, 31, 33].

Feedback at health facility.

The way comprehensive and action-oriented feedback was given to all stakeholders at health facility level was another key feature of the e-TIQH approach. Instead of primarily focusing on negative aspects, the language was supportive and the advices were constructive and summarized in writing. Also, joint discussions were solution-oriented and clear, achievable tasks were assigned to all stakeholders involved. Thus, as hypothesised by Mboya et al. [37], the feedback led to increased knowledge and skills, was more accepted and improved motivation and ownership of subsequent quality improvement measures at facility level [39].The need for constructive and supportive feedback is supported by other literature showing its importance for effective supportive supervision [4, 5, 24, 26, 48–52].

Data usage.

Overall, the improved data quality of the e-TIQH approach allowed for more systematic follow ups, better monitoring of changes as well as more timely and adequate actions. Additionally, as speculated by Mboya et al. [37], the approach increased usage of collected data during planning and budgeting, leading to more evidence-based resource allocation. If used at national scale the e-TIQH supportive supervision approach would also allow for comparison between councils and regions, addressing another major gap of current routine CHMT supportive supervision [20]. This further opens up the opportunity to strengthen and facilitate the role of RHMTs in supervising CHMTs, which in turn would be likely to stimulate motivation and ownership of CHMTs to conduct adequate supportive supervision. At the same time, the e-TIQH approach would offer a great possibility to overcome the lack of national indicators for monitoring quality of healthcare and ensure improved alignment with indicators of vertical programs, development partners and national accreditation initiatives [18, 20, 27, 28, 31, 53]. The later was for example the case in Rufiji DC, a council where a national star rating system was introduced in 2016 (Table 1) [53]. Most importantly, by making the e-TIQH approach the standard approach for routine CHMT supportive supervision, it would ensure that required improvements are actually happening and therewith would accelerate the ministry’s efforts to move towards accreditation of all health facilities [31].

Financial resources.

Financial concerns in terms of purchasing tablets and covering platform cost could not be confirmed, as they represented only a small part of the overall cost. Additionally, tablets could be substituted by personal smart phones and an open-source platform could be made available to reduce running cost in the medium to long term. One-time financial start-up cost for introducing e-TIQH supportive supervision in a council was within the range of one round of supervision, depending on the type of council. In contrast, the financial cost for conducting an annual dissemination meeting, which exceeded the amount of one round of supportive supervision in rural councils, would occur yearly. Thus, it may be difficult to maintain the implementation of dissemination meetings in resource constraint settings. However, qualitative data indicated that dissemination meetings contributed substantially to increased knowledge and skills through mutual learning and understanding. It thereby supported the aim of the government to promote peer learning and exchange of experiences [18]. The dissemination meetings also strongly improved result acceptance, ownership of quality improvement measures and motivation amongst all stakeholders, similar to what has been shown elsewhere [54]. Thus, it was effective in rewarding good health facility performance without financial incentives. Spotlighting quality of care set the bar for performance and managed to create a system for recognition, something which is well known to improve motivation and retention of healthcare providers [3, 33, 55–57]. This was the case despite the fact that two out of three study councils already had experiences with pay-for-performance (P4P) schemes (Table 1).