Medical Device Regulation Efforts for mHealth Apps during the COVID-19 Pandemic—An Experience Report of Corona Check and Corona Health

Abstract

Within the healthcare environment, mobile health (mHealth) applications (apps) are becoming more and more important. The number of new mHealth apps has risen steadily in the last years. Especially the COVID-19 pandemic has led to an enormous amount of app releases. In most countries, mHealth applications have to be compliant with several regulatory aspects to be declared a “medical app”. However, the latest applicable medical device regulation (MDR) does not provide more details on the requirements for mHealth applications. When developing a medical app, it is essential that all contributors in an interdisciplinary team—especially software engineers—are aware of the specific regulatory requirements beforehand. The development process, however, should not be stalled due to integration of the MDR. Therefore, a developing framework that includes these aspects is required to facilitate a reliable and quick development process. The paper at hand introduces the creation of such a framework on the basis of the Corona Health and Corona Check apps. The relevant regulatory guidelines are listed and summarized as a guidance for medical app developments during the pandemic and beyond. In particular, the important stages and challenges faced that emerged during the entire development process are highlighted.

1. Introduction

Regardless of pandemic mobile apps, regulatory requirements in the area of medical mobile apps have been neglected for a very long time. In 2016, 2.5 million medical apps were listed by Statista in the two prominent app stores of Apple and Google [1]. The number is increasing almost daily by orders of magnitude that were unthinkable 10 years ago [2,3]. In general, the Medical Device Directive (MDD) 93/42/EEC [4] has been revised and substituted by the Medical Device Regulation (MDR(EU)) 2017/745 in 2017 [5]. In [6,7], the authors presented a new Medical Device Regulation and its key elements. The author of [7] pointed out the differences between new and foregone regulations, whereas in [6], the elements of a new MDR were revealed. Particularly, the workflow and important steps in technical documentation during the app development process were examined. Moreover, approaches to classifying software and applications with respect to risk classes were shown [6]. Trektere et al. [8,9] pointed out the necessity for an mHealth application development framework and its key criteria. They presented an existing framework for medical device software development and an approach to how it can be tailored for use in the context of mHealth application development. In [9], the author drew particular attention toward the key criteria Traceability, which is essential in the development process to ensure reliable software [9].

However, since regulatory frameworks for medical apps have not yet been the focus of governmental audits, anyone can deploy a medical app to the mentioned stores without much responsibility. From a regulatory point of view, this is accompanied by the following effects:

• Initially, little can be said about whether a selected app actually helps or is effective, with limited overall quality in most health promotion and healthcare domains [10,11,12,13,14,15]. Regulatory requirements cannot provide conclusive support in this regard, but they nevertheless give an indication of whether an app in question adheres to standards regarding validation, verification, and the fundamentals of data integrity.
• Given the vast number of medical apps offered, patients cannot know which app is actually the right one for their needs. Regulatory frameworks, including procedural quality assurance, can be a quality feature here and, at the same time, can serve as a clue to facilitate better navigation for those affected and to offer a more reliable assessment of an app.

Medical apps that adhere to the same specifications are therefore easier to compare. Hence, their mode of action can also be compared. Fortunately, both legislators and many initiatives (VDI: Association of German Engineers; DIN: German Institute for Standardization; and DKE: German Commission for Electrical, Electronic, and Information Technologies) are increasingly creating regulatory frameworks for medical apps. There are also an increasing number of initiatives working to provide navigation assistance for medical apps, such as MHAD [13] or AppRadar [16]. However, because such initiatives and guidelines have only recently begun to exist, the app market (both in research and academia as well as in the medical usage space) has rarely implemented these guidelines. This could also be due to the fact that too few of these guidelines are actually used and enforced by the relevant authority representatives as a basis for their audits.

Regulatory requirements must not be a stumbling block because speed together with affordable costs and the best possible reliability are other very important factors in the development of mHealth apps. In the context of pandemic apps, the mentioned aspects are particularly relevant. To be more precise, an app that works well and is accepted is not worthwhile if its regulation and application make the app only ready for use when the actual pandemic has already subsided. The regulatory requirements must be formulated in such a way that they are understandable and do not interfere with upcoming developmental work but rather support it. In the following, a field report of developing two mobile pandemic apps, namely Corona Check [17] and Corona Health [18], is presented, including a derived realization concept on how these apps could be developed quickly, cost-effectively, and tailored to pandemic needs. Important for the field report, the authors of this paper were part of the development team of both apps.

Corona Check was developed within the framework of a scientific collaboration between German university partners, the Bavarian Health Authority, and software companies. It mainly provides seven questions to help with the early detection of COVID-19. Corona Check therefore incorporates patient-reported outcome and mobile sensing with direct feedback to users via the app based on their answers to the seven questions. The overall goal was to provide a quick test that can be easily performed at any time and for any change in symptoms.

Corona Health, in turn, was also developed within a broader framework of participants and provides studies that are used to evaluate the impact of the COVID-19 pandemic on mental and physical health. The gathered results should show the need for improvements in the health systems and should pertain to individual coping skills. The outcome should also be used for recommendations to decrease negative impacts, which arise through the pandemic itself as well as through the steps to curb it. Corona Health is based on the following principles: patient reported outcomes, ecological momentary assessments, mobile crowdsensing, and sensor measurements. The latter encompasses GPS measurements as well as data on the usage of apps such as Facebook only if users allow these measurements. For a deeper understanding of these concepts, see [19,20]. As such, data collection and intervention strategies can be complex outside the world of mobile applications as assessed by many other works, e.g., [21]. Although Corona Check [17] and Corona Health [18] only provide a little feedback and intervention mechanisms, the overall picture that has to be drawn in the regulatory field is very challenging.

When developing these two apps, adherence to the Medical Device Regulation (MDR) arose as a requirement. Both apps were released onto the official app stores from Google and Apple and are compliant with the MDR. To achieve the objective of compliance, the following aims were set up by the teams that developed these apps:

• Establish a harmonized regulatory approach that draws the best from all rules and standards, thus avoiding over-regulation, but still showing conformity with the Medical Devices Regulation.
• Possibility of an iterative development approach while maintaining the necessary regulations.
• Creation of a simple and feasible development procedure, so that the high development speed is not slowed down, but at the same time, all necessary regulatory requirements are met for the app in question.

As a kind of side effect, an approach through which efficient error prevention becomes possible due to a more detailed requirements phase was created:

• Reliable project schedules can be created with reliable cost estimates.
• Reduction of error costs (i.e., a detailed requirements phase prevents errors in testing or after going live).

The important aspects to achieve the aforementioned objectives are discussed in this paper. We hereby want to contribute with the following insights:

• We share take-home messages that we gained through the development of the two pandemic apps Corona Check and Corona Health in light of regulatory aspects.
• We share insights about the important aspects of the development process.
• We share insights of the costs incurred.
• We share insights about the perception of users based on a conducted study within the Corona Health app.

With these insights, we hope to help other mobile app developers create mHealth apps in the context of regulatory needs during pandemic times. We further think that these insights are of interest beyond the scope of the COVID-19 pandemic.

The remainder of this paper is organized as follows. In Section 2, existing standards that are useful and suitable as a regulatory basis for the development of medical apps—in general and for a pandemic app in particular—are presented. The creation/implementation of a pragmatic procedural model is the basis for the provision of correctly created regulatory apps and thus for gaining the trust of citizens. Section 3 includes the detailed concept of a regulatory compliant app creation process, whereas Section 4 exemplifies this process on the basis of two realized application development projects. The conclusion of the paper, consisting of a summary and an outlook, is covered in Section 5.

2. Background and Regulatory Basis

Since neither the Medical Devices Act nor the Medical Device Regulation provide more detail on the requirements for a medical app, first, common standards and regulations were identified [4,5]. Furthermore, they had to be analyzed and compared with regards to relevant topics. Following this, a suitable and harmonized regulatory basis was created, on the basis of which project managers, developers, and quality managers could jointly develop an app that was feature-complete, (almost) error-free, and of high quality. In the following sections, relevant regulations from the healthcare sector are listed, analyzed, and compared.

2.1. Collection of Common Standards in the Healthcare Sector

In the field of software development, especially for mHealth apps, various regulations and standards can be considered. For our mHealth apps Corona Check and Corona Health, we incorporated the standards mentioned in this section. Finally, we combined the list of regulations used into a harmonized approach. We focused on conforming to EU regulations without neglecting international standards. Note that the involved company LA2 GmbH is experienced in developing software with regulatory compliance and helped the academic and software engineering teams. As currently less works exist about how mHealth apps have to be developed in this context, the experiences of LA2 were transferred for the approach used in Corona Check and Corona Health. Further evaluations, the procedure to finally obtain a CE marking, and comparisons with other works must reveal whether the accomplished procedures shown here will last over time. However, we received a positive ethical vote as well as the acceptance by Google and Apple based on this procedure. It must be further stated that the validation procedure was performed by ourselves; however, we strictly adhered to the mentioned regulations and standards. As the LA2 GmbH works within the regulatory field and has much experience in validations, we hope to finally get also a CE marking in future. The final list used to accomplish the self-validation covers the following regulations and standards:

• IEC 62304—Medical Devices Software—Software Life Cycle Processes
  → The IEC 62304 standard specifies the requirements for software life cycle processes for the development of medical software and software in medical devices. The processes, activities, and tasks prescribed in this standard form a common framework for action for all life cycle processes for software in the field of medical devices [22].
• GAMP5—Good Automated Manufacturing Practice Supplier Guide for Validation of Automated Systems in Pharmaceutical Manufacture
  → This guide has become the standard set of rules for the validation of computerized systems in the pharmaceutical industry (manufacturers and suppliers). However, the GAMP5 set of rules is not legally binding. Therefore, different forms of validation of computerized systems are possible, which is useful for many systems [23].
• General Principles of Software Validation—Regulations of the U.S. Food and Drug Administration (FDA)—U.S. Agency for Food and Drugs.
  → This guidance outlines general validation principles that the FDA considers applicable to the validation of medical device software or the validation of software used in the design, development, or manufacture of medical devices [24].
• PICS 11-3 PIC/S Guide to Good Manufacturing Practices (GMP)
  → The PIC/S Guide to GMP is the basis for GMP inspections. In particular, its Annex 11, Computerized Systems, is used in the inspection of such systems.The purpose of this document is to provide recommendations and background information on computerized systems to assist inspectors in training and during the inspection of computerized systems. The document is intended to serve as good practice for inspectors that are responsible for inspecting applications in the regulated pharmaceutical sector [25].
• IEC 82304 Health software—Part 1: General requirements for product safety.
  → This standard covers software as a medical device, including Mobile Medical Apps as well as other Health Software (outside the application of Regulation (EU) 2017/745 on medical devices). IEC 82304-1 has closed a gap in the normative coverage of validation and documentation of software placed on the market without specific hardware. The key points of this standard are requirements for product safety (SAFETY) as well as for information security (SECURITY) of health software products. This also includes requirements for usability (USABILITY) and instructions for use (INSTRUCTION FOR USE). The standard requires compliance with a software development process and refers to the IEC 62304/A1:2015 [26].

2.2. Overview of Relevant Regulatory Standards

After the successful collection of the required regulations, a topic catalog was created by the team of Corona Check and Corona Health, in which all important topics and regulatory requirements on the topic of app development were collected. The following points were agreed upon: software life cycle with the topics validation plan/documentation (incl. verification), risk management as well as the involvement of service providers, and the training of employees. Subsequently, it was examined whether the abovementioned main topics were addressed by the various sets of rules. If this was the case for a set of rules, then the requirements of the respective key topics in the relevant set of rules were checked for the scope and proportionality of the activities required there. In summary, all of the aforementioned sets of rules addressed the specified topics and were each very similar in the scope and proportionality of the activities required therein. Finally,

Table 1. Overview of relevant regulatory standards.

Subjects and Standard	General Principles of SW-Validation (2002)	IEC 62304-2015	PIC/S 11-3-2007	IEC 82304-2016	AAMI TIR36:2007	GAMP5:2008
(1) Software Life Cycle—general, incl. validation plan	Sections 4.4 and 5.1	Sections 5 and 5.1	Section 9	Section 6.1	Chapter 4	Sections 2, 3, 4, 3.3, 6.1.6 and M1
(2) Software Life Cycle—V-model documentation (Planning-RE-Design-Test)	Sections 5.1–5.2.6	Sections 5.2–5.8	Sections 5–14	Sections 5 and 6	Sections 4.1–4.3	Section 4
(3a) Risk Management (classification)	Section 5.2	Section 4.3	Section 15 (reference to GAMP)	Section A3 (sub-item to 5.), as well as reference to IEC 62304 and thus to Section 7	Section 7B	Sections 4.2.6 and M4
(3b) Risk Management (continuous risk management process)	Section 5.2	Section 7	Sections 5–14	Section A3 (sub-item to 5.), as well as reference to IEC 62304 and thus to Section 7	Section 7B	Sections 5 and M3
(4) Software Lifecycle—Software Maintenance and Support (CR Development)	Sections 4.7 and 5.2.7	Sections 6.1–6.3	Sections 17–18	Sections 1.2 and 8.2 as well as reference to IEC 62304 and thus to Sections 6.1–6.3	Section 4.4	Sections 3.1, 4.2 and 4.3
(5) Software Life Cycle—Decommissioning	Section 5.1	-	Sections 14.2 and 17.2	Sections 4.2, 7.2.2.9 and 8	Section 4.5	Sections 3.1, 4.2, 4.4 and M10
(6) Involvement of software service providers	Section 6.3	Referred in Section 4.1—by reference to QM system	Section 11	Reference to IEC 62304 and thus indirectly to Section 4.1—reference to QM system	Indirectly in Section 2.2—reference to QM system	Sections 6.1.4 and 7
(7) Training of employees	Section 2.4— Reference to QM system	Referred in Section 4.1—by reference to QM system	Section 22	Chapters 6.1.e, as well as reference to IEC 62304 and thus indirectly to Section 4.1	Indirectly in Section 2.2—reference to QM System	Section 6.1.3

provides a general overview.

The previously defined main topics were also adhered to after completion of the analysis. After further investigations, it was decided that, with IEC 82304, a set of rules could be used that was designed precisely for such a case (standalone software systems and health apps). However, a major problem was that the requirements of this standard still had quite extensive consequences on the scope of possible documentation and the development speed of an app project. Essential points from the other standards were therefore updated in the planned development process, which was then additionally tailored in a risk-based approach, i.e., finalized (one could also say streamlined). In this way, the other standards were taken into account and, despite this, the regulatory requirements for an app development project could be reduced to a sensible minimum level of effort. In other words, the initial development process included the good stuff from all of the previously outlined standards and still contains all of the relevant specifications of IEC 82304-1. This concrete conceptual procedure is described in detail in Section 3. It can be used as a template for upcoming projects. It should also be noted that compliance with this regulatory-compliant development process was ultimately essential for medical apps developed to be included in the Google (Google Play) and Apple (App Store) stores. However, this did not take the form of sending a release certificate directly to Google or Apple, but this release certificate in turn formed the basis for the following to be maintained:

• on the one hand, a positive vote by an ethics committee (here, from the University of Würzburg, which ultimately examines the content according to ethical, medical, and scientific aspects, among others), and,
• on the other hand, the support of a public body (Robert Koch Institute (RKI), Bavarian State Office for Health and Food Safety, …)

Both were ultimately key conditions for store operators to include a COVID-19 medical app as such in the stores. However, in order to obtain final approval from the ethics committee and the public body in addition to a release certificate further suitable proof had to be provided that the aforementioned standard(s)—including the harmonized cut—were actually complied with in the software project. This was provided in the form of so-called validation documents, training certificates, and standard operating procedures (SOPs). They had to be submitted to these bodies as proof of compliance with Medical Device Regulation/Ordinance on Medical Device specifications or their (abovementioned) underlying standards. The following section describes how this process was structured, which documents were required, and what their basic contents look like. Finally, we want to mention that the fact that a public body had to confirm its collaboration for store approval is a particular aspect for COVID-19 apps, and we assume that in the future, more standardized procedures will be established by store operators for times like the pandemic.

3. Resulting Concept of the Regulatory Compliant App Creation Process

Proof that regulatory requirements have been met in an app development project is provided by supplying the documents mentioned in the previous section. In order to provide these documents properly, it is necessary for the development project to be planned in a regulatory manner. Since all of the regulations mentioned here recommend a risk-based approach, this also sets up the basis for further planning. More details can be found in Section 3.3.

3.1. Software Life Cycle Requirements

During planning, in addition to the development phases and the corresponding documents (e.g., requirements phase = specifications, acceptance test = system test, etc.), service providers, training courses, test concept, and the procedure for requirement tracing (ensuring that all requirements are actually contained in the software at the end) must be defined. Before this step, first, the roles, their responsibilities in the project, and their assignments to the current project members must be defined.

3.1.1. Roles and Assignment to Project Team Members

In most software development projects, different development roles are involved. If software should be developed compliant with the medical device regulation, the following roles should exist and be trained in the aspects required for the medical device regulation validation procedure:

• Project Management:
  Responsible for the implementation and management of the entire product development.
• Quality Management:
  Responsible for ensuring software quality, regulatory requirements as well as the archiving of documents.
• Development:
  Responsible for development planning as well as related testing (except system testing).
• Test:
  Responsible for overall system testing (planning and execution) and risk management.
  Note: Risk management can be accomplished by a separate role.
• Main responsible person:
  Main responsible for the whole project. Most importantly, this person is responsible for the issuance of the final release.

Current team members must be assigned to their respective roles in the project. Necessary trainings still have to be defined (see also involved service providers, trainings, and necessary SOPs below). Once all of this has been defined, the next step is to define the development phases (including the corresponding documents) and the role responsibilities for documents as well as review planning. The mentioned development team members worked tightly together with medical experts and epidemiologists.

3.1.2. Development Phases/Documents, Responsibilities, and Reviews

Since app development processes usually have a manageable complexity, development steps required by regulations can usually be combined during planning. A harmonized approach from IEC 62304 Class B and GAMP5 Cat. 4 (configurable software) was used as the basis for further planning of the development process. Accordingly, the development phases of the project with the associated documents are structured as follows:

• Planning phase (document title: Software Quality Management Plan or also Master Validation Plan)
• Requirements elicitation (document title: requirements specification or requirements specification)
• Summarized functional description + design (document title: combined functional specification/design specification)
• Programming (document title: no separate document is created here)
• Combined module/integration test (document title: Combined module/integration test with specification/protocol/evaluation)
• System test (document title: System test with specification/protocol/evaluation)
• Release (document title: release protocol, e.g., for ethics committee and store operator)

The commonly known waterfall process (here, the so-called procedure model of software development or also called V-model), with the corresponding documents, would usually be as seen in Figure 1. However, since we recommend an agile software development process for (development) speed reasons, the actual development process is planned as shown in Figure 2.

In the planning phase, it is defined that the requirements must be first collected in detail in order to have the basis (the so-called backlog) to implement these requirements in an agile process design-programming-test (green-shaded area). Once all requirements have been implemented and tested in the various sprints (short development cycles), they must be finally tested in the system test. All validation and verification documents can be left in the design status until the final release is due. Interim test results after each sprint are documented. Towards the end of the project, these documents can be released step by step. Now that roles, development process, and responsibilities have been defined, the requirements for service providers, training, and necessary SOPs must be formulated.

3.1.3. Involved Service Providers, Training, and Necessary SOPs

The service providers involved are listed, their qualification and availability requirements are formulated, and any organizational framework conditions are defined. Before the start of the project, the project manager must ensure that the employees involved (both their own and those of the service provider) have sufficient experience and are trained in the regulatory principles relating to app development/validation. Likewise, all training required of the respective project members (according to their roles) should be defined. Among other things, the team members should have read and understood the procedure described here in the Master Validation Plan (or also called Software Quality Management Plan). The results must be documented in a log file or in training cards/passes. In addition, all SOPs necessary for the use and operation of an app are presented. Once all of this is completed, it must now be ensured that the app actually meets all of the required specifications. The verification is performed formally in a so-called requirement tracing process.

3.1.4. Tracing Processes

It must be planned that the requirement tracing (tracing of requirements across the development process) must be carried out in three passes. This ensures the presence of all loads in the combined requirements specification/design specification and at least one test in each of the respective test specifications (module/integration test specification—system test specification). The following documents are therefore created:

• Trace matrix Requirements specification—Requirements specification/design specification
• Trace Matrix Specifications—System Test Specification
• Trace matrix Requirements/design specification—Module/integration test specification

After the requirements for the tracing process have been completed, last, but not least, a lean test concept must be formulated.

3.1.5. Test Concept

The tests to be performed have already been planned in the development process and the underlying documents named. The module/integration tests can take place in a simulated environment (it is recommended here to test in the productive environment as soon as possible). The final system tests must take place in a productive environment. The tests are usually carried out in two phases. In phase 1, the so-called module/integration tests have to be passed, and, in phase 2, the system tests (or also called acceptance tests) have to be passed. In this process, the module/integration tests are always completely defined (and released) per iteration and executed, and the test results are finally logged and evaluated. If complete tests cannot be performed in an iteration, the regression tests to be performed must be defined in advance. The printed test specification represents the test protocol, which must be completed by the testers. After completion of the tests, each tester checks the completeness and validity of the entries and confirms this by date, abbreviation, name, and signature on the printed test specification for each of the requirements tested by him/her. Alternatively, the assignment of signatures and abbreviations can also be recorded once in tabulator form, in which case it would be sufficient for the tester to confirm the completeness and validity of the entries by date and abbreviation per test case. Note: where appropriate, the test result should be supported by one or more screenshots. The test results are then to be evaluated by the respective responsible persons. This can occur in an additional document. Once the successful system tests have been completed, the software can be released (see Section 3.2).

3.1.6. Interfaces

Not every app has interfaces to other systems. However, if this is the case, then these interfaces must also be validated and verified. As an exception, this is now performed on the basis of the GAMP5 set of rules. The procedure described there for Category 3: Non-configured product is used here. This means that requirements must be formulated for the relevant interface and the corresponding system test cases. The documentation can be combined in a single document, a so-called Project Summary (also called Fact File or Project Profile). This document contains a short organizational section, requirements specification, acceptance test specification including protocol, traceability matrix, and risk management elements. In terms of content, the data to be imported (data structure) must be defined, the transfer type/route (including secure/near-time data transfer, interpretation logic, and configuration) must be specified, and the data storage locations (data transfer or storage) must be formulated.

3.2. Release

After completion of the training courses and creation of the aforementioned documents, including successful system tests and the formulation of any necessary SOPs and project summaries for interfaces, the app can be released: for this purpose, an official release document must be created as proof for authorities. It must contain a list of all created and released (verification) documents and must be signed by all project participants with primary responsibility.

3.3. Risk Management Process

The risk management process taking place for a development project in the regulated environment should always happen at least on the basis of ICH Q9 Quality Risk Management (ICH: International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use) [27]. In this process, at the beginning of the project, and during the development process, risks are constantly

• collected,
• rated,
• mitigated, and
• monitored.

Note: The process shown here is simplified for explanatory purposes. Risks have to be divided into at least into one of the following groups:

• Product Risks;
• Patient Risks; and
• Risks related to data integrity.

All stakeholders should be represented in the risk management process. These can be, for example, project managers, developers, quality managers, users, representatives of the authorities, or people with IT development experience. If these risks cannot be mitigated by the IT, efforts can also be made to reduce the risks to a necessary minimum through training and detailed SOPs (Standard Operational Procedures). Since the actual risks can vary from app to app and situation to situation, no presentation or summary of possible risks was provided. However, to give an impression of the effectiveness and efficiency of the risk management process, its results in the Corona Health project have been summarized in Figure 3 as an example:

The left table shows the classification of the risks before mitigation, while the right table shows that after risk mitigation. Care must be taken to ensure that none of the risks fall into the red defined range (intolerable range) and that the risks are reduced as far as possible to a tolerable minimum. Risk management should be an essential part of future IT development projects for medical apps.

3.4. Acceptance and Evidence

In general, a high level of quality achieved for Corona Check and Corona Health might influence the acceptance when using the apps. Compliance with the MDR can help in two ways with respect to user acceptance. First, the practical steps that have to be accomplished for the MDR help to improve the quality of the apps, i.e., inherently helping users through the identification of risks and errors. Although this should be the goal for every software project, the intensive tests and harmonized procedures required for the MDR help to improve the overall quality as many more perspectives are taken into account (e.g., standard operating procedures). Second, it can be assumed that users take notice of the compliance and feel safer and more comfortable. In the Corona Health app, we are able to conduct different studies separately. In the context of a large German project on apps for future pandemics in Germany [28], an anonymous study was carried out to deal with questions on acceptance of pandemic apps. The study can be found in the Corona Health app [18] under the label University Medicine Network: Compass project on the acceptance of pandemic apps (18 years and older). It comprises 75 questions that can be only answered anonymously. The description of this study, which is presented to interested users, is as follows:

• Study on University Medicine Network: Compass project on the acceptance of pandemic apps (18 years and older): The variety of pandemic apps, apps that have been developed, for example, in hackathons to control COVID-19, shows the great potential that many experts see in them. However, for apps to be effective in the pandemic, they must be used by many people. This applies not only to the Corona warning app but also in particular to apps for assessing individual risk, for example in the case of certain pre-existing conditions. It is therefore necessary that such apps enjoy trust among the general population and that data can be analyzed together for medical research with the consent of the users. In COMPASS, scientists from a wide range of disciplines from university hospitals are joining forces with partners from science and industry in an interdisciplinary project to jointly develop a coordination and technology platform for pandemic apps. Help us with this survey so that pandemic apps can be developed even better and with broad acceptance and transparency in the future.

In this study, two questions aim to find out whether compliance with the MDR creates trust in mHealth apps. We have descriptively analyzed these questions for the paper at hand; Figure 4 shows the results of 304 users that participated in this study so far. On the left-hand side of Figure 4, it is shown whether apps being developed that are compliant to the MDR enjoy greater trust in general: 106 participants said yes, 18 participants said no, 47 said that it does not play any role, while 133 said that they did not know what MDR means. On the right-hand side of Figure 4, it is shown whether apps being developed that are compliant to the MDR enjoy greater trust compared to apps that are not compliant. Here, 112 participants said yes, 31 said no, 46 said that it is not important, while 115 participants said that they did not know what MDR means. As more than one third said that it creates more trust, in a market where the overall usage is still low, the aspect to be compliant with the MDR can be considered positive based on this result. However, more data have to be collected in future works.

In the same COMPASS project [28], a study was carried out to reveal evidence-based findings at the intersection of mHealth apps and the pandemic. The search was also extended to evidence of mHealth apps outside the scope of the pandemic. Interestingly, only guidelines and meta-analyses were found, which show aspects that should be considered. Evidence-based results have been not found so far. This underlines that the field of mHealth is still in its infancy for larger settings in the medical field. Consequently, the rigorous procedures of Google and Apple that do not easily accept all apps can be better understood and evaluated in light of these findings.

3.5. Costs

The LA2 GmbH has helped the academic team members to validate Corona Check and Corona Health, including training how the documents have to be prepared. The latter helped to reduce the overall project costs as the academic team members were able to create many documents without help from the company. However, to show what costs have to be typically planned in Germany, a detailed calculation for Corona Check is provided. Based on this, we present the number of costs for Corona Health based on the summarized costs from the same stages as that for Corona Check. Although the costs are aligned to a German setting, we consider the numbers to be s reliable cost indicator outside the scope of Germany as we provide the periods of time (i.e., required weeks) that must be considered to complete the documents needed for adhering to the MDR during all stages. To be more specific, the cost planning for Corona Check can be divided into four stages:

• Organizational matters (stakeholder definition, training, creation of SOPs, approval, and application to ethics committee). Involved persons: 2, work output needs per person: 1 week. Based on the following cost formula: 1.000 Euro/day full costs (=125, Euro/h mixed calculation rate), this phase can be summarized to 10.000 Euro.
• Planning/requirement phase. Involved persons: 3, work output needs per person: 3 weeks. Based on the following cost formula: 1.000 Euro/day full costs (=125, Euro/h mixed calculation rate), this phase can be summarized to 45.000 Euro.
• Development phase. Involved persons: 5, work output needs per person: 3 weeks. Based on the following cost formula: 1.000 Euro/day full costs (=125, Euro/h mixed calculation rate), this phase can be summarized to 75.000 Euro.
• Test phase iterative. Involved persons: 2, work output needs per person: 3 weeks. Based on the following cost formula: 1.000 Euro/day full costs (=125, Euro/h mixed calculation rate), this phase can be summarized to 30.000 Euro.

All four stages can be summarized to 160.000 Euro for the MDR for Corona Check. The same calculation procedure results in 250.000 Euro for Corona Health. Corona Health was more complex due to two aspects. First, Corona Check is only provided in two languages, while Corona Health provides eight languages. Second, Corona Health is comprised of five studies, while Corona Check has only one study included.

4. Discussion

In 2020, the mobile apps Corona Check and Corona Health were created one after another. Both projects went through the planning and software development processes described above. The sequences and internal durations were very similar in both projects (depending on the project scope). External factors were responsible for the different overall project durations. The first thing to mention here are the releases in the Apple and Google stores during the pandemic, which took about 11–50 days. In general, how can the differences be explained? Between March–August 2020, Google and Apple changed their store approval conditions due to pandemic reasons. Initially, the two companies did not set any requirements for so-called Corona apps, but this changed after a few weeks and votes from an ethics committee became mandatory. After another 3–4 months, the recommendation of an official body (for example, for Corona Health, the RKI) was required as well. Furthermore, the additional time needed for multiple internal release rounds of the store operators (duration of about 5–6 days each) subsequent to fulfilling the requirements were among the external factors. Moreover, the subsequent approval of central search terms such as Corona and Covid took at least 45 days for the Google Play store. More specifically, the latter period of time had to be passed until Google found the apps when using these search terms. For example, from approval by the ethics committee to the moment Corona Health was found in the Google Play Store, under the search terms Corona or Covid, 82 days passed. Why is the release for central search terms so important? If an app does not appear in the hit list for a general search term such as Corona, it is practically not found at all in the store. In our case, the search term Corona Health App had to be entered into the Google Play Store for the Corona Health app to be found and displayed at all. When the app was finally unlocked for the central search terms, the download numbers multiplied by orders of magnitude. However, it must be also clearly articulated that, in the case of the Google Play store, it was and still is a courtesy of Google to maintain a special COVID-19 search realm. Apps that are listed in this realm are particularly displayed. The mentioned wait time for being listed in the Google Play Store is therefore related to this particular search realm during the pandemic. It is reasonable that Google evaluates mobile apps in a time such as a pandemic in an in-depth manner. In the case of the Apple app store, no particular realms were offered. Figure 5 summarizes these relevant time aspects.

It is easy to see that the difference in the development times of the two app projects is only about two weeks. If we solely rely on the development period, the following schedule results in the basis of these two projects for an app with medium scope and complexity (the company selection must have already taken place here):

• Determination of the project participants (Day 1–Day 8)
• Training Phase (Day 9–Day 11)
• Planning Phase and Requirement Phase (Day 8–Day 35)
• Development phase (Day 30–Day 72)
• Test phase (Day 44–Day 79)
• Create and release SOPs (Day 70–Day 79)
• Release (Day 79)
• Request Ethics Committees (Day 79–Day 86)
• Release by the store operators (Day 86–Day?)
• Found on Google under the search term Corona (Day 86–Day?)

In the project sequence shown here, it is noteworthy that the relatively long requirements phase and regulatory framework hardly slow down the actual development speed. On the contrary, the relatively long requirements phase and regulatory specifications mean that ambiguities and problems can be identified and eliminated in a reasonable amount of time, including potential errors (sources) being identified and eliminated in a rather short period of time. Thus, a solid, consistent, testable, and complete requirement documentation is created on the basis of which the following can be achieved: short development cycles, combined with extensive testing and fast feedback loops, should be the aim so that app development can be completed in three iterations, in the best case. The validation documentation then helps with further developments of the apps, additionally shortening the times for the onboarding and training of the project members in question. Existing tests can be planned in detail and, if necessary, expanded through existing extensive test specifications, thus forming the basis for finally performing the tests successfully [29].

We want to conclude with some take-home messages we gained through our app developments of Corona Check and Corona Health during the COVID19-pandemic.

Take-home Messages: As the development of software compliant to the MDR is not new itself, we want to emphasize which take-home messages could be unveiled by this work. Our major messages for mHealth apps during the pandemic are as follows:

• Although mobile apps pose different characteristics to many other software projects (e.g., different mobile operating systems must be considered), a harmonized approach based on the existing standards is finally possible to accomplish, even in a short period of development time.
• Due to a clear distribution of roles and tasks as well as a detailed and extensive requirement engineering process (which is required for app development based on MDR anyway)—the complete development time of the app was hardly extended, since potential errors and the resulting unnecessary additional cycles were avoided. The product quality of medical apps required by the MDR was thus already given in a version 1.0.
• Especially for the risk analysis, a tight integration of the interdisciplinary team is required as mHealth apps require perspectives different from other medical software projects. For example, it includes app store involvement, particular requirements of an ethics committee, shorter development cycles, etc.
• Against the background that less mHealth apps that are compliant with the MDR exist, we assume that frequent change requirements (much more than in other software projects; e.g., changes in the underlying mobile operating system) are one major issue that prevent initiating endeavors aimed at the development of mHealth apps compliant to the MDR.
• The latter aspect emphasizes, among others, that experience reports such as that shown in this work could help and support others. In addition, more frameworks and templates are needed in the mHealth field that can be used in an off-the-shelf manner from a regulatory point of view.
• No strict, dogmatic approach, as specified in the extensive IEC 82304, was followed, but a pragmatic, harmonized approach based on IEC 82304 was chosen (in combination with other relevant standards), which was accepted by the ethics committee. Therefore, industrial quality can also be generated on this basis in the university environment.
• Costs play an important role and should be considered and minimized; we therefore recommend training an interdisciplinary team over time to manage costs that are caused by the MDR.

Beyond the scope of the COVID-19 pandemic, considerations of regulatory aspects in the context of mobile apps are still scarce. A good overview of pitfalls and problems can be found in [30]. Although it discusses the situation in the USA, many aspects are of general importance and interest. In particular, the authors discuss the risks for patients as well as doctors and how regulatory aspects can help mitigate these risks. Regarding the COVID-19 pandemic, many works have been presented recently. For example, this work discusses the need for regulatory aspects but also shows that the Food and Drug Administration (FDA) has relaxed the conditions during the pandemic [31]. Regarding relaxation during the pandemic, other works also deal with the decision of the FDA [32]. A very good debate for COVID-19 apps and regulatory aspects can be found in [33]. The authors state that ”regulatory bodies will need to question their current approaches by adopting comprehensive evaluation processes that adequately consider the specific features of mHealth apps.” We therefore see reports such as that shown here as being valuable to work on specific mHealth features and the needs of users and patients. However, other works also deal with concrete examples of COVID-19 apps. For example, in [34], regulatory aspects are discussed within the context of contact tracing apps, whereas in [35], voice assistants during the pandemic are discussed. The readiness of the healthcare system during the pandemic and mobile health apps is also discussed by these works [36,37]. They conclude that further efforts and examples are needed to better cope with regulatory aspects and the needs of users and healthcare professionals. In [38], a good overview combining regulatory aspects, quality, and acceptance can be found; the work was also inspired by the developments during the COVID-19 pandemic. These mostly recent works impressively show that the field of regulatory aspects in the context of mHealth is rapidly evolving, but many aspects are still challenging; therefore, we hope to contribute with this work through the discussion of two real-world examples.

5. Conclusions

Based on years of experience in the field of app development as well as regulatory software development in general, especially those that were responsible for the Corona Health/Corona Check projects, we were able to reduce the actual development times to a minimum (see above) while still achieving proper development costs. The Corona Warn app, which is more complex in terms of its efforts, had a realization time of roughly 50 days but without regulatory compliance. We estimate future app development projects as follows: with proper planning and taking regulatory compliance as well as a feasible requirement engineering process into account, it can be reliably estimated that the development time will be extended by a maximum of approximately 10% due to compliance with regulatory requirements. Experiences show that the collaboration of experienced app developers, good RE experts, and regulatory experienced and pragmatic quality managers can even accelerate development times while preserving product quality on the other hand. To conclude, the following results, set by the project management at the beginning of both projects, could be achieved by the overall app team:

• Conformity to the Medical Devices Regulation and
• Consensus on iterative development approach, including high development speed, feasible project schedules, and low overall project costs.

In total, the two Corona apps were created each in less than 4 months at proper costs, including approvals from the ethics committee of the University of Würzburg. An early contact with Google and Apple in order to be quickly included in the stores, on the one hand, and to be found easily by potential users on the other hand is the only point that could not be planned sufficiently at the beginning of both projects. If official bodies (such as the RKI or the Federal Ministry of Health) can even be involved in the development process, then, novel medical apps can be quickly made available for a pandemic. This includes the provision of quality-driven apps, the compliance with regulatory aspects, as well as the achievements of predictable costs. Currently, we prepare our framework on which Corona Check and Corona Health are based to be able to get a CE marking for future versions of our mHealth apps. The CE marking is used to be conform with the European health, safety, and environmental protection standards for products sold within the European Economic Area (EEA).