Concept for the testing of automated functions in therapeutic medical devices

2.1 Database

In this ontology-based testing approach a database containing qualitative expert knowledge forms the basis for the selection of the test cases of interest. In order to manage the complexity of modelling a wide range of medical information a systematic generalization of failure mode and effect analysis (FMEA) is used which is usually applied in risk management [8]. Since complex structures and large amounts of data are common in risk management, an adaption of the system design from FMEA as a basic structure of the acquired expert knowledge base is deemed appropriate. Following the FMEA structure of cause-effects chains, this knowledge base consists of four main elements: system components, functions, failures and resulting harms (see legend in Figure 2). Those main elements are structured hierarchically. With these components and the hierarchical structure, two systems are described: the human body and the therapeutic device. The human system can be described on the one hand by literature sources and the therapeutic device on the other hand by technical documentation. The two systems are described by dividing each into hierarchical components to which individual functions are assigned. Both expert knowledge and research data can be used to describe the interaction of these systems and the interdependencies. With the help of the dependencies, cause and effect can be determined. If a cause leads to a hazardous situation (a function cannot be performed properly) for the patient, a fault is triggered in the database. Because of the hierarchical structure a cascade of cause-and-effect faults can be mapped.

Figure 2:

Shown is the hierarchical database of components, functions and failures of both the patient and the ventilator. A harm for the patient arises from a particular combination of patient characteristics and device settings.

The result is a structured knowledge base, that can be easily extended with further knowledge in various directions, enabling a modular growth and input from different sources. The database can be used to extract a preselection of the relevant failures to find test SoI for the intended use. Based on the SoI and the intended use, patient and therapeutic device models with the necessary input parameters can be derived. The defined dependencies of the database are also used to calculate patient-dependent limit values for therapeutic device parameters.

2.2 Selection of concrete scenarios

In this step, the SoI, which contain the parameter ranges to be investigated, are converted into concrete test scenarios. In order to provide sufficient data for an informed risk assessment, the selection of the test scenarios is a critical challenge. A concrete scenario is thereby the selection of a specific value and its change over time for all parameters modeled in the database, e.g. oxygen saturation of the patient during ventilation. The problem of selecting the critical scenarios is manifold, among others due to the fact that the underlying distribution of the parameters, their dependencies and changes over time are generally unknown. This problem is amplified, by the challenge of gathering adequate patient data. Therefore, we propose a design of experiment (DoE) approach to select parameter sets similar to the concept of the automotive industry [5].

With methods of DoE, the smallest possible number of test cases can be found with simultaneously high test coverage through mathematical and statistical approaches. There are different methods to design matrix experiments with DoE e.g. D-Optimal Designs, Response Surface Designs or Quasi-Random Designs. The methods must always be balanced between the highest possible resolution and the lowest possible number of experiments. Therefore, there is no generally applicable method for a specific application. A suitable DoE method must always be selected iteratively. Starting with basic methods such as orthogonal fields to reduce the experimental effort is recommended. Those orthogonal fields are tables that contain parameter settings [9]. These are combined with each other in such a way that one can obtain results with an approximate statistical accuracy. The parameter settings result from the parameter ranges determined with the database. The single settings of the parameters are referred to as factors. A specific set of factors is one scenario to be evaluated. The number of tested factor combinations in relation to all possible scenarios is called a fraction. For a full fraction N parameters with F factors in the parameter space require N ^F scenarios. To reduce the amount of the scenarios and the associated runtime, the quantity of factors and fractions can be systematically reduced.

Furthermore, the influence of a single parameter can be determined by using interaction effects matrix plots [9]. This information can be used in further iterations to optimise the results. In addition, determined non-linear and dynamic interactions of parameters can be taken into account.

2.3 Simulation

For the simulation, the patient and therapeutic device models determined from the database and the parameter settings of the individual trials of the DoE are used. The therapeutic device model uses the therapeutic device parameters to generate the input signal for the patient model, with which the various patients are represented. The input signal corresponds to theoretically possible behaviours of the HAF. In the simulations the different trials of the DoE are then performed virtually.

The patient and therapeutic device models can be iteratively extended and adapted on the basis of the database and the intended use. Other patient characteristics such as diseases or different therapy modes can be added. Further dependencies between patient and therapeutic device can also be included in the database. This can then be taken into account by extending the models for the simulations. The simulations result in a data set that includes the response of the individual patients to the different behaviours of the therapeutic device. The behaviour of the therapeutic device corresponds to the possible behaviour of a HAF. With this method, not only the desired behaviour is depicted, but also possible over and under reactions are simulated.

2.4 Risk analysis

In the risk analysis, the simulation results are evaluated. For this purpose, values for characteristic ventilation parameters are first determined from the physiological models and simulation curves (e.g. CO₂ concentration, pressure, volume). Characteristic parameters are used as a basis for examining the assumed behaviour of a potential HAF with regard to possible patient harms. The desired ranges of the characteristic parameters of an HAF are defined individually for each patient. For this purpose, the dependencies described in the database can be used. Now the characteristic parameters can be used to determine whether the theoretically possible behaviours of the HAF are within the desired range or whether limit values have been exceeded. If limit values are exceeded, the risk for the patient is determined. The risk results from the severity of harm caused by not complying with the limit values and the probability that this scenario will occur in practice. This risk evaluation can be done in accordance with ISO 14971 [10] using a risk acceptance matrix. With this a decision can be made whether the risk of a scenario is acceptable. Scenarios in which an insufficient function of an HAF must be excluded because the risk for a patient would be too high are examined in physical simulations.

2.5 Bench test

In order to ensure the correct function of the HAF, system level tests are carried out on a test bench, capable of emulating the physiological system at the interface to the device. This requires a highly flexible simulator design, in order to represent the large number of potential scenarios. Such a simulator could eventually also be used, for a replacement of animal trials in pre-clinical evaluation and research [11].

2.6 Safety argumentation

The aim of the concept shown for testing HAF is to develop a safety argumentation. For a comprehensive statement about the safety of the functions, a wide range of parameter setting, linear and non-linear behaviour and interactions of parameters need to be taken into account within the database. The more properties and dependencies are modelled and analysed, the more precise statements can be made. According to DIN 60601-1 [12] however, risks resulting from data gaps or incorrect data must be taken into account.

For the DoE step it is important that every theoretically possible behaviour of the automated function is assumed in order to be able to make an informed and reliable assessment of the safety. When DoE is applied, it must also be considered that an adequately high resolution is chosen for sufficient test coverage. Insufficient test coverage can weaken the safety argumentation.

The simulations are based on patient models. With regard to the safety conclusion, it should be mentioned here that these models are a replica and do not exactly represent the behaviour of a real patient. Credible patient models and lung simulators would have the advantage of increasing the validity of the safety argumentation [11]. The safety argumentation is highly dependent on the accuracy of the models and simulators, and may not be reliable if real-world conditions are not adequately represented. This dependency must be taken into account in the safety argumentation according to ISO 14971 [10]. If the HAF does not show any undesired behaviour in the bench tests and the risks of the single steps of the proposed test concept are acceptable in accordance to ISO 14971 [10], then a conclusion can be made that the relevant cases tested are very likely to be safe. Pre-clinical tests could thus be reduced.

3.1 Example database

As a first iteration, a small model was set up using the FMEA tool Plato e1ns (PLATO AG, Lübeck, Germany), containing all relevant system components from the respiratory system and the ventilator system, as well as their functions and failures. This model was used to investigate cause and effect relationships. The model is represented in a tree structure conceptually depicted in Figure 2. From the dependencies described in this model, it can e.g. be concluded that a person’s BMI in combination with certain ventilation parameters can be a cause of various harms. It may be further analyzed that the pathophysiological changes of the respiratory system contribute significantly to an increased risk of complications in patients with a high BMI during ventilation.

Obesity is considered a widespread risk factor for diseases and roughly 67% of men and 53% of women are overweight in Germany [14]. This results in a very high relevance for everyday clinical practice. The probability of obesity in ventilated patients is therefore high. It follows that for the SoI, the patient parameters of height and weight are relevant.

The respiratory rate (RR) and ΔP (difference between plateau pressure and positive end exspiratory pressure) can be set by the ventilator. A scenario consists of a chosen combination of values of these parameters and additional constant parameters within the mathematical model representing the patient and his therapy setting. The range of possible values of the single parameters is stored in the database.

3.2 Example selection of concrete scenarios

In order to illustrate the selection of concrete scenarios, the parameters weight, height, RR and ΔP selected with the database are used. The chosen value ranges of the variable parameters are listed in Table 1. On the patient side, value ranges were chosen that lead to a BMI for an overweight person. For the ventilator parameters, the range that can occur in practice was used. With these four parameters and for example ten factors, 4¹⁰ scenarios would result. This corresponds to a full fraction with a high resolution because of the ten factors. To reduce the simulation effort an iterative approach is used to find a suitable sub-fraction and a representative number of factors [9]. For this first iteration the chosen method is the Box–Behnken design. Each row in the generated output matrix contains the settings for all listed parameters, either −1 (minimum), 0 (mean) or 1 (maximum) [15].

For four parameters the Box–Behnken design yields an experimental design that contains 27 trials. The individual trials of the experiment are centered between the worst-case combination of extreme cases of the enclosed space. These worst case combinations are omitted from the experiments, so that this does not result in extreme cases, which only occur with a negligible probability, but in boundary cases, which are more likely to be possible. An extract of the first seven trials of the generated output DoE-matrix is shown in Table 2.

3.3 Example simulation

For the simulation of the concrete scenarios, a one-compartment model as shown in Figure 3 used as a simple example for patient modelling [16]. The model can be extended to represent other aspects of patient behaviour. The one-compartment model, displayed in Figure 3, contains a pressure source (P _aw), a resistance (R) and compliance (C), which are connected to

Figure 3:

RC-model with airway pressure (P _aw), respiratory muscle pressure (P _mus), respiratory resistance (R), flow ( V ̇ ) , lung compliance (C) based on [17].

In general the airway resistance is defined as the pressure difference between the mouth (atmospheric pressure) and the alveoli of the lungs, divided by the airflow. At high body weight, abdominal fat accumulation affects lung mechanics, thus a BMI dependency exists. The airways can also be narrowed by fat deposits [18]. Therefore, the resistance of the airway (R) is increased and the compliance of the lung (C) is decreased [19]. Pelosi et al. [19] descried a simple heuristic relationship between BMI and airway resistance (R) as follows

The airway resistance (R) expressed in cm H₂O  s L⁻¹ describes the flow resistance of the entire airway system. It depends on the dimensions of the upper airways, the pulmonary airway tree and the acini with its alveoli [16]. A high BMI, i.e. obesity increases breathing resistance. This indicates that the caliber of the airways are reduced during the entire tidal breathing cycle [20]. In addition, Pelosi et al. [19] described a simple heuristic relationship between BMI and lung compliance (C) as follows

Pelosi et al. [19] demonstrated that lung compliance (C) expressed in mL  cm H₂O⁻¹ decreased with increased BMI, and decrease was observed even with a small increase in body mass. Using the respiratory rate (RR) expressed in min⁻¹, the time for one breathing cycle t _tot can be calculated according to

and is expressed in s as well as t _i and t _e . The inspiratory expiratory ratio (I:E ratio) is set to 1:1.5 [21]. By inserting t _e = 1.5 · t _i into the equation

and solving for t _i , we obtain

The factor 2/5 is given by the I:E ratio. The airway pressure (P _aw) is assumed to be a square wave and depends on the parameters PEEP, ΔP, t _i and t _e . The P _aw curve was smoothed to be closer to physiological conditions (see Figure 4). The respiratory muscle pressure (P _mus) is set to zero as a fully sedated state is considered with no muscle activity. The PEEP, which is not considered a relevant parameter for the function under consideration in this simple example, is fixed to the lowest suitable value of 5 cm H₂O, cf [21]. In Figure 5 the plots of the first seven trials can be seen. The differential equations (see Eq. (1)) need to be solved to plot the volume curves (see Figure 5) of the trials of the DoE. Based on the simulation, the minima and maxima of the respective curves were determined. The tidal volume (V _T ) expressed in L indicates the amount of air that enters the patient during normal inspiration [22]. To calculate the tidal volume (V _T ), the respective minima is subtracted from the corresponding maxima. The minute volume

expressed in L min⁻¹ is the product of the respiratory rate (RR) and the tidal volume (V _T ) [22].

Figure 4:

Plots of the first seven exemplary P _aw square waves for the DoE’s respective trials. The P _aw square waves are smoothed. The plots relating to trials 1 to 4 are superimposed. The plots are shown over the time section between 13 and 21 s. The plots of the experiments are differentiated by color and line structure.

Figure 5:

Plots of the first seven exemplary trials of the DoE. The plots of trials 1 to 4 are superimposed. The plots are shown over the time section between 13 and 21 s. The plots of the experiments are differentiated by color and line structure.

3.4 Example risk analysis

Threshold values calculated from the parameter settings are determined for this example. For this purpose, the ideal body weight (IBW) is calculated for each given patient based on height and gender. For the experiments, a male patient was assumed. The ideal body weight for men (IBW _M ) is expressed in kg and is calculated with

The height and the value 152.4 are expressed in cm. The factor 0.91 is expressed in kg cm⁻¹ and the value 50 is expressed in kg. For the ventilated patients, a V _T of 0.006–0.008 L kg⁻¹ standard body weight is set [21]. Multiplying IBW by 0.008 L kg⁻¹ results in

and multiplying IBW by 0.006 L kg⁻¹ one obtains

In general the tidal volume V _T is expressed in L.

To determine the maximum minute volume

one uses V _Tmax and for the minimum minute volume

one uses V _Tmin. The individually calculated parameters MV and V _T are compared with the respective determined threshold values. The result for V _T is display in Figure 6. The evaluation of MV is carried out analogously. Based on these results, a risk assessment is carried out. The risk results from the combination of the severity of the harm and the probability of occurrence (see Figure 7). The severity of the harm is assessed by how much it deviates from the desired area. The probability of occurrence of harms is composed of the probability of occurrence of obesity in combination with unfavourable ventilation parameters. In this example the intended use of the HAF is the ventilation of patients with a high BMI. The occurrence of obesity is therefore always present. The time of use of the HAF during therapy can be used to evaluate the probability of occurrence of faulty ventilation parameters. Since the function is used permanently, there is a continuous possibility of selecting incorrect parameters for ventilation.

Figure 6:

The calculated tidal volumes for all 27 experiments performed. For visualization, the different V _T are normalized to the patient-specific limits (V _Tmin, V _Tmax).

Figure 7:

Shown is the risk that arises for the simulated patients depending on the probability and severity of harm.

If a concrete scenario (see Figure 7) results in an unacceptable risk, the scenario is critical and must be evaluated in a physical simulation, e.g. on a test bench. The manufacturer addresses the identified risks as part of the risk management process. This implies, that depending on the type of risk, the intended use must be restricted or a redesign is necessary.

3.5 Example bench test

With the bench tests the critical scenarios are simulated and it is tested whether the HAF can handle those situations. This includes also the option of alarming. These tests can be performed on lung simulators which generate the simulated physical conditions on the interface of the actual therapy device running the HAF. A lung simulator is being developed for this bench test. The concept of the lung simulator by Bautsch et al. [23] includes a system that can represent a wide range of scenarios. Other commercially available lung simulators maybe used as well depending on their functionality and the HAF under test.

3.6 Example safety argumentation

Under the consideration of two conditions the safety of the HAF may be assessed and potential weak spots and faults be addressed. The first aspect investigates that the HAF can handle the critical scenarios in the physical simulation. The other aspect corresponds to compliance with ISO 14971 [10] throughout the entire testing process. This includes taking into account the risks that arise from the individual test steps as described in Section 2.6.

For this example a conclusion about which behaviour of the automated function could lead to damage can only be made for the defined range of overweight, e.g. of a BMI of 25–50 kg m⁻², as defined as the SoI within the database. This would therefore also correspond to the scope of application for this example.

The chosen Box–Behnken design operates with maxima, minima and mean values and therefore gives a good overview of the borders of the parameter space. However, with the Box–Behnken design only the borders and the centre can be examined. Intermediate states or local critical states cannot be investigated with this method. To close this data gap for compliance with DIN EN 60601-1 [12] further iterations of experimental designs are necessary, in which other parameters and different types of dependencies of these parameters can also be taken into account. For example, an increased BMI tends to have a positive effect in combination with chronic obstructive pulmonary disease (COPD).

The one-compartment model used in this example implementation corresponds to an approximation of patient behaviour. The risks that this simplification entails must be taken into account. This also applies to test benches that are only an approximate representation of human physiology.