Improving the quality of data models: empirical validation of a quality management framework

Abstract

This paper describes the results of a 5-year research programme into evaluating and improving the quality of data models. The theoretical base for this work was a data model quality management framework proposed by Moody and Shanks (In: P. Loucopolous (Ed.), Proceedings of the 13th International Conference on the Entity Relationship Approach, Manchester, England, December 14–17, 1994). A combination of field and laboratory research methods (action research, laboratory experiments and systems development) was used to empirically validate the framework. This paper describes how the framework was used to: (a) quality assure a data model in a large application development project (product quality); (b) reengineer application development processes to build quality into the data analysis process (process quality); (c) investigate differences between data models produced by experts and novices; (d) provide automated support for the evaluation process (the Data Model Quality Advisor). The results of the research have been used to refine and extend the framework, to the point that it is now a stable and mature approach.

Introduction

The choice of an appropriate representation of data is one of the most crucial tasks in information systems development. Although data modelling represents only a small proportion of the total systems development effort, its impact on the quality of the final system is probably greater than any other phase [1]. The data model is a major determinant of system development costs [2], system flexibility [3], integration with other systems [4] and the ability of the system to meet user requirements [5].

The traditional thrust of software quality assurance has been to use “brute force” testing at the end of development [6]. However, Total Quality Management (TQM) approaches suggest that it is faster and cheaper to concentrate effort during the early development phases of a product, in order to detect and correct defects as early as possible [7]. According to Boehm [8], relative to removing a defect discovered during the requirements stage, removing the same defect costs on average 3.5 times more during design, 50 times more at the implementation stage, and 170 times more after delivery (Fig. 1). Empirical studies have shown that moving quality assurance effort up to the early phases of development can be 33 times more cost effective than testing done at the end of development [9].

This suggests that substantially more effort should be spent during early development phases to catch defects when they occur, or to prevent them from occurring altogether. However, it is during analysis that the notion of software development as a craft rather than an engineering discipline is strongest, and quality is therefore most difficult to assess. There are relatively few guidelines for evaluating the quality of data models, and little agreement even among experts as to what makes a “good” data model. As a result, the quality of data models produced in practice is almost entirely dependent on the competence of the data modeller [10], [11].

In the quality management literature, the distinction is frequently made between product and process quality [12]:

• Product quality focuses on the characteristics of the product. Product quality criteria are used to carry out inspections of the finished product and detect and correct defects. This is the traditional approach to quality assurance.

• Process quality focuses on the process used to produce the product. The objective is to build quality into the production process rather than trying to add it in at the end through reviews and inspections of the finished product. The focus of process quality is on defect prevention rather than detection, and aims to reduce reliance on mass inspections as a way of achieving quality [13]. This is the TQM approach to quality assurance.

In the context of data modelling, product quality is concerned with evaluating and improving the quality of the data model (the product) while process quality is concerned with improving the data analysis process (the production process) (see Fig. 2). Product quality is most important in the context of an individual project—it is important to ensure that the data model is free of defects so that a database can be built which will meet user requirements. However process quality is more important in the wider organisational context: to improve the organisation's ability to efficiently deliver high quality information systems.

Previous research on data model quality has focused almost exclusively on product quality. A summary of approaches to quality in data modelling is shown in Table 1.

The simplest type of quality evaluation approach is where quality is defined as a list of desirable properties of a data model (e.g. [1], [14], [15], [16]). Such lists provide a useful starting point for understanding and evaluating quality in data models, but are mostly unstructured, use imprecise definitions, often overlap, and properties of models are often confused with language and method properties [17].

More comprehensive approaches to quality evaluation develop theoretical frameworks which define the key concepts underlying data model quality. Lindland et al. [17] define a framework based on semiotic theory, which defines a conceptual model as a set of statements in a language. For each semiotic level (syntactic, semantic, pragmatic) the framework defines quality goals and means to achieve them. Krogstie et al. [10] extend the framework to include a fourth semiotic level: the social level. These frameworks apply to conceptual models generally, not just data models. Kesh [18] develops a framework for evaluating data models based on ontological concepts. This framework defines criteria and metrics for evaluating the quality of data models.

The most serious deficiencies in the existing literature are:

• None of the approaches have been empirically validated in practice: all are either justified based on theory or the author(s)’ experience. Theoretical justification is limited because methods have no “truth” value—the validity of a method is an empirical rather than a theoretical question [19], [20]. Experiential justification is also limited because personal experience is subject to bias. Also, a method which works well for one person may not work for another [21].

• None of the approaches adequately addresses the issue of process quality: they define criteria and, in some cases, measures for evaluating the quality of data models (error detection) but not how to develop models in a high quality manner (error prevention).

Both of these issues are addressed in this paper.

The structure of the paper is:

• Section 2 describes the quality management framework used as the theoretical basis for this research—this represents the a priori theory being tested.

• Section 3 outlines the research methodology used to validate the framework.

• Section 4 describes how the framework was used to quality assure a data model for an application development project as part of an action research study (product quality).

• Section 5 describes how the framework was used to re-engineer the analysis process in an organisation as part of a longitudinal action research study (process quality).

• Section 6 describes how the framework was used to analyse differences in the quality of models produced by expert and novice data modellers using a laboratory experiment.

• Section 7 describes how the framework was used to provide automated support for the evaluation process (the Data Model Quality Advisor), and analyses its effectiveness using a laboratory experiment.

• Section 8 summarises the research findings and their implications for research and practice.