Using mapping studies as the basis for further research – A participant-observer case study
Introduction
In 2004–2005, Kitchenham, Dybå and Jørgensen wrote three papers suggesting that the concept of evidence-based practice, (as initially developed in medicine, and subsequently adopted by many different disciplines including economics, psychology, social science and most health care disciplines) should be adopted in software engineering [1], [2], [3]. By analogy with medicine, they suggested that evidence-based software engineering (EBSE) should be concerned with the aggregation of empirical evidence and should use systematic literature reviews (SLRs) as a methodology for performing unbiased aggregation of empirical results. Based on the five stages in evidence-based medicine, Kitchenham et al. [3] suggested equivalent stages for EBSE. Stages 1 to 4 are to:
- 1.
construct an answerable question;
- 2.
track down evidence to answer the question;
- 3.
critically appraise the evidence, and
- 4.
use the evidence to address the question.
Stage 5 is rather different in nature. It is about seeking ways to improve the way in which we undertake evidence-based software engineering and provides the rationale for this paper.
One of the main technologies underpinning EBSE is a rigorous procedure for searching research literature called a systematic literature review (SLR). SLRs are secondary studies (i.e. studies that are based on analyzing previous research) used to find, critically evaluate and aggregate all relevant research papers (referred to as primary studies) on a specific research question or research topic. The methodology is intended to ensure that the literature review is unbiased, rigorous and auditable. The basic SLR methodology is similar, irrespective of the discipline where it is employed; although medical standards emphasize meta-analysis (a means of statistically aggregating the results from different studies of the same phenomena) more than other disciplines (see for example [4], [5], [6], [7]).
We have been undertaking a program of case study-based research that is aimed at better understanding the role of SLRs in software engineering [8]. This is part of the Evidence-based Practices Informing Computing (EPIC) project, funded by the UK Engineering and Physical Sciences Research Council. In this paper we assess the value that mapping studies (also called scoping reviews) provide to the research community.
Mapping studies use the same basic methodology as SLRs but aim to identify and classify all research related to a broad software engineering topic rather than answering questions about the relative merits of competing technologies that conventional SLRs address [9]. They are intended to provide an overview of a topic area and identify whether there are sub-topics with sufficient primary studies to conduct conventional SLRs and also to identify sub-topics where more primary studies are needed. In our experience, there are a relatively large number of mapping studies [10], but relatively little discussion of their value as a research tool (see Section 3). Since mapping studies do not discuss the outcomes of the primary studies their relevance to practitioners is limited, so if they are to be of value at all they must deliver value to the research community. Thus, the overall goal of this paper to explore the value of mapping studies as a research tool and in particular to investigate how mapping studies contribute to further research.
We discuss the difference between mapping studies and SLRs and some examples of mapping studies and follow-on research activities in Section 2. In Section 3, we review related research. In Section 4 we present the methodology for our case study and for two additional activities we undertook to validate our initial results. Section 5 presents our results and we answer our research questions in Section 6. The conclusions are presented in Section 7.
A short version of this paper was presented at the EASE 2010 Conference [11]. This paper has extended the discussion of related work and also reports additional research aimed at validating the original results.
Section snippets
Mapping studies and systematic literature reviews
This section explains the differences between mapping studies and SLRs. We also present some examples of mapping studies and identify some benefits and problems with mapping studies.
Related research
Three recent papers have discussed the use of mapping studies in software engineering [9], [13], [31].
Petersen et al. [13] compared the methods used in mapping studies and systematic reviews and identified a set of guidelines for mapping studies. They advocate using mapping studies to map all of the literature in a topic area including empirical and non-empirical studies. They discuss the mapping study procedures related to five stages:
- 1.
Definition of research questions.
- 2.
Conducting the search for
The research question
The research question addressed by this case study is: “How do mapping studies contribute to further research?” In order to address the research question we consider two sub-questions:
- •
RQ.S1: What are the advantages and disadvantages of basing research on a previous mapping study?
- •
RQ.S2: What makes a mapping study suitable for supporting further research activities?
The first sub-question addresses the general value of the research method. It relates directly to our overall research question but
Results
This section reports the results of the original case study and the follow-on activities.
Discussion
This section discusses our research questions and the study limitations.
Conclusions
Table 7 presents a summary of the major benefits and problems with basing research on a preceding mapping study. It indicates that mapping studies can be of significant benefit to researchers in establishing baselines for further research activities. Such a baseline can be used in a variety of ways, either as the starting point for investigating research trends, or as the starting point for conventional SLRs. However, for each benefit there are corresponding problems that can arise if the
Acknowledgements
This work was funded through EPSRC award EP/E046983 (EPIC). We would like to thank all those who helped with this by providing their responses to our questionnaires particularly Dr. Afzal and Dr. Zhang.
References (43)
A review of studies on expert estimation of software development effort
Journal of Systems and Software
(2004)What’s up with metrics? – a preliminary mapping study
Journal of Systems and Software
(2010)- et al.
A systematic review of statistical power in software engineering experiments
Information and Software Technology
(2006) - et al.
A systematic review of effect size in software engineering experiments
Information and Software Technology
(2007) - et al.
A systematic review of quasi-experiments in software engineering
Information and Software Technology
(2009) - et al.
A systematic review of software fault prediction studies
Expert Systems with Applications
(2009) - et al.
Systematic literature reviews in software engineering – a systematic literature review
Information and Software Technology
(2009) - et al.
A systematic review of search-based testing for non-functional system properties
Information and Software Technology
(2009) - et al.
Evidence-based software engineering for practitioners
IEEE Software
(2005) - M. Jørgensen, T. Dybå, B.A. Kitchenham, Teaching evidence-based software engineering to university students, in: 11th...
Systematic Reviews to Support Evidence-Based Medicine
Systematic Reviews in the Social Sciences: A Practical Guide
A cross-domain investigation of empirical practices
IET Software
Systematic literature reviews in software engineering – a tertiary study
Information and Software Technology
Cited by (471)
The nexus between ICT, top-down and bottom-up approaches for sustainability activities: A systematic mapping study
2024, Journal of Cleaner ProductionA review on security implementations in soft-processors for IoT applications
2024, Computers and SecuritySecure software design evaluation and decision making model for ubiquitous computing: A two-stage ANN-Fuzzy AHP approach
2024, Computers in Human BehaviorA systematic mapping study on group work research in computing education projects
2023, Journal of Systems and SoftwareFintech research: systematic mapping, classification, and future directions
2024, Financial InnovationCollaborative research competencies in supply chain management: the role of boundary spanning and reflexivity
2024, International Journal of Logistics Management