Guidelines for conducting systematic mapping studies in software engineering: An update
Introduction
Systematic mapping studies or scoping studies are designed to give an overview of a research area through classification and counting contributions in relation to the categories of that classification [1], [2]. It involves searching the literature in order to know what topics have been covered in the literature, and where the literature has been published [2]. Though, a systematic mapping study and a systematic literature review share some commonalities (e.g. with respect to searching and study selection), they are different in terms of goals and thus approaches to data analysis. While systematic reviews aim at synthesizing evidence, also considering the strength of evidence, systematic maps are primarily concerned with structuring a research area.
Systematic mapping studies are used by many researchers on a number of areas using different guidelines or methods. A sample of mapping studies is mentioned below with their areas of research and the guidelines used.
- •
Condori-Fernandez et al. [3] provided a mapping of the research articles on software requirement specifications combining two guidelines (cf. [2], [1]).
- •
Jalali and Wohlin [4] performed mapping of the literature available on Global software Engineering considering the guidelines by [2], [1].
- •
Barreiros et al. [5] constructed systematic maps on the published research on software engineering test beds based on Kitchenham and Charters’s [1] guidelines.
- •
Qadir and Usman [6] conducted a mapping on curriculum in software engineering using the guidelines by [2], [1].
Recently, Wohlin et al. [7] compared systematic mapping studies that were conducted on the same topic by two groups of researchers working and publishing independently. That is, the review protocols of the two reviews were developed independently. Some questions on the reliability of systematic mapping studies have been raised. For example, even though the same classification scheme has been used by the two studies the same articles have been classified in a different way. Both studies used the guidelines by Petersen et al. [2].
Overall, the number of mapping studies is continuously increasing, and there is a great interest in the methodology. To increase the confidence and reliability of mapping studies, there is a need to understand how they are conducted at large. Furthermore, given that many new insights have been gained through the conduct of systematic reviews, which have been synthesized by Kitchenham and Brereton [8], we may use this knowledge to determine the state of quality of mapping studies, taking the differences between SLRs and mapping studies into account.
This study makes the following contributions to improve systematic mapping guidelines:
- •
Assessing the current practice of conducting systematic mapping studies in software engineering.
- •
Comparing the identified guidelines for mapping studies with best practices as identified in Kitchenham and Brereton [8].
- •
Consolidating the findings to propose updates to systematic mapping guidelines.
The practical benefit is that the quality of systematic mapping studies could be improved further by consolidating the knowledge about good ides in the currently published guidelines.
The remainder of the paper is structured as follows: Section 2 presents the related work. Section 3 explains the research method used. Section 4 presents the results of the mapping of systematic maps in software engineering. Section 5 describes the updated guidelines. Section 6 concludes the paper.
Section snippets
The differences between systematic maps and reviews
Kitchenham et al. [9] contrasted the different characteristics of the process of systematic literature reviews and mapping studies. There are differences with respect to the research questions, search process, search strategy requirements, quality evaluation, and results.
The research questions in mapping studies are general as they aim to discover research trends (e.g. publication trends over time, topics covered in the literature). On the other hand, systematic reviews aim at aggregating
Research questions
The goal of this mapping study (following the guidelines in [2], [1]) is to determine how systematic mapping processes have been executed in software engineering. This leads to the following research questions (RQs):
- •
RQ1: Which guidelines are followed to conduct the systematic mapping studies in software engineering?
- •
RQ2: Which software engineering topics are covered?
- •
RQ3: Where and when were mapping studies published?
- •
RQ4: How was the systematic mapping process performed? This includes, for
Frequency of publication (RQ1)
Fig. 2 shows the number of mapping studies identified within the years 2007–2012. The first mapping study was published by Bailey et al. [32], and this was the only study in 2007. While the interest in mapping studies was moderately increasing 2008–2010, a significant increase can be observed in 2011 and 2012. Besides an increased interest, another potential reason may be the better distinction between systematic literature reviews and mapping studies. This increase in the number of mapping
Guideline updates
In this section we propose updated guidelines for systematic mapping consolidating the practices identified through this mapping study and the practices suggested in the guidelines proposed by other researchers [48], [1], [2], [56], [57], [49]. Furthermore, where applicable we also incorporate the learnings from a synthesis of research on evidence-based approaches in software engineering (cf. [8]). Thereby, important information on how mapping studies have been conducted in the past is
Conclusions
In this systematic mapping study we identified existing systematic maps and evaluated them with respect to topics investigated, frequency of publication over time, venues of publication, and the process of mapping studies.
Existing guidelines only partially represent the activities actually conducted in systematic mapping studies, hence to have a comprehensive list of activities (and thus being aware of the possible options to make informed decision to use an activity or not) motivated the
Acknowledgments
This work was partly funded by ELLIIT (The Linköping-Lund Initiative on IT and Mobile Communication). We also would like to thank Claes Wohlin for valuable comments on the manuscript.
References (100)
- et al.
On the reliability of mapping studies in software engineering
J. Syst. Softw.
(2013) - et al.
A systematic review of systematic review process research in software engineering
Inf. Softw. Technol.
(2013) - et al.
Risks and risk mitigation in global software development: a tertiary study
Inf. Softw. Technol.
(2014) - et al.
A systematic mapping study of software product lines testing
Inf. Softw. Technol.
(2011) - et al.
Software product line testing–a systematic mapping study
Inf. Softw. Technol.
(2011) - et al.
Using mapping studies as the basis for further research–a participant-observer case study
Inf. Softw. Technol.
(2011) - et al.
Tools used in global software engineering: a systematic mapping review
Inf. Softw. Technol.
(2012) Whats up with software metrics?–A preliminary mapping study
J. Syst. Softw.
(2010)Measuring and predicting software productivity: a systematic map and review
Inf. Softw. Technol.
(2011)- et al.
Software quality trade-offs: a systematic map
Inf. Softw. Technol.
(2012)