Guidelines for conducting systematic mapping studies in software engineering: An update

https://doi.org/10.1016/j.infsof.2015.03.007Get rights and content

Abstract

Context

Systematic mapping studies are used to structure a research area, while systematic reviews are focused on gathering and synthesizing evidence. The most recent guidelines for systematic mapping are from 2008. Since that time, many suggestions have been made of how to improve systematic literature reviews (SLRs). There is a need to evaluate how researchers conduct the process of systematic mapping and identify how the guidelines should be updated based on the lessons learned from the existing systematic maps and SLR guidelines.

Objective

To identify how the systematic mapping process is conducted (including search, study selection, analysis and presentation of data, etc.); to identify improvement potentials in conducting the systematic mapping process and updating the guidelines accordingly.

Method

We conducted a systematic mapping study of systematic maps, considering some practices of systematic review guidelines as well (in particular in relation to defining the search and to conduct a quality assessment).

Results

In a large number of studies multiple guidelines are used and combined, which leads to different ways in conducting mapping studies. The reason for combining guidelines was that they differed in the recommendations given.

Conclusion

The most frequently followed guidelines are not sufficient alone. Hence, there was a need to provide an update of how to conduct systematic mapping studies. New guidelines have been proposed consolidating existing findings.

Introduction

Systematic mapping studies or scoping studies are designed to give an overview of a research area through classification and counting contributions in relation to the categories of that classification [1], [2]. It involves searching the literature in order to know what topics have been covered in the literature, and where the literature has been published [2]. Though, a systematic mapping study and a systematic literature review share some commonalities (e.g. with respect to searching and study selection), they are different in terms of goals and thus approaches to data analysis. While systematic reviews aim at synthesizing evidence, also considering the strength of evidence, systematic maps are primarily concerned with structuring a research area.

Systematic mapping studies are used by many researchers on a number of areas using different guidelines or methods. A sample of mapping studies is mentioned below with their areas of research and the guidelines used.

  • Condori-Fernandez et al. [3] provided a mapping of the research articles on software requirement specifications combining two guidelines (cf. [2], [1]).

  • Jalali and Wohlin [4] performed mapping of the literature available on Global software Engineering considering the guidelines by [2], [1].

  • Barreiros et al. [5] constructed systematic maps on the published research on software engineering test beds based on Kitchenham and Charters’s [1] guidelines.

  • Qadir and Usman [6] conducted a mapping on curriculum in software engineering using the guidelines by [2], [1].

Recently, Wohlin et al. [7] compared systematic mapping studies that were conducted on the same topic by two groups of researchers working and publishing independently. That is, the review protocols of the two reviews were developed independently. Some questions on the reliability of systematic mapping studies have been raised. For example, even though the same classification scheme has been used by the two studies the same articles have been classified in a different way. Both studies used the guidelines by Petersen et al. [2].

Overall, the number of mapping studies is continuously increasing, and there is a great interest in the methodology. To increase the confidence and reliability of mapping studies, there is a need to understand how they are conducted at large. Furthermore, given that many new insights have been gained through the conduct of systematic reviews, which have been synthesized by Kitchenham and Brereton [8], we may use this knowledge to determine the state of quality of mapping studies, taking the differences between SLRs and mapping studies into account.

This study makes the following contributions to improve systematic mapping guidelines:

  • Assessing the current practice of conducting systematic mapping studies in software engineering.

  • Comparing the identified guidelines for mapping studies with best practices as identified in Kitchenham and Brereton [8].

  • Consolidating the findings to propose updates to systematic mapping guidelines.

The practical benefit is that the quality of systematic mapping studies could be improved further by consolidating the knowledge about good ides in the currently published guidelines.

The remainder of the paper is structured as follows: Section 2 presents the related work. Section 3 explains the research method used. Section 4 presents the results of the mapping of systematic maps in software engineering. Section 5 describes the updated guidelines. Section 6 concludes the paper.

Section snippets

The differences between systematic maps and reviews

Kitchenham et al. [9] contrasted the different characteristics of the process of systematic literature reviews and mapping studies. There are differences with respect to the research questions, search process, search strategy requirements, quality evaluation, and results.

The research questions in mapping studies are general as they aim to discover research trends (e.g. publication trends over time, topics covered in the literature). On the other hand, systematic reviews aim at aggregating

Research questions

The goal of this mapping study (following the guidelines in [2], [1]) is to determine how systematic mapping processes have been executed in software engineering. This leads to the following research questions (RQs):

  • RQ1: Which guidelines are followed to conduct the systematic mapping studies in software engineering?

  • RQ2: Which software engineering topics are covered?

  • RQ3: Where and when were mapping studies published?

  • RQ4: How was the systematic mapping process performed? This includes, for

Frequency of publication (RQ1)

Fig. 2 shows the number of mapping studies identified within the years 2007–2012. The first mapping study was published by Bailey et al. [32], and this was the only study in 2007. While the interest in mapping studies was moderately increasing 2008–2010, a significant increase can be observed in 2011 and 2012. Besides an increased interest, another potential reason may be the better distinction between systematic literature reviews and mapping studies. This increase in the number of mapping

Guideline updates

In this section we propose updated guidelines for systematic mapping consolidating the practices identified through this mapping study and the practices suggested in the guidelines proposed by other researchers [48], [1], [2], [56], [57], [49]. Furthermore, where applicable we also incorporate the learnings from a synthesis of research on evidence-based approaches in software engineering (cf. [8]). Thereby, important information on how mapping studies have been conducted in the past is

Conclusions

In this systematic mapping study we identified existing systematic maps and evaluated them with respect to topics investigated, frequency of publication over time, venues of publication, and the process of mapping studies.

Existing guidelines only partially represent the activities actually conducted in systematic mapping studies, hence to have a comprehensive list of activities (and thus being aware of the possible options to make informed decision to use an activity or not) motivated the

Acknowledgments

This work was partly funded by ELLIIT (The Linköping-Lund Initiative on IT and Mobile Communication). We also would like to thank Claes Wohlin for valuable comments on the manuscript.

References (100)

  • B. Kitchenham et al.

    Systematic literature reviews in software engineering–a systematic literature review

    Inf. Softw. Technol.

    (2009)
  • P. Brereton et al.

    Lessons from applying the systematic literature review process within the software engineering domain

    J. Syst. Softw.

    (2007)
  • D.S. Cruzes et al.

    Research synthesis in software engineering: a tertiary study

    Inf. Softw. Technol.

    (2011)
  • F.Q. Da Silva et al.

    Six years of systematic literature reviews in software engineering: an updated tertiary study

    Inf. Softw. Technol.

    (2011)
  • F. Elberzhager et al.

    A systematic mapping study on the combination of static and dynamic quality assurance techniques

    Inf. Softw. Technol.

    (2012)
  • F. Elberzhager et al.

    Reducing test effort: a systematic mapping study on existing approaches

    Inf. Softw. Technol.

    (2012)
  • A. Fernandez et al.

    Usability evaluation methods for the web: a systematic mapping study

    Inf. Softw. Technol.

    (2011)
  • R. Wendler

    The maturity of maturity model research: a systematic mapping study

    Inf. Softw. Technol.

    (2012)
  • T. Dybå et al.

    Empirical studies of agile software development: a systematic review

    Inf. Softw. Technol.

    (2008)
  • M. Palacios et al.

    Testing in service oriented architectures with dynamic binding: a mapping study

    Inf. Softw. Technol.

    (2011)
  • F.O. Bjørnson et al.

    Knowledge management in software engineering: a systematic review of studied concepts, findings and research methods used

    Inf. Softw. Technol.

    (2008)
  • P.A. da Mota Silveira Neto et al.

    Corrigendum to: a systematic mapping study of software product lines testing [inf. softw. technol. 53(5) (2011) 407–423]

    Inf. Softw. Technol.

    (2012)
  • F.Q. Da Silva et al.

    Six years of systematic literature reviews in software engineering: an updated tertiary study

    Inf. Softw. Technol.

    (2011)
  • B. Kitchenham, S. Charters, Guidelines for Performing Systematic Literature Reviews in Software Engineering, Tech....
  • K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, Systematic mapping studies in software engineering, in: 12th...
  • N. Condori-Fernandez et al.

    A systematic mapping study on empirical evaluation of software requirements specifications techniques

  • S. Jalali et al.

    Agile practices in global software engineering-a systematic map

  • E. Barreiros et al.

    A systematic mapping study on software engineering testbeds

  • M.M. Qadir et al.

    Software engineering curriculum: a systematic mapping study

  • B.A. Kitchenham et al.

    The value of mapping studies-a participant-observer case study

  • R. Wieringa et al.

    Requirements engineering paper classification and evaluation criteria: a proposal and a discussion

    Requirements Eng.

    (2006)
  • B. Kitchenham, P. Brereton, D. Budgen, Mapping Study Completeness and Reliability-A Case...
  • B. Kitchenham et al.

    The educational value of mapping studies of software engineering literature

  • T. Dyba et al.

    Applying systematic reviews to diverse study types: an experience report

  • M. Petticrew et al.

    Systematic Reviews in the Social Sciences: A Practical Guide

    (2008)
  • S. Jalali et al.

    Systematic literature studies: database searches vs. backward snowballing

  • S. Jalali et al.

    Global software engineering and agile practices: a systematic review

    J. Softw.: Evol. Process

    (2012)
  • A. Tahir et al.

    A systematic mapping study on dynamic metrics and software quality

  • J.W.C. Llanos et al.

    Differences between traditional and open source development activities

  • J. Sousa Gomes et al.

    25 years of software engineering in brazil: an analysis of sbes history

  • K. Petersen, N.B. Ali, Identifying strategies for study selection in systematic reviews and maps, in: Proceedings of...
  • K. Petersen et al.

    Worldviews, research methods, and their relationship to validity in empirical software engineering research

  • J. Bailey, D. Budgen, M. Turner, B. Kitchenham, P. Brereton, S.G. Linkman, Evidence relating to object-oriented...
  • S.T. Acuña, J.W. Castro, O. Dieste, N. Juristo, A systematic mapping study on the open source software development...
  • A. Arshad, M. Usman, Security at software architecture level: a systematic mapping study, in: 15th Annual Conference on...
  • J.F. Bastos, P.A. da Mota Silveira Neto, E.S. de Almeida, S.R. de Lemos Meira, Adopting software product lines: a...
  • D.S. Kusumo, M. Staples, L. Zhu, H. Zhang, R. Jeffery, Risks of Off-the-Shelf-Based Software Acquisition and...
  • T. Shippey, D. Bowes, B. Chrisianson, T. Hall, A Mapping Study of Software Code...
  • L.L. Lobato, I. do Carmo Machado, P. d. M.S. Neto, E.S. de Almeida, S.R. de Lemos Meira, Risk Management in Software...
  • M. Anjum, D. Budgen, A mapping study of the definitions used for service oriented architecture, in: 16th International...
  • Cited by (0)

    View full text