Next Article in Journal
Improved 2D Coprime Array Structure with the Difference and Sum Coarray Concept
Previous Article in Journal
Argus: Traffic Behavior Based Prediction of Internet User Demographics through Hierarchical Neural Network
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Electronics
Volume 9
Issue 2
10.3390/electronics9020272
electronics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Computer Network Simulation with ns-3: A Systematic Literature Review

by
Lelio Campanile
1,
Marco Gribaudo
2,
Mauro Iacono
1,*,
Fiammetta Marulli
1 and
Michele Mastroianni
1
1
Dipartimento di Matematica e Fisica, Università degli Studi della Campania “Luigi Vanvitelli”, 81100 Caserta, Italy
2
Dipartimento di Elettronica, Informatica e Bioingegneria, Politecnico di Milano, 20133 Milano, Italy
*
Author to whom correspondence should be addressed.
Electronics 2020, 9(2), 272; https://doi.org/10.3390/electronics9020272
Submission received: 31 December 2019 / Revised: 24 January 2020 / Accepted: 28 January 2020 / Published: 5 February 2020
(This article belongs to the Section Networks)
Browse Figures
Versions Notes

Abstract

:
Complexity of current computer networks, including e.g., local networks, large structured networks, wireless sensor networks, datacenter backbones, requires a thorough study to perform analysis and support design. Simulation is a tool of paramount importance to encompass all the different aspects that contribute to design quality and network performance (including as well energy issues, security management overheads, dependability), due to the fact that such complexity produces several interactions at all network layers that is not easily modellable with analytic approaches. In this systematic literature review we aim to analyze, basing our investigation on available literature, the adoption of a popular network simulator, namely ns-3, and its use in the scientific community. More in detail, we are interested in understanding what are the impacted application domains in which authors prefer ns-3 to other similar tools and how extensible it is in practice according to the experience of authors. The results of our analysis, which has been conducted by especially focusing on 128 papers published between 2009 to 2019, reveals that 10% of the evaluated papers were discarded because they represented informal literature; most of the studies presented comparisons among different network simulators, beyond ns-3 and conceptual studies related to performance assessment and validation and routing protocols. Only about 30% of considered studies present extensions of ns-3 in terms of new modules and only about 10% present effective case studies demonstrating the effectiveness of employing network simulator in real application, except conceptual and modeling studies.

1. Introduction

Computer networks are currently a cornerstone not only in traditional computing or business environment, but also in many different application fields, such as cloud facilities, Industry 4.0, Wireless Sensor Networks (WSN), CyberPhysical Systems (CPS), 5G communication systems, critical infrastructures protection, automotive, railways, military applications such as ground support, Command, Control, Communications and Intelligence (C3I), modern military air force systems, and many other possible examples. Computer networking took the place of other traditional technologies with the aim of providing richer services, adaptation to different conditions, reconfigurability, system-wide and node-wide intelligence, flexibility, interoperability, thanks to the fact that network architectures are layered, scalable, evolvable and adaptable, and to the intrinsic advantages of digital data orientation. The modularity of network technologies allows the independent development of standards that focus on individual problems and allow reuse of solutions and concurrent design of different layers for the same architecture or for different architectures to be made compatible or interoperable.
Modern networking technologies include hardware and software components. The availability of more reliable and faster computing hardware is changing the balance between hardware and software and the structure of network devices as well: currently, both embedded, mainly hardware-based nodes may coexist with mainly software-based nodes, in which analogous functions are provided with different tools, and Software Defined Networks (SDN) are spreading aside conventional hardware-based infrastructures, basically reducing the logical portion of a network that is necessarily implemented in hardware to the essential components that ensure the connection. This evolution, mainly pushed by the needs of cloud infrastructures and large computing infrastructures in general, such as warehouse scale computers, to allow easier deployment, control and management of complex networks, in turn enables a change of paradigm from distributed network control and logic to a centralized approach, including the ability of a nearly complete reconfiguration of all network nodes in a large installation by a single control and management node. In some scenarios, moreover, the higher levels of the network stacks might be completely implemented in the cloud, such as in the case of 5G technologies that blur the communication and computing portions of the system, or, less extremely, 4G technologies that delegate some functions from the cell antenna to the terminals.
When analyzing or integrating a computer network, this heterogeneity in hardware and software and the wide transparency in the interoperability of components results in a potentially lower knowledge of non-controllable nodes. The higher the number of vendors, the number of interacting technologies, the dimension and extension of the network, the amount of numbers, the heterogeneity in the services and layers, the security requirements, the survivability requirements, the more difficult is to model and understand faithfully the dynamics of the network. A low fidelity impacts significantly the results of assessments when the complexity of the network is not trivial, so the availability of modular simulation-based tools to support the process is a real need to avoid dramatic underestimation of network problems or erroneous assumptions on network behavior that may have serious consequences: for example, in evaluating the extension of the attack surface in a IoT system, the reliability of an Industry 4.0 installation or the performance of multimedia delivery systems.
The use of modular simulation allows building a coherent model of a network with different levels of detail. The level of detail may, in principle, vary between high-level, behavioral modeling and emulation. The availability of modular, extensible, programmable, open, open-source, community-supported and community-driven simulation tools or simulation frameworks gives the modeler the possibility of producing simulation scenarios with any desired level of detail, even with an heterogeneous level in different simulated nodes, portions of network or subsystems, provided that a component for the simulation tool is available or autonomously developed by the modeler for the purpose. The extensibility allows the definition of new components for experimental or future real-world networks components, as well as the implementation of different versions of the same real-world networks component with a different inner complexity to manage with the duration of large simulation runs or to interface, in case, real-world networks components with simulated components. Modularity and extensibility enable component reuse, thus empowering the modeling process with well-tested, well-documented third-party components with little effort by the modeler, faster model development, incremental modeling strategies, comparative simulation for cross-validation by means of alternative modules, parameterization of models, definition of semifinished models or model portions, up to the support for structured modeling and analysis processes, including standardized processes in the framework of industrial certifications. Openness guarantees the legal rights needed for extension, while open-sourceness guarantees inspectability of code for debug, verification or certification of simulations, or easy refactorization of components to evolve them with the simulation software infrastructure, and portability between version. Community-supportedness fosters the collective effort to produce or improve new components, to augment the usefulness of the simulation tool and to strengthen the effectiveness of its use into simulation processes, or, together with open-sourceness, the implementation of third-party tools to support or automate simulation campaigns, simulation generation, model-based simulation development or other useful tools that ease or empower the management of simulation and its application in structured development and assessment processes and methodologies.
Many alternatives exist for computer network simulation: in this Systematic Literature Review (SLR) we focus on ns-3 (www.nsnam.org/), a modular, programmable, extensible, open, open-source, community-supported simulation framework for computer networks. Our choice is due to the fact that the technical characteristics of ns-3 match the needs of our research activities, and to the increasing popularity that this tool is gaining among the computer networks research community and, slowly, among professionals.
ns-3 is an open computer networks simulation environment that is based on discrete-event simulation. In such a simulator, each event is associated with its execution time, and the simulation proceeds by executing events in the temporal order of simulation time. When an event is processed, it may generate zero, one or more events. As a simulation executes, events are consumed, but more events may (or may not) be generated. The simulation will stop automatically when no further events are in the event queue, or when a special “Stop” event is found [1].
It is designed for research use, thinking of the needs of the research community, and fosters a community-based collaboration model both to support the development of new modules and to perform validation or peer review activities across different groups. ns-3 is distributed under an open-source model and a free software paradigm. The simulation environment is designed to allow its use as a real-time emulator as well, integrating simulated portions and physical portions in the same network. This level of integration is provided by a real-time scheduler, besides the simulation support, to enable simulation-in-the-loop. The simulator is intended to be used by means of its API inside programmatic descriptions of users’ scenarios and configurations, which may be developed following usual workflows and processes. The API is used to produce outputs in terms of traces that are coherent with the traces that may be obtained by observing traffic in real-world networks with tools like Wireshark (https://www.wireshark.org/) or TCPdump (https://www.tcpdump.org/), so that they may be analyzed by the same tools that are already in use for experimental setups or diagnostics. The authors of ns-3 explicitly state that they aim at allowing the reuse of real-world implementation of protocols in ns-3, so that the effort for the design and production of very detailed simulation components is reduced as much as possible. The goal is to provide a very realistic simulation support for a large number of different type of network infrastructures, communication and routing protocols, technologies, layers, allowing network-enabled software to run on top of the simulation with minimal effort (e.g., virtual machines).
ns-3 is the result of a long experience that evolved through two previous simulators (namely ns-1 and ns-2): however, the development team decided to completely restart from scratch the design and development process with ns-3 on different premises, with no backward compatibility, actually giving birth to a completely new artifact but keeping and leveraging the experience from previous versions. ns-3 simulations may be written in C++ and Python, and their basic setup and execution is guided by a process suggested by the development team. ns-3 has been designed to be scalable.
We believe that ns-3 has a significant potential and we expect its community to grow. We compared it against alternatives, and our internal evaluation led us to choose ns-3 over them. However, after a test period on several scenarios that are in the scope of our research activities, we decided to corroborate our evaluation with a SLR before using it to support the activities of a research project.
Our investigation aims to understand what are the actual diffusion and the actual use of ns-3 in the research community, also to leverage literature to individuate possible weaknesses and limits of the tool that may affect the progress and the result of our research activities, and to classify the application areas in which adoption is wider and potential community support is stronger, more reactive and more intense, as well as the practically available modules and extensions that have been published and integrated. We also decided not to contact or involve the ns-3 development team while conducting this SLR to avoid any bias, to actually rely on the scientific papers we were able to find, identify and obtain by ourselves by using the main research papers search engines, listed in Table 2: by the way, the development team does not provide a list of papers mentioning ns-3 to be used for a SLR. We excluded non-indexed documents to be sure that our sources have been peer reviewed or at least referred or considered to be reliable by the scientific community. We considered all scientific papers indexed by the indexes between 2009 and mid October 2019.
This paper is organized as follows: in Section 2 a set of popular networking oriented simulators is presented and examined; in Section 3 the review process is described and the methodology is presented that has been used for this review; in Section 4 the research questions that guide this study are presented and discussed; in Section 5 the methods for conducting the study are detailed; in Section 6 the results of the study are provided and commented; finally, Section 7 draws the conclusions resulting from the study.

2. Related Work

Simulation is a widely accepted tool in the field of computer networks. Literature offers both free and commercial simulation tools that aim at supporting, at different levels and with different purposes, design and analysis of computer networks. With no claim to completeness, this Section presents some relevant cases to provide a first reference to the readers.
Within the category of open-source simulators, besides ns-3 literature reports ns-2, OMNET++ and SWANS as noticeable examples.
ns-2 (https://www.isi.edu/nsnam/ns/), actually the previous version of ns-3, deserves to be mentioned as a different tool, because it is still widely used notwithstanding it is not maintained anymore and its last version has been released in 2011, since, as previously reported, ns-3 has been completely redesigned on a different basis. ns-2 implements discrete-event simulation. ns-2 is characterized by a separation between the approach to the definition of the simulated components and the approach to the definition of simulation management and setup: the first is based on a compiled software base, programmed in C++, including the components defined by the user or third parties, if any, while the second is managed by interpreted code written in OTcl, an object-oriented extension of the popular Tcl language, to ease the process. ns-2 has a significant base of available simulation components, due both to its popularity and lifespan, including protocols, reusable simulation objects that represent real-world equipment or routing algorithms. Support exist for wired and wireless networks, mobile networks, generation of specific patterns of traffic and energy-oriented evaluation. Its wide adoption fostered the development of third-party tools that partially compensate its limited support for visual management and analysis.
OMNET++ (https://omnetpp.org/) is an event-based simulator that aims to provide a generic support to all kind of networked systems, from mobile networks to on-chip networks. The core of the simulator is a C++-based kernel that provides several services on which different application areas are made available by additional third-party frameworks, including very different cases such as photonic networks or sensor networks, or more abstract system performance modeling tools such as queuing networks: the official website reports a very comprehensive list of domains that have been covered by users and made publicly available under open-source licenses. OMNET++ provides a proprietary scripting language to describe topologies in model setups and can be used both from the command line and a rich graphical user interface. Models and components may be specified by an event-based paradigm and a process-oriented paradigm. The simulation kernel, designed for portability, allows distributed and parallel execution to support large simulation scenarios and speed up the runs. The simulation definition process is integrated in Eclipse to ease setup and management, and interactive simulation is provided. OMNET++ supports analysis of performance metrics by visual tools.
SWANS (http://jist.ece.cornell.edu/) is a simulator for wireless networks and sensor networks that has been designed with special attention to allow high simulation performance and larger simulation scenarios with respect to ns-2 and GloMoSim, the open-source version of the QualNet simulator described later in this section. SWANS uses JiST, a Java-based high-performance discrete-event simulation engine, as simulation support. The main aim of SWANS is scalability. Its architecture is component-based, and it can run Java network applications over simulated networks. The last version has been released in 2005.
Within the category of commercial simulators, the market offers tools such as NetSim and QualNet.
NetSim (https://www.tetcos.com/) is a tool that can be licensed for professional, research or teaching use. It is reported by developers to be or have been used in different professional environments such as military and space organizations, utilities distribution companies, railways industry, network equipment manufacturers and services providers for the design, development and evaluation of a rich variety of computer and communication networks. NetSim provides user friendly tools to support the various activities and allows both simulation and emulation. It provides packet animation and a native analysis support. Simulations can be based on existing modules or user defined modules written in C.
QualNet (https://www.scalable-networks.com/qualnet-network-simulation) is designed to allow users to test, be trained on or plan large realistic networks by means of simulation (the same company provides an emulation tool as well). QualNet is originally based on GloMoSim and supports operationally accurate contexts. It provides a suite of tools for the configuration, the tracing and the analysis of traffic events.
In Table 1 the main characteristics of cited network simulators are shown.
It is worth noting that literature also reports the implementation of network simulations based on general purpose simulation frameworks that have not been considered in the scope of this analysis.

3. SLR Process

A SLR is a methodology designed to evaluate and obtain information about a specific argument and about one or more punctual research question(s). A SLR should provide an accurate and rigorous analysis of the topic of interest by means of a well-documented and standard process that evolves into three main phases (see Figure 1):
  • Planning
  • Conducting
  • Reporting the results

3.1. SLR Target

This work wants to study in a systematic manner the research literature about ns-3 in the period between 2009 and 2019. The aim is to find the main research areas in which it has been used and how it could adapt, in an effective way, to new fields of application, by the development of new modules or extensions.

3.2. SLR Protocol

The review protocol used in this work is a slightly modified version of the protocol used in [2], because it was already validated, and it worked well for our purpose: it is depicted in Figure 2.
The review protocol enactment started from the definition of the search query that has then been executed to retrieve data by a scientific literature search engine (namely Scopus) to automatically find the most relevant papers. This first list of papers was augmented with manually selected papers considered of special relevance and already known to the authors; all abstracts and conclusions sections of the papers in the list have been read to select which of those paper were relevant for the SLR. Finally, these selected papers have been carefully examined and contents have been evaluated by using of the data extraction form to collect the most relevant data.

4. Research Questions

To manage the goals of our review, we defined two research questions to guide the process:
  • RQ1: which are the impacted application domains, according to the literature?
  • RQ2: how easily extensible did ns-3 prove to be with respect to its adaptation to new domains or uses?
As we aim at providing a way to introduce to ns-3 to potential users (like our research group) that do research activities on which it could have a positive impact, the purpose of the two questions is to document that the scientific community actually uses it in several areas and that those areas have been covered by native or third-party modules or extensions, so it is likely that other areas that may be of interest for potential users may be easily covered in the future or by investing in the development of new modules or extensions in first person.
RQ1 has been used to search and categorize the main research areas in which ns-3 has been successfully adopted: from this question three other refining questions have been derived that help to categorize with more details the use cases:
  • RQ1.1: what kind of network technologies have been simulated?
  • RQ1.2: which are the related application field? (e.g., medicine, astrology, ...)
  • RQ1.3: what kind of problems have been addressed?
RQ2 tries and measures qualitatively how simple and effective is to implement custom modules and extensions to use ns-3 in new application areas or in a way it was not designed for. This being a qualitative measure, the related answer derives from an evaluation, done by this authoring team, that is partially founded onto explicit elements provided in the papers by their authors, partially founded on the technical descriptions provided in the papers and partially founded onto the number and frequency of presented extensions in the literature. Two other refining questions have been derived:
  • RQ2.1 Is it a new extension or module?
  • RQ2.2 Does it use an external module or extension?

5. Review Methods

5.1. Data Source and Search Strategy

We searched through the most important scientific literature search engines to retrieve a wide perspective on the topic, keeping as main reference Scopus and integrating results with search results from the other databases.
The reasons behind this choice is the will to consider all major journals, conferences and workshops that are considered by the scientific community relevant to the topic or are declared as relevant by publishers or editors, but filtered and included by well-known indexes. The set of examined papers is the union of the set of papers obtained by the queries from each database. We decided to exclude all informal literature (power point slides, conference reviews, informal reports, work in progress and technical notes) because this kind of literature is difficult to evaluate in terms of quality assessment: this choice gives our analysis more stability and better guarantees about the quality of the selected papers. Table 2 shows the used databases.
To identify our set of documents, we submitted some “free keywords”-based queries in 4 Research Documents Portals, namely Google Scholar Search Engine, Elsevier Search, Springer Open Access, IEEE Explore Digital Library (IEEE E.D.L) and The ACM Digital Library. All the queries were submitted filtering documents in the English language only. As first, we tried with the only keyword “ns3”: all of these search portals provided research papers dealing with a unique topic related to medicine and chemistry domain; almost in the most of these documents, the keyword “ns3” is an acronym standing for a multifunctional inhibitor protein for the hepatitis C virus. Conversely, adopting the bi-gram “ns-3”, which is the correct spelling of the name of the simulator, we just observed different behaviors for the considered search portals. A numerical comparison between the number of documents retrieved by adopting the “ns3” and “ns-3” keywords over the 4 considered sources, for the English language and in the time interval in 2009–2019, is summarized in Table 3.
Elsevier and Springer exhibited the worst behaviors, since they were not able to provide a fine-grained selection of documents, even if the keywords changed. Springer retrieved 21 documents for the “ns3” keyword and only 2 over 21 were related to Computer Science (e.g., we retrieved the 2019 Lecture Notes in Networks and Systems), even if the research category filter “Engineering” was applied. Conversely, looking for keyword “ns-3”, the number of retrieved documents amount to over 70 thousands but no matching documents to the network simulator were found yet. Documents retrieved by Google Scholar well represented the discrepancy among the “ns3” and the “ns-3” keywords. In the first case, the most of retrieved documents were related to the inhibitor protein; in the latter case, the most of documents matched the network simulation domain and were compliant to our research aim. Also results obtained by IEEE E.D.L. were interesting. Unlike Google Scholar, it was able to retrieve a significant number of relevant documents to the network simulator ns-3 both with “ns3” and “ns-3” keywords. The ACM Digital Library reported respectively 394 and 290,550 papers. Finally, we applied a cross selection strategy over all the relevant documents we retrieved by submitting the described queries to the 5 sources; we adopted, in order to select the 112 analyzed documents, as selection criteria the number of references and citation, the novelty of proposals for ns-3 modules extensions and the applications detailed in the following Sections.
The final search strategy followed to form the searching query makes use of the traditional 1998 version of the ACM Category classification as in Table 4 (available at https://www.acm.org/publications/computing-classification-system/1998/ccs98). We decided to use this categorization to leverage an authoritative, well known and widely accepted set of reference criteria and to avoid any kind of bias that might have been influencing the selection process because of our knowledge about the topic or our personal research experiences, points of view and perspectives. We wanted to avoid a search strategy that could be too subject to our personal understanding of the topic, which is driven by the way in which we approached it in the past, and may have suggested biased keywords for queries. For the same reason, we decided to give a primary role to Scopus as main reference for retrieving the starting set of papers, as Scopus indexes papers from different publishers, from different kind of sources and from different disciplines, while others are limited to own publications and tend to suggest more papers than the ones that seemed actually relevant to our purpose for us. Analogously, we decided to use the old version of the ACM classification with respect to the new one because it is used by other publishers and institutions than ACM, and it does not especially privilege papers published by ACM in the considered period, as the new version has been adopted in 2012.
To obtain the standard set of keywords for the definition of queries, we selected Topic C, “Computer Systems Organization”, with sub-topic C.2, “COMPUTER-COMMUNICATION NETWORKS”, so we performed our queries referring to the sub-sub-topics of C.2 listed in Table 4, using the most significant words as keywords.
Based on this classification, equivalent queries have been defined for each index with the needed syntax. For example, the query defined for Scopus is:
“(TITLE-ABS-KEY (ns3 AND computer AND communication AND networks) AND ((network AND architecture AND design) OR (Network AND Protocols) OR (Network AND Operation) OR (Distributed OR System) OR (local OR wide AND networks) OR (internetworking)))”.
The query has been obtained by considering the keyword “ns3” and relevant keywords from Table 4. As stated previously, Scopus has been chosen as the main reference because of its primary role, because it indexes papers from different publishers and because its filtering mechanisms was a better fit to obtain an appropriate coherence between resulting papers and relevance to the focus; the other databases have been then manually searched with similar ad hoc queries to look for relevant results not indexed by Scopus, as they tend to expand the results set with borderline or non-relevant papers.

The WNS3 Workshop on ns-3

The WNS3 workshop (https://www.nsnam.org/research/wns3/) is a yearly meeting that gathers developers and users of ns-3, with proceedings published by ACM in The ACM Digital Library. This venue is of paramount importance for the community that develops and supports ns-3, it is the flagship meeting of the ns-3 Consortium and hosts important training sessions. We carefully considered all 90 papers published by the workshop as from the web site, in order to find out if they match our needs, and one of the authors participated in 2019 to get to know the community and have an hands on approach to ns-3, its applications and the spirit behind it. A first analysis shows that the most of the papers deal with ns-3 specific implementation of features or standard modules, extension of existing modules or improvement of features: out of 90 titles, 33 explicitly mention implementations, 11 improvements or extensions, 2 integrations, for a total of 46 (see Table 5).
These papers are not the only ones that are about implementations, improvements, extensions or integrations. In general, the focus of the papers are very specific to ns-3 internals and technical aspects, and rarely present other research related aspects, with some significant exception, so they are more bound to the ns-3 development community than to a general audience, with respect to the rest of the papers collected for our study. Out of the list of papers of the conference we isolated [3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24], that showed to extend significantly the domains coverable by means of ns-3 or to show noticeable research-oriented case studies.

5.2. Selection Criteria

We defined a general inclusion/exclusion criteria set to be applied to the papers obtained from the queries. These criteria are summarized in Table 6.

5.3. Data Extraction Form

The data extraction form is used to categorize and analyze the papers in an organic and objective way. It consists of two different parts:
  • the first part collects all general information about the paper, as from Table 7;
  • the second part collects the information that is more related with the research questions, to address the research questions (see Table 8):
    -
    for RQ1 the questions characterize the field in which ns-3 is used in terms of networks technologies, area of application and kind of problem;
    -
    for RQ2 the answers are binary (yes/no), and describe if the paper presents a new module or extension for ns-3 and if it uses an already existing module.

6. Results

6.1. Included and Excluded Studies

After the first search in Scopus and an integration by other search campaigns in the databases from Table 2, as described in Section 5.2 we selected 128 papers [1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128]. Then we applied to those papers the inclusion/exclusion criteria as described in Section 5.2, and after this phase we excluded 8 papers. More details are provided in Table 9 and Table 10, which reports the number of papers selected by the query and of the remaining papers after the selection process, divided by years.
After reading abstract and conclusion we excluded 2 papers of them based on exclusion criterion 2 [25,26], 3 based on exclusion criterion 3 [27,28,29], 2 based on exclusion criterion 4 [30,31] and 1 based on exclusion criterion 1 [32]. In the end, we included in the study 94% (120/128) of examined papers [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128]. Figure 3 shows that in recent years there has been a significant growth of the number of paper talking about ns-3 (the query was performed in October 2019, so the number of paper of year 2019 must be considered partial because of the typical delays in indexing and actual availability of published papers).
Table 11 and Figure 4 show that the selected papers mainly aim to developing applications or to perform conceptual study.

6.2. Data Synthesis

Once we selected the papers that are more pertinent and in line with the aim of our research, we evaluated the general quality level of the picked papers. A first analysis did show not a polarization of the number of papers on specific venues (as, for the sake of space, readers can directly verify by browsing the bibliography of this paper), suggesting that the adoption of ns-3 is not specifically proper of any sub-community nor restricted to specific groups of users or research groups at large, so we decided to omit a summary table and to avoid further analysis about venues.
Table 12 shows all details of the adopted Quality Assessment Scores form. Scores have been used to verify the validity of the selection and to support answering the research questions presented in the next Subsections.

6.3. RQ1

We wanted to address different side of this research question, basically to help the researchers when they must choose a network simulation tool and help to evaluate if ns-3 is a good choice. To achieve this goal, we examined the fields of application in which ns-3 was most used, both the application field, intended as topic, and specific network technology applied, and also as specific problem addressed.
Table 13 summarizes the information extracted from the papers that are related to this question.
Results reported in Table 13 suggest that the most of the application domains in networking are covered, with a (unsurprising) peak for ordinary networking, which is anyway not prominent, followed by VANET (Vehicle Ad Hoc Networks), MANET (Mobile Ad Hoc Networks) and common Mobile Networks on similar positions (see Figure 5). The lowest percentage of application, corresponding to WSN (Wireless Sensor Networks), yet a popular application field, may be due to the fact that support for energy management, which is of paramount importance in this subdomain, is still under development, and quite novel as a topic in the literature per se, and to the fact that the implementation of the simulation stack for typical real-world configuration is recent and still to be completed: anyway, the aggregation of all wireless networking related subdomains (WSN, Wireless Networks and Mobile Networks) that share the most of the low layers of the ISO/OSI stack, constitutes more than one third of all papers. The flexibility of the tool is witnessed by the composition of the group of papers labeled as “others”, as it includes very peculiar and specialistic domains such as Delay Tolerant Network, Named Data Network, Optical wireless, SDN, Visible Light Communication (VLC), while domains that are still emerging as autonomous from more general topics have been included in the relevant general topic (e.g., a paper about 5G mobile networks has been reported as belonging to Mobile Networks).
Variety of problems is also rich: anyway, here the analysis shows that the detail of the simulation stack fosters application to network assessment and protocol design, analysis and verification, and minor attention is paid by researchers and practitioners interested into physical layer problems, probably because of the closeness with the telecommunications domain, that also has proper specific tools and simulation support capturing physical issues in deeper details (see Figure 6).
Unfortunately, the analysis did not provide enough information to answer Q1.2 with a significant level of details, as the most of the papers were not especially focusing on specific application fields that were not implicitly stated in the application domain.

6.4. RQ2

The aim of the second research question is to address another kind of issue about simulation in general. The question investigates the need and the availability of additional modules to cover issues that are not natively dealt with by ns-3, to understand how much it is a good choice to approach problems that exceed the original extent of modeling support included in ns-3. The research question is articulated in two different yes/no questions, the first of which is oriented to quantify in how many papers a new extension or module is proposed for ns-3 to cover a specific need (see Figure 7), the second to quantify how many papers document cases in which additional modules or extensions have been actually used to reach the goals (see Figure 8). In Table 14 we show the results, in Table 15 the distribution of papers over years.
According to results, the practice of extending ns-3 is established, but not especially needed. There is a non-negligible number of papers that present new modules or extensions, but in a good majority of the cases, assuming that researchers that adopted this simulator consider it suitable for their purposes, the standard package seems to be sufficient to satisfy the most of the needs. In fact, considering the number of the papers that use extensions or contributed modules, almost one third of the examined literature actually exploits additions. Jointly with the percentage of positive answers to Q2.1, this seems to us to prove that the tool is easily and profitably extensible and needed, but the standard package is sufficient for the most of the needs.

7. Conclusions and Future Works

In this study, we documented the large success of ns-3 as a network simulator, its popularity and its flexibility. Data show that the scientific community considers it a useful tool in different fields, and dedicate third-party resources to extend it and develop add-ons for new adjacent application domains.
A further useful contribution may be provided by analogous analyses on literature on the main “competitors”, e.g., ns-2 and OMNET++, based on the same research questions and an overall meta-analysis to obtain a general view on the topic, to spot what is still not covered and would be useful for the scientific community and practitioners and develop focused built-in extensions or design a new, comprehensive simulation tool.

Author Contributions

All authors participated with the same effort and contribution to all phases in the preparation of this paper. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Acknowledgments

This work has been partially funded by the internal competitive funding program “VALERE: VAnviteLli pEr la RicErca” of Università degli Studi della Campania “Luigi Vanvitelli”.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Riley, G.F.; Henderson, T.R. The ns-3 Network Simulator. In Modeling and Tools for Network Simulation; Wehrle, K., Güneş, M., Gross, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2010; pp. 15–34. [Google Scholar] [CrossRef]
  2. Barisic, A.; Amaral, V.; Goulao, M. Domain-Specific Language Domain Analysis and Evaluation: A Systematic Literature Review; Technical Report; Faculdade de Ciencias e Tecnologia, Universidade Nova de Lisboa: Colombia, Portugal, 2015. [Google Scholar] [CrossRef]
  3. Piro, G.; Grieco, L.A.; Boggia, G.; Camarda, P. Nano-Sim: Simulating Electromagnetic-Based Nanonetworks in the Network Simulator 3. In Proceedings of the 6th International ICST Conference on Simulation Tools and Techniques, Cannes, France, 6–8 March 2013; ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering): Brussels, Belgium, 2013; pp. 203–210. [Google Scholar]
  4. Raman, A.; Chou, K.; Mastorakis, S. A simulation framework for Peer-to-Peer file sharing in named data networking. In Proceedings of the 2019 Workshop on ns-3, Florence, Italy, 18–21 June 2019; pp. 57–64. [Google Scholar]
  5. Bakshi, S.; Tahiliani, M.P. Recent Acknowledgement Support for ns-3 TCP. In Proceedings of the 2019 Workshop on ns-3, Florence, Italy, 18–21 June 2019; pp. 9–16. [Google Scholar]
  6. Rohrer, J.; Mauldin, A. Implementation of epidemic routing with IP convergence layer in ns-3. In Proceedings of the 10th Workshop on ns-3, Mangalore, India, 13–14 June 2018; ACM: New York, NY, USA, 2018; pp. 69–76. [Google Scholar]
  7. Mittal, V.; Jain, V.; Tahiliani, M.P. Proportional Rate Reduction for ns-3 TCP. In Proceedings of the 10th Workshop on ns-3, Mangalore, India, 13–14 June 2018; ACM: New York, NY, USA, 2018; pp. 9–15. [Google Scholar]
  8. Selinis, I.; Katsaros, K.; Vahid, S.; Tafazolli, R. Exploiting the capture effect on DSC and BSS color in dense IEEE 802.11ax deployments. In Proceedings of the Workshop on ns-3, Porto, Portugal, 13–14 June 2017; ACM: New York, NY, USA, 2017; Volume Part F128360, pp. 47–54. [Google Scholar]
  9. Fontes, H.; Campos, R.; Ricardo, M. A trace-based ns-3 simulation approach for perpetuating real-world experiments. In Proceedings of the Workshop on ns-3, Porto, Portugal, 13–14 June 2017; ACM: New York, NY, USA, 2017; Volume Part F128360, pp. 118–124. [Google Scholar]
  10. Franchi, M.; Pecorella, T.; Ridolfi, A.; Fantacci, R.; Allotta, B. Kinematic constraints and ns-3 mobility models: The AUV issue. In Proceedings of the Workshop on ns-3, Porto, Portugal, 13–14 June 2017; ACM: New York, NY, USA, 2017; Volume Part F128360, pp. 103–109. [Google Scholar]
  11. Ford, R.; Zhang, M.; Dutta, S.; Mezzavilla, M.; Rangan, S.; Zorzi, M. A framework for end-to-end evaluation of 5G mmWave cellular networks in ns-3. In Proceedings of the Workshop on ns-3, Seattle, WA, USA, 15–16 June 2016; Volume Part F132163, pp. 85–92. [Google Scholar]
  12. Dandoush, A.; Tuholukova, A.; Alouf, S.; Neglia, G.; Simoens, S.; Derouet, P.; Dersin, P. ns-3 Based framework for simulating Communication Based Train Control (CBTC) systems. In Proceedings of the Workshop on ns-3, Seattle, WA, USA, 15–16 June 2016; Volume Part F132163, pp. 116–123. [Google Scholar]
  13. Lehmann, A.; Kreuzer, M.; Deutschmann, J.; Berold, U.; Huber, J. Topology simulation for aeronautical communication protocols with ns-3 and DCE. In Proceedings of the Workshop on ns-3, Seattle, WA, USA, 15–16 June 2016; Volume Part F132163, pp. 132–138. [Google Scholar]
  14. Hood, C.; Riley, G. On predicting the performance characteristics of the NS-3 distributed simulator for scale-free internet models. In Proceedings of the 2015 Workshop on ns-3, Castelldefels, Spain, 13–14 May 2015; ACM: New York, NY, USA, 2015; pp. 54–59. [Google Scholar]
  15. Conceicao, S.; Ribeiro, F.; Campos, R.; Ricardo, M. Novel NS-3 model enabling simulation of electromagnetic wireless underground networks. In Proceedings of the 2015 Workshop on ns-3, Castelldefels, Spain, 13–14 May 2015; ACM: New York, NY, USA, 2015; pp. 9–16. [Google Scholar]
  16. Gupta, R.; Bachmann, B.; Ford, R.; Rangan, S.; Kundargi, N.; Ekbal, A.; Rathi, K.; Sanchez, M.; De La Oliva, A.; Morelli, A. Ns-3-based real-time emulation of LTE testbed using LabVIEW platform for software defined networking (SDN) in CROWD project. In Proceedings of the 2015 Workshop on ns-3, Castelldefels, Spain, 13–14 May 2015; ACM: New York, NY, USA, 2015; pp. 91–97. [Google Scholar]
  17. Cerqueira, T.; Albano, M. RoutesMobilityModel: Easy realistic mobility simulation using external information services. In Proceedings of the 2015 Workshop on ns-3, Castelldefels, Spain, 13–14 May 2015; ACM: New York, NY, USA, 2015; pp. 40–46. [Google Scholar]
  18. Newton, B.; Aikat, J.; Jeffay, K. Simulating large-scale airborne networks with NS-3. In Proceedings of the 2015 Workshop on ns-3, Castelldefels, Spain, 13–14 May 2015; ACM: New York, NY, USA, 2015; pp. 32–39. [Google Scholar]
  19. Gómez, D.; Rodríguez, E.; Agüero, R.; Muñoz, L. Reliable communications over wireless mesh networks with inter and intra-flow network coding. In Proceedings of the 2014 Workshop on ns-3, Atlanta, USA, 7 May 2014. [Google Scholar]
  20. Tariq, M.; Swenson, B.; Narasimhan, A.; Grijalva, S.; Riley, G.; Wolf, M. Cyber-physical co-simulation of smart grid applications using NS-3. In Proceedings of the 2014 Workshop on ns-3, Atlanta, GA, USA, 7 May 2014. [Google Scholar]
  21. Zugno, T.; Polese, M.; Lecci, M.; Zorzi, M. Simulation of Next-generation Cellular Networks with ns-3: Open Challenges and New Directions. In Proceedings of the 2019 Workshop on Next-Generation Wireless with ns-3, Florence, Italy, 19 June 2019; ACM: New York, NY, USA, 2019; pp. 38–41. [Google Scholar]
  22. Gawłowicz, P.; Baldo, N.; Miozzo, M. An extension of the ns-3 LTE module to simulate fractional frequency reuse algorithms. In Proceedings of the 2015 Workshop on ns-3, Castelldefels, Spain, 13–14 May 2015; ACM: New York, NY, USA, 2015; pp. 98–105. [Google Scholar]
  23. Tian, L.; Šljivo, A.; Santi, S.; De Poorter, E.; Hoebeke, J.; Famaey, J. Extension of the IEEE 802.11 ah ns-3 Simulation Module. In Proceedings of the 10th Workshop on ns-3, Mangalore, India, 13–14 June 2018; ACM: New York, NY, USA, 2018; pp. 53–60. [Google Scholar]
  24. Bojovic, B.; Abrignani, M.D.; Miozzo, M.; Giupponi, L.; Baldo, N. Towards LTE-Advanced and LTE-A Pro Network Simulations: Implementing Carrier Aggregation in LTE Module of ns-3. In Proceedings of the Workshop on ns-3, Porto, Portugal, 13–14 June 2017; ACM: New York, NY, USA, 2017; pp. 63–70. [Google Scholar]
  25. Rahman, M.U.; Rehman, A.U.; Liu, H.; Chen, H.F. Inhibitory mechanism of 5-bromo-3-indoleacetic acid for non-structural-3 helicase hepatitis C virus with dynamics correlation network analysis. Comput. Biol. Chem. 2018, 77, 167–177. [Google Scholar] [CrossRef] [PubMed]
  26. Kwo, P.Y.; Lawitz, E.J.; McCone, J.; Schiff, E.R.; Vierling, J.M.; Pound, D.; Davis, M.N.; Galati, J.S.; Gordon, S.C.; Ravendhran, N.; et al. Efficacy of boceprevir, an NS3 protease inhibitor, in combination with peginterferon alfa-2b and ribavirin in treatment-naive patients with genotype 1 hepatitis C infection (SPRINT-1): An open-label, randomised, multicentre phase 2 trial. Lancet 2010, 376, 705–716. [Google Scholar] [CrossRef] [Green Version]
  27. Xu, L.; Wang, X.; Shen, J. Strategy and simulation of trust cluster based key management protocol for ad hoc networks. In Proceedings of the 2009 4th International Conference on Computer Science & Education, Nanning, China, 25–28 July 2009; pp. 269–274. [Google Scholar]
  28. Yu, F.; Jain, R. A Survey of Wireless Sensor Network Simulation Tools; Washington University: St. Louis, MO, USA, 2011; pp. 1–10. Available online: https://www.cse.wustl.edu/~jain/cse567-11/ftp/sensor/ (accessed on 3 September 2019).
  29. Pereira, R.M.; Ruiz, L.B.; Ghizoni, M.L.A. MannaSim: A NS-2 extension to simulate wireless sensor network. In Proceedings of the ICN 2015, Barcelona, Spain, 19–24 April 2015; p. 107. [Google Scholar]
  30. Wang, Z.; Feng, X.; Qin, H.; Guo, H.; Han, G. An AUV-Aided routing protocol based on dynamic gateway nodes for underwater wireless sensor networks. J. Internet Technol. 2017, 18, 333–343. [Google Scholar] [CrossRef]
  31. Melo, R.; Polisciuc, R.; Nogueira, M.; Santos, A. Access Decision System for Seamless Connectivity in Heterogeneous Wireless Networks. In Proceedings of the 2015 XXXIII Brazilian Symposium on Computer Networks and Distributed Systems, Vitoria, Brazil, 18–22 May 2015; pp. 60–69. [Google Scholar] [CrossRef]
  32. Maygua-Marcillo, L.; Urquiza-Aguiar, L.; Paredes-Paredes, M. Creating a New Model in NS3 Network Simulator. Preprints 2019, 1–5. [Google Scholar] [CrossRef]
  33. Wang, Y.; Jiang, Y.N.; Ma, Q.; Tian, C.; Bai, B.; Zhang, G. RDMA load balancing via data partition. In Proceedings of the 2019 28th International Conference on Computer Communication and Networks (ICCCN), Valencia, Spain, 29 July–1 August 2019; pp. 1–8. [Google Scholar] [CrossRef]
  34. Ayaida, M.; Messai, N.; Najeh, S.; Boris Ndjore, K. A Macroscopic Traffic Model-based Approach for Sybil Attack Detection in VANETs. Ad Hoc Netw. 2019, 90, 101845. [Google Scholar] [CrossRef] [Green Version]
  35. Abbas, M.; Song, W.C. THERA: Two-level hierarchical hybrid road-aware routing for vehicular networks. KSII Trans. Internet Inf. Syst. 2019, 13, 3369–3385. [Google Scholar] [CrossRef] [Green Version]
  36. Eder-Neuhauser, P.; Zseby, T.; Fabini, J. Malware propagation in smart grid networks: Metrics, simulation and comparison of three malware types. J. Comput. Virol. Hacking Tech. 2019, 15, 109–125. [Google Scholar] [CrossRef] [Green Version]
  37. Fogwell, T.; Ochola, E. Comparison analysis of AODV and DSR under attack by black hole nodes in a NS3 simulation. In Proceedings of the International Conference on Cyber Warfare and Security, Stellenbosch, South Africa, 28 February–1 March 2019; pp. 574–587. [Google Scholar]
  38. Bhattacharya, H.; Chattopadhyay, S.; Chattopadhyay, M. NS3 Based HDFS data placement algorithm evaluation framework. In Proceedings of the 2017 International Conference on Computer, Electrical & Communication Engineering (ICCECE), Kolkata, India, 22–23 December 2017; pp. 1–8. [Google Scholar] [CrossRef]
  39. Anadu, D.; Mushagalusa, C.; Alsbou, N.; Abuabed, A. Internet of Things: Vehicle collision detection and avoidance in a VANET environment. In Proceedings of the 2018 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), Houston, TX, USA, 14–17 May 2018; pp. 1–6. [Google Scholar] [CrossRef]
  40. Tiennoy, S.; Saivichit, C. Using a Distributed Roadside Unit for the Data Dissemination Protocol in VANET with the Named Data Architecture. IEEE Access 2018, 6, 32612–32623. [Google Scholar] [CrossRef]
  41. Wang, T.; Hussain, A. PySNS3: A real-time communication interface and protocol for vehicular Ad-Hoc networks. In Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Springer: Cham, Switzerland, 2018; pp. 163–171. [Google Scholar] [CrossRef]
  42. Xie, X.; Wang, J.; Guo, X.; Wu, X. Performance evaluation of ad-hoc routing protocols in hybrid MANET-satellite network. In Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Springer: Cham, Switzerland, 2018; Volume 251, pp. 500–509. [Google Scholar] [CrossRef]
  43. Tada, N.; Patalia, T.; Rupani, P. A new approach to mitigate jamming attack in wireless adhoc network using ARC technique. In Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Springer: Cham, Switzerland, 2018; pp. 192–204. [Google Scholar] [CrossRef]
  44. Liu, D.; Ni, J.; Li, H.; Shen, X. Achieving Adaptive Linkability for Cellular V2X Group Communications in 5G. In Proceedings of the 2018 IEEE Global Communications Conference (GLOBECOM), Abu Dhabi, UAE, 9–13 December 2018; pp. 1–7. [Google Scholar] [CrossRef]
  45. Pospelova, I.; Chebotayev, P.; Klimenko, A.; Myakochin, Y.; Polyakov, I.; Shelupanov, A.; Zykov, D. Research of G3-PLC net self-organization processes in the NS-3 modeling framework. AIP Conf. Proc. 2017, 1899, 060017. [Google Scholar] [CrossRef]
  46. Drissi, M.; Oumsis, M.; Aboutajdine, D. A Multi-Criteria Decision Framework for network selection over LTE and WLAN. Eng. Appl. Artif. Intell. 2017, 66, 113–127. [Google Scholar] [CrossRef]
  47. Abdelsalam, A.; Luglio, M.; Roseti, C.; Zampognaro, F. TCP Connection management through combined use of terrestrial and satellite IP-based links. In Proceedings of the 2017 40th International Conference on Telecommunications and Signal Processing (TSP), Barcelona, Spain, 5–7 July 2017; pp. 37–42. [Google Scholar] [CrossRef]
  48. Mao, Q.; Yue, P.; Xu, M.; Ji, Y.; Cui, Z. OCTMAC: A VLC based MAC protocol combining optical CDMA with TDMA for VANETs. In Proceedings of the International Conference on Computer, Information and Telecommunication Systems (CITS), Dalian, China, 21–23 July 2017; pp. 234–238. [Google Scholar] [CrossRef]
  49. Karimzadeh, M.; Valtulina, L.; Van Den Berg, H.; Pras, A.; Ortiz, P.; Sadre, R. MultiPath TCP to support user’s mobility in future LTE network. In Proceedings of the 2017 10th IFIP Wireless and Mobile Networking Conference (WMNC), Valencia, Spain, 25–27 September 2017; pp. 1–8. [Google Scholar] [CrossRef] [Green Version]
  50. Agarwal, R.; Shukla, P.; Goyal, S. A Practical Approach of Network Simulation. In Detecting and Mitigating Robotic Cyber Security Risks; Kumar, R., Pattnaik, P., Pandey, P., Eds.; IGI Global: Hershey, Pennsylvania, 2017; pp. 12–27. [Google Scholar] [CrossRef]
  51. Liu, Y.; Piao, X.; Hou, C.; Lei, K. A CUBIC-Based explicit congestion control mechanism in named data networking. In Proceedings of the 2016 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), Chengdu, China, 13–15 October 2016; pp. 360–363. [Google Scholar] [CrossRef]
  52. Chiu, J.C.; Liu, A.T.; Liao, C.C. Design the DNS-Like Smart Switch for Heterogeneous Network Base on SDN Architecture. In Proceedings of the 2016 International Computer Symposium (ICS), Chiayi, Taiwan, 15–17 December 2016; pp. 187–191. [Google Scholar] [CrossRef]
  53. Wu, W.; Li, Z.; Zhang, Y.; Guo, J.; Zhao, J. Design and implementation of distributed broadcast algorithm based on vehicle density for vanet safety-related messages. Commun. Comput. Inf. Sci. 2017, 728, 271–285. [Google Scholar] [CrossRef]
  54. Sahu, A.; Goulart, A.; Butler-Purry, K. Modeling AMI network for real-time simulation in NS-3. In Proceedings of the 2016 Principles, Systems and Applications of IP Telecommunications (IPTComm), Chicago, IL, USA, 17–20 October 2016; pp. 1–8. [Google Scholar]
  55. Yang, H.; Li, Z. A genetic-algorithm-based clustering protocol in MANET. In Proceedings of the 7th International Conference on Computing Communication and Networking Technologies, Dallas, TX, USA, 6–8 July 2016; pp. 1–5. [Google Scholar] [CrossRef]
  56. Deng, X.; He, L.; Li, X.; Liu, Q.; Cai, L.; Chen, Z. A reliable QoS-aware routing scheme for neighbor area network in smart grid. Peer-to-Peer Netw. Appl. 2016, 9, 616–627. [Google Scholar] [CrossRef]
  57. Wang, Z.; Guo, H.; Jiang, L.; Feng, X. AUV-aided communication method for underwater mobile sensor network. In Proceedings of the OCEANS 2016-Shanghai, Shanghai, China, 10–13 April 2016; pp. 1–7. [Google Scholar] [CrossRef]
  58. Wang, S.; Sun, D.; Zhang, Y. Performance analysis for ALOHA protocol of underwater acoustic networks with a serial route. Harbin Gongcheng Daxue Xuebao/J. Harbin Eng. Univ. 2016, 37, 360–367. [Google Scholar] [CrossRef]
  59. Choudhury, A.; Maszczyk, T.; Math, C.; Li, H.; Dauwels, J. An integrated simulation environment for testing V2X protocols and applications. Procedia Comput. Sci. 2016, 80, 2042–2052. [Google Scholar] [CrossRef] [Green Version]
  60. Kang, H.; Jiang, X.; Xiong, W. Research on wireless networking mode for the distributed satellite communication system. J. Comput. Inf. Syst. 2015, 11, 4211–4220. [Google Scholar] [CrossRef]
  61. Aburumman, A.; Seo, W.; Islam, R.; Khan, M.; Choo, K.K. A secure cross-domain SIP solution for mobile ad hoc network using dynamic clustering. In Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Springer: Cham, Switzerland, 2015; Volume 164, pp. 649–664. [Google Scholar] [CrossRef]
  62. Guo, Y.; Ten, C.W.; Hu, S.; Weaver, W. Modeling distributed denial of service attack in advanced metering infrastructure. In Proceedings of the 2015 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT), Washington, DC, USA, 17–20 February 2015; pp. 1–5. [Google Scholar] [CrossRef]
  63. Mallissery, S.; Manohara Pai, M.; Ajam, N.; Pai, R.; Mouzna, J. Transport and traffic rule violation monitoring service in ITS: A secured VANET cloud application. In Proceedings of the 2015 12th Annual IEEE Consumer Communications and Networking Conference (CCNC), Las Vegas, NV, USA, 9–12 January 2015; pp. 213–218. [Google Scholar] [CrossRef]
  64. Tang, C.; Song, L.; Balasubramani, J.; Wu, S.; Biaz, S.; Yang, Q.; Wang, H. Comparative investigation on CSMA/CA-based opportunistic random access for internet of things. IEEE Internet Things J. 2014, 1, 171–179. [Google Scholar] [CrossRef]
  65. Molloy, T.; Yuan, Z.; Muntean, G.M. Real time emulation of an LTE network using NS-3. In Proceedings of the 25th IET Irish Signals & Systems Conference 2014 and 2014 China-Ireland International Conference on Information and Communications Technologies (ISSC 2014/CIICT 2014), Limerick, Ireland, 26–27 June 2014; Volume 2014, pp. 251–257. [Google Scholar] [CrossRef]
  66. Satria, T.; Karimzadeh, M.; Karagiannis, G. Performance evaluation of ICN/CCN based service migration approach in virtualized LTE systems. In Proceedings of the 2014 IEEE 3rd International Conference on Cloud Networking (CloudNet), Luxembourg, 8–10 October 2014; pp. 461–467. [Google Scholar] [CrossRef]
  67. Shukla, S.; Chan, S.; Tam, A.W.; Gupta, A.; Xu, Y.; Chao, H. TCP PLATO: Packet labelling to alleviate time-out. IEEE J. Sel. Areas Commun. 2014, 32, 65–76. [Google Scholar] [CrossRef]
  68. Marques, H.; Leguay, J.; Khalifé, H.; Conan, V.; Lavaux, D. Transparent IP proxy for tactical ad hoc networks. In Proceedings of the MILCOM 2013-2013 IEEE Military Communications Conference, San Diego, CA, USA, 17–20 November 2013; pp. 842–847. [Google Scholar] [CrossRef] [Green Version]
  69. Wang, Y.; Rozhnova, N.; Narayanan, A.; Oran, D.; Rhee, I. An improved hop-by-hop interest shaper for congestion control in named data networking. ACM SIGCOMM Comput. Commun. Rev. 2013, 43, 567–572. [Google Scholar] [CrossRef] [Green Version]
  70. Tsokalo, I.; Mudriievskyi, S.; Lehnert, R. Development and calibration of a PLC simulation model for UPA-compliant networks. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Berlin/Heidelberg, Germany, 2013; Volume 8115 LNCS, pp. 197–208. [Google Scholar] [CrossRef] [Green Version]
  71. Meneguette, R.; Bittencourt, L.; Madeira, E. User-centric mobility management architecture for vehicular networks. In Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Springer: Berlin/Heidelberg, Germany, 2013; Volume 58, pp. 42–56. [Google Scholar] [CrossRef]
  72. Carvin, D.; Kremer, G.; Owezarski, P.; Berthou, P. Assessment and event based analysis of dynamic wireless networks. In Proceedings of the 9th International Conference on Network and Service Management (CNSM 2013), Zürich, Switzerland, 14–18 October 2013; pp. 175–179. [Google Scholar] [CrossRef] [Green Version]
  73. Biddlestone, S.; Redmill, K.; Miucic, R.; Ozguner, U. An integrated 802.11p WAVE DSRC and vehicle traffic simulator with experimentally validated urban (LOS and NLOS) propagation models. IEEE Trans. Intell. Transp. Syst. 2012, 13, 1792–1802. [Google Scholar] [CrossRef]
  74. Vetrivelan, P.; Narayanasamy, P.; John Charlas, J. A multi-constraint real-time vehicular (MCRV) mobility framework for 4G heterogeneous Vehicular Ad-Hoc Networks. In Proceedings of the International MultiConference of Engineers and Computer Scientists, Hong Kong, China, 14–16 March 2012; Volume 2195, pp. 423–428. [Google Scholar]
  75. Vetrivelan, P.; Narayanasamy, P. SMIRT with call admission control (CAC) based vertical handover decision for seamless mobility in multi-access 4G heterogeneous wireless overlav networks. In Proceedings of the International Multiconference of Engineers & Computer Scientists, Hong Kong, China, 14–16 March 2012; Volume 2195, pp. 408–413. [Google Scholar]
  76. Qwasmi, N.; Ahmed, F.; Liscano, R. Simulation of DDOS attacks on P2P metworks. In Proceedings of the 2011 IEEE International Conference on High Performance Computing and Communications, Banff, AB, Canada, 2–4 September 2011; pp. 610–614. [Google Scholar] [CrossRef]
  77. Katkar, P.S.; Ghorpade, D.V.R. Comparative study of network simulator: NS2 and NS3. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 2016, 6, 608–612. [Google Scholar]
  78. Siraj, S.; Gupta, A.; Badgujar, R. Network simulation tools survey. Int. J. Adv. Res. Comput. Commun. Eng. 2012, 1, 199–206. [Google Scholar]
  79. Pan, J.; Jain, R. A Survey of Network Simulation Tools: Current Status and Future Developments. 2008, Volume 2, p. 45. Available online: https://www.cse.wustl.edu/~jain/cse567-08/ftp/simtools/index.html (accessed on 3 September 2019).
  80. Korkalainen, M.; Sallinen, M.; Kärkkäinen, N.; Tukeva, P. Survey of wireless sensor networks simulation tools for demanding applications. In Proceedings of the 2009 Fifth International Conference on Networking and Services, Valencia, Spain, 20–25 April 2009; pp. 102–106. [Google Scholar]
  81. Venkataramanan, V.; Lakshmi, S. A Case Study of Various Wireless Network Simulation Tools. Int. J. Commun. Netw. Inf. Secur. 2018, 10, 389–396. [Google Scholar]
  82. Mehta, S.; Ullah, N.; Kabir, M.H.; Sultana, M.N.; Kwak, K.S. A Case Study of Networks Simulation Tools for Wireless Networks. In Proceedings of the 2009 Third Asia International Conference on Modelling Simulation, Bali, Indonesia, 25–29 May 2009; pp. 661–666. [Google Scholar] [CrossRef]
  83. Dorathy, I.; Chandrasekaran, M. Simulation tools for mobile ad hoc networks: A survey. J. Appl. Res. Technol. 2018, 16, 437–445. [Google Scholar] [CrossRef] [Green Version]
  84. Mallapur, S.V.; Patil, S.R. Survey on simulation tools for mobile ad-hoc networks. In. J. Comput. Netw. Wirel. Commun. 2012, 2, 241–248. [Google Scholar]
  85. Toor, A.S.; Jain, A. A survey on wireless network simulators. Bull. Electrical Eng. Inform. 2017, 6, 62–69. [Google Scholar]
  86. Miozzo, M.; Bartzoudis, N.; Requena, M.; Font-Bach, O.; Harbanau, P.; López-Bueno, D.; Payaró, M.; Mangues, J. SDR and NFV extensions in the ns-3 LTE module for 5G rapid prototyping. In Proceedings of the 2018 IEEE Wireless Communications and Networking Conference (WCNC), Barcelona, Spain, 15–18 April 2018; pp. 1–6. [Google Scholar]
  87. Giupponi, L.; Henderson, T.; Bojovic, B.; Miozzo, M. Simulating LTE and Wi-Fi coexistence in unlicensed spectrum with NS-3. arXiv 2016, arXiv:1604.06826. [Google Scholar]
  88. Zhong, P.; Li, Y.; Huang, W.; Kui, X.; Zhang, Y.; Chen, Y. An Extension to ns-3 for Simulating Mobile Charging with Wireless Energy Transfer. In Communications in Computer and Information Science; Zou, B., Han, Q., Sun, G., Jing, W., Peng, X., Lu, Z., Eds.; Data Science. ICPCSEE 2017; Springer: Singapore, 2017; Volume 728, pp. 256–270. [Google Scholar]
  89. Alwahab, D.A.; Laki, S. A simulation-based survey of active queue management algorithms. In Proceedings of the 6th International Conference on Communications and Broadband Networking, Singapore, 24–26 February 2018; pp. 71–77. [Google Scholar]
  90. Jian, Y.; Krishnaswamy, B.; Austin, C.M.; Bicen, A.O.; Einolghozati, A.; Perdomo, J.E.; Patel, S.C.; Fekri, F.; Akyildiz, I.F.; Forest, C.R.; et al. nanoNS3: A network simulator for bacterial nanonetworks based on molecular communication. Nano Commun. Netw. 2017, 12, 1–11. [Google Scholar] [CrossRef] [Green Version]
  91. Bisio, I.; Delucchi, S.; Lavagetto, F.; Marchese, M.; Portomauro, G.; Zappatore, S. An Ns-3 based simulative and emulative platform. In Modeling and Simulation of Computer Networks and Systems; Elsevier: Amsterdam, The Netherlands, 2015; pp. 555–575. [Google Scholar]
  92. Kim, E.; Riley, G.F. Automatic State Saving and Rollback in ns-3. In Proceedings of the 2017 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation, Singapore, 24–26 May 2017; pp. 263–266. [Google Scholar]
  93. Maskooki, A.; Sabatino, G.; Mitton, N. Chapter 21—Analysis and performance evaluation of the next generation wireless networks. In Modeling and Simulation of Computer Networks and Systems; Obaidat, M.S., Nicopolitidis, P., Zarai, F., Eds.; Morgan Kaufmann: Boston, MA, USA, 2015; pp. 601–627. [Google Scholar] [CrossRef]
  94. Fernandes, D.A.; Neto, M.; Soares, L.F.; Freire, M.M.; Inácio, P.R. Chapter 10—On the self-similarity of traffic generated by network traffic simulators. In Modeling and Simulation of Computer Networks and Systems; Obaidat, M.S., Nicopolitidis, P., Zarai, F., Eds.; Morgan Kaufmann: Boston, MA, USA, 2015; pp. 285–311. [Google Scholar] [CrossRef]
  95. Amoretti, M.; Picone, M.; Zanichelli, F.; Ferrari, G. Chapter 16—Simulating wireless and mobile systems: The Integration of DEUS and Ns-3. In Modeling and Simulation of Computer Networks and Systems; Obaidat, M.S., Nicopolitidis, P., Zarai, F., Eds.; Morgan Kaufmann: Boston, MA, USA, 2015; pp. 465–484. [Google Scholar] [CrossRef]
  96. Tsioliaridou, A.; Liaskos, C.; Dedu, E.; Ioannidis, S. Packet routing in 3D nanonetworks: A lightweight, linear-path scheme. Nano Commun. Netw. 2017, 12, 63–71. [Google Scholar] [CrossRef] [Green Version]
  97. Hossain, Z.; Xia, Q.; Jornet, J.M. TeraSim: An ns-3 extension to simulate Terahertz-band communication networks. Nano Commun. Netw. 2018, 17, 36–44. [Google Scholar] [CrossRef]
  98. Barolli, A.; Oda, T.; Ikeda, M.; Barolli, L.; Xhafa, F.; Loia, V. Node placement for wireless mesh networks: Analysis of WMN-GA system simulation results for different parameters and distributions. J. Comput. Syst. Sci. 2015, 81, 1496–1507. [Google Scholar] [CrossRef] [Green Version]
  99. Habbal, A.; Goudar, S.I.; Hassan, S. A Context-aware Radio Access Technology selection mechanism in 5G mobile network for smart city applications. J. Netw. Comput. Appl. 2019, 135, 97–107. [Google Scholar] [CrossRef]
  100. Aldalbahi, A.; Rahaim, M.; Khreishah, A.; Ayyash, M.; Ackerman, R.; Basuino, J.; Berreta, W.; Little, T.D. Extending ns3 to simulate visible light communication at network-level. In Proceedings of the 2016 23rd International Conference on Telecommunications (ICT), Thessaloniki, Greece, 16–18 May 2016; pp. 1–6. [Google Scholar]
  101. Rajankumar, P.; Nimisha, P.; Kamboj, P. A comparative study and simulation of AODV MANET routing protocol in NS2 & NS3. In Proceedings of the 2014 International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India, 5–7 March 2014; pp. 889–894. [Google Scholar]
  102. Aldalbahi, A.; Rahaim, M.; Khreishah, A.; Ayyash, M.; Little, T.D. Visible light communication module: An open source extension to the ns3 network simulator with real system validation. IEEE Access 2017, 5, 22144–22158. [Google Scholar] [CrossRef]
  103. Liu, W.; Wang, X.; Zhang, W.; Yang, L.; Peng, C. Coordinative simulation with SUMO and NS3 for vehicular ad hoc networks. In Proceedings of the 2016 22nd Asia-Pacific Conference on Communications (APCC), Yogyakarta, Indonesia, 25–27 August 2016; pp. 337–341. [Google Scholar]
  104. Jha, R.K.; Kharga, P. A comparative performance analysis of routing protocols in MANET using NS3 simulator. Int. J. Comput. Netw. Inf. Security 2015, 7, 62–68. [Google Scholar]
  105. Bhatia, D.; Sharma, D.P. A comparative analysis of proactive, reactive and hybrid routing protocols over open source network simulator in mobile ad hoc network. Int. J. Appl. Eng. Res. 2016, 11, 3885–3896. [Google Scholar]
  106. Naik, L.; Khan, R.; Mishra, R. Analysis of node velocity effects in MANET routing protocols using network simulator (NS3). Int. J. Comput. Appl. 2016, 144, 1–5. [Google Scholar] [CrossRef]
  107. Aman, A.H.M.; Hashim, A.H.A.; Ramli, H.A.M. Simulation Analysis for Multicast Context Delivery Network Mobility Management. Indones. J. Electr. Eng. Inform. (IJEEI) 2017, 5, 390–394. [Google Scholar]
  108. Saluja, A.; Dargad, S.; Mistry, K. A Detailed Analogy of Network Simulators—NS1, NS2, NS3 and NS4. Int. J. Future Revolut. Comput. Sci. Commun. Eng. 2017, 3, 291–295. [Google Scholar]
  109. Spaho, E.; Bylykbashi, K.; Barolli, L.; Takizawa, M. Routing in a DTN: Performance evaluation for random waypoint and steady state random waypoint using NS3 simulator. In Advances on P2P, Parallel, Grid, Cloud and Internet Computing, 3PGCIC 2017; Xhafa, F., Caballé, S., Barolli, L., Eds.; Lecture Notes on Data Engineering and Communications Technologies; Springer: Cham, Switzerland, 2017; Volume 13, pp. 133–141. [Google Scholar]
  110. Patel, R.; Patel, N.; Patel, S. An Approach to Analyze Behavior of Network Events in NS2 and NS3 Using AWK and Xgraph. In Information and Communication Technology for Competitive Strategies; Springer: Cham, Switzerland, 2019; pp. 137–147. [Google Scholar]
  111. Amiri-Nezhad, M.; Guerrero-Zapata, M.; Bellalta, B.; Cerdà-Alabern, L. Simulation of multi-radio multi-channel 802.11-based mesh networks in ns-3. EURASIP J. Wirel. Commun. Netw. 2014, 2014, 118. [Google Scholar] [CrossRef] [Green Version]
  112. Nurchis, M.; Bellalta, B. Performance Evaluation of LAA-LTE and WiFi Coexistence in Unlicensed 5 GHz Band Under Asymmetric Network Deployments Using NS3. In Proceedings of the International Workshop on Multiple Access Communications, Aalborg, Denmark, 21–22 November 2016; Springer: Cham, Switzerland, 2016; pp. 86–97. [Google Scholar]
  113. Paliwal, G.; Taterh, S. Impact of Dense Network in MANET Routing Protocols AODV and DSDV Comparative Analysis Through NS3. In Soft Computing: Theories and Applications; Springer: Cham, Switzerland, 2018; pp. 327–335. [Google Scholar]
  114. Khattak, R.; Chaltseva, A.; Riliskis, L.; Bodin, U.; Osipov, E. Comparison of wireless network simulators with multihop wireless network testbed in corridor environment. In Proceedings of the Multiple Access Communications, Lecture Notes in Computer Science, MACOM 2016, Thessaloniki, Greece, 25–27 May 2016; Madsen, T., Nielsen, J., Pratas, N., Eds.; Springer: Cham, Switzerland, 2011; Volume 10121, pp. 80–91. [Google Scholar]
  115. Nabou, A.; Laanaoui, M.D.; Ouzzif, M. The Effect of Transmit Power on MANET Routing Protocols Using AODV, DSDV, DSR and OLSR in NS3. In Advanced Intelligent Systems for Sustainable Development (AI2SD’2018), Proceedings of the Advances in Intelligent Systems and Computing AI2SD 2018, Tangier, Morocco, 12–14 July 2018; Ezziyyani, M., Ed.; Springer: Cham, Switzerland, 2018; Volume 915, pp. 274–286. [Google Scholar]
  116. Henderson, T.R.; Lacage, M.; Riley, G.F.; Dowell, C.; Kopena, J. Network simulations with the ns-3 simulator. SIGCOMM Demonstr. 2008, 14, 527. [Google Scholar]
  117. Afanasyev, A.; Moiseenko, I.; Zhang, L. ndnSIM: NDN Simulator for NS-3; University of California: Los Angeles, CA, USA, 2012; Volume 4. [Google Scholar]
  118. Mastorakis, S.; Afanasyev, A.; Moiseenko, I.; Zhang, L. ndnSIM 2: An Updated NDN Simulator for NS-3; NDN, Technical Report NDN-0028. 2016, pp. 1–8. Available online: http://named-data.net/publications/techreports/ndn-0028-2-ndnsim-v2/ (accessed on 3 September 2019).
  119. Pei, G.; Henderson, T.R. Validation of OFDM error rate model in ns-3. Boeing Res. Technol. 2010, 1–15. Available online: https://pdfs.semanticscholar.org/3f0a/b039b235bd0fa1e833876ba78e7ea99d9a04.pdf (accessed on 3 September 2019).
  120. Carneiro, G.; Fortuna, P.; Ricardo, M. FlowMonitor: A network monitoring framework for the network simulator 3 (NS-3). In Proceedings of the Fourth International ICST Conference on Performance Evaluation Methodologies and Tools, Pisa, Italy, 20–22 October 2009; ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering): Brussels, Belgium, 2009; pp. 1–10. [Google Scholar]
  121. Arbabi, H.; Weigle, M.C. Highway mobility and vehicular ad-hoc networks in ns-3. In Proceedings of the Winter Simulation Conference, Baltimore, MA, USA, 5–8 December 2010; pp. 2991–3003. [Google Scholar]
  122. Stoffers, M.; Riley, G. Comparing the ns-3 propagation models. In Proceedings of the 2012 IEEE 20th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, San Francisco, CA, USA, 7–9 August 2012; pp. 61–67. [Google Scholar]
  123. Narra, H.; Cheng, Y.; Cetinkaya, E.K.; Rohrer, J.P.; Sterbenz, J.P. Destination-sequenced distance vector (DSDV) routing protocol implementation in ns-3. In Proceedings of the 4th International ICST Conference on Simulation Tools and Techniques, Barcelona, Spain, 21–25 March 2011; ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering): Brussels, Belgium, 2011; pp. 439–446. [Google Scholar]
  124. Van den Abeele, F.; Haxhibeqiri, J.; Moerman, I.; Hoebeke, J. Scalability analysis of large-scale LoRaWAN networks in ns-3. IEEE Internet Things J. 2017, 4, 2186–2198. [Google Scholar] [CrossRef] [Green Version]
  125. Wu, X.; Brown, K.N.; Sreenan, C.J.; Alvarez, P.; Ruffini, M.; Marchetti, N.; Payne, D.; Doyle, L. An XG-PON module for the NS-3 network simulator. In Proceedings of the 6th International ICST Conference on Simulation Tools and Techniques, Cannes, France, 5–7 March 2013; ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering): Brussels, Belgium, 2013; pp. 195–202. [Google Scholar]
  126. Lu, Z.; Qu, G.; Liu, Z. A Survey on Recent Advances in Vehicular Network Security, Trust, and Privacy. IEEE Trans. Intell. Transp. Syst. 2019, 20, 760–776. [Google Scholar] [CrossRef]
  127. Dong, Q.; Chen, Y.; Li, X.; Zeng, K. A Survey on Simulation Tools and Testbeds for Cognitive Radio Networks Study. arXiv 2018, arXiv:1808.09858. [Google Scholar]
  128. Alenazi, M.J.F.; Cheng, Y.; Zhang, D.; Sterbenz, J.P.G. Epidemic routing protocol implementation in ns-3. In Proceedings of the 2015 Workshop on ns-3, Castelldefels, Spain, 13–14 May 2015; ACM: New York, NY, USA, 2015; pp. 54–59. [Google Scholar]
Electronics 09 00272 g001 550
Figure 1. SLR Process.
Figure 1. SLR Process.
Electronics 09 00272 g001
Electronics 09 00272 g002 550
Figure 2. SLR Protocol Used.
Figure 2. SLR Protocol Used.
Electronics 09 00272 g002
Electronics 09 00272 g003 550
Figure 3. Year distribution of selected papers.
Figure 3. Year distribution of selected papers.
Electronics 09 00272 g003
Electronics 09 00272 g004 550
Figure 4. Type of content.
Figure 4. Type of content.
Electronics 09 00272 g004
Electronics 09 00272 g005 550
Figure 5. RQ1–Q1.1.
Figure 5. RQ1–Q1.1.
Electronics 09 00272 g005
Electronics 09 00272 g006 550
Figure 6. RQ1–Q1.3.
Figure 6. RQ1–Q1.3.
Electronics 09 00272 g006
Electronics 09 00272 g007 550
Figure 7. RQ2–Q2.1.
Figure 7. RQ2–Q2.1.
Electronics 09 00272 g007
Electronics 09 00272 g008 550
Figure 8. RQ2–Q2.2.
Figure 8. RQ2–Q2.2.
Electronics 09 00272 g008
Table
Table 1. Comparison of network simulators.
Table 1. Comparison of network simulators.
Toolns-3ns-3OMNET++SWANSNetSimQualNet
InterfaceC++/PythonC++/OTclC++/NEDJavaC/Java/.NETParsec (C Based)
LicenseFreeFreeFree for academicFreeCommercialCommercial
ParallelismNoNoMPI/PVMNo-SMP/Beowulf
O.S.Linux, FreeBSD, Mac OS X, WindowsLinux, FreeBSD, Mac OS X, WindowsLinux, Mac OS X, WindowsLinux, Mac OS X, WindowsWindowsLinux, Mac OS X, Windows, Unix
Mobility SupportYesYesNoYesYesYes
GUI SupportLimitedYesYesYesYesYes
Table
Table 2. Searched databases.
Table 2. Searched databases.
Data SourceWeb Site
Scopushttp://scopus.com
The ACM Digital Libraryhttp://dl.acm.org/
IEEE Xplorehttp://ieeexplore.ieee.org/Xplore/
Springerhttps://link.springer.com/
Elsevier Science Directhttps://www.elsevier.com/solutions/sciencedirect/
Table
Table 3. Documents retrieved for research keywords and source [2009–2019].
Table 3. Documents retrieved for research keywords and source [2009–2019].
SourceNumber of Docs for ns3Number of Docs for ns-3
1Google Scholar28,70032,500
2Elsevier727,109
3Springer2173,518
4IEEE Explore D.L.4571816
5The ACM Digital Library394290,550
Table
Table 4. ACM Classification: Topic C.2—Computer-Communication Networks.
Table 4. ACM Classification: Topic C.2—Computer-Communication Networks.
C.2.0General
C.2.1Network Architecture and Design
C.2.2Network Protocols
C.2.3Network Operations
C.2.4Distributed Systems
C.2.5Local and Wide-Area Networks
C.2.6Internetworking
C.2.mMiscellaneous
Table
Table 5. Occurrence of keywords in titles of WNS3 papers.
Table 5. Occurrence of keywords in titles of WNS3 papers.
implementation31
implementing2
improved2
improvement1
improving2
enhancing1
extending2
extension3
integration1
integrating1
Table
Table 6. Inclusion/exclusion criteria.
Table 6. Inclusion/exclusion criteria.
TypeCriteria
1ExclusionInformal literature
2ExclusionDuplicated papers
3ExclusionPapers that did not comply research questions
4ExclusionPapers not written in English language
5InclusionPapers that report any kind of use of ns-3
Table
Table 7. Data Extraction Form—part 1.
Table 7. Data Extraction Form—part 1.
Id...
Author...
Title...
Year...
Venue...
Citations...
Table
Table 8. Data Extraction Form—part 2.
Table 8. Data Extraction Form—part 2.
RQ1Which are the impacted Application Domains?
Q1.1What kind of the network technologies are simulated? (e.g., WSN, wireless networks,
mobile networks, internetworking...)
Q1.2Which are the related application field? (e.g., medicine, astrology, ...)
Q1.3What kind of problem is addressed? (e.g., security, assessment, experimental methodology...)
RQ2How much is this approach extensible?
Q2.1Is it a new extension or module? yes/no
Q2.2Does it use an external module or extension? yes/no
Table
Table 9. Selected papers by year (2009–2014).
Table 9. Selected papers by year (2009–2014).
200920102011201220132014
Query2%0%2%6%5%5%
search(3)(0)(3)(8)(7)(7)
Selected2%0%2%7%6%6%
papers(2)(0)(2)(8)(7)(7)
Table
Table 10. Selected papers by year (2015–2019).
Table 10. Selected papers by year (2015–2019).
20152016201720182019TOTAL
Query16%19%23%13%9%100%
search(21)(23)(28)(17)(11)128
Selected16%19%22%12%8%100%
papers(19)(23)(27)(15)(10)120
Table
Table 11. Type of content.
Table 11. Type of content.
Application study68
Module extension5
Conceptual study36
Survey11
Table
Table 12. Quality Assessment Scores form.
Table 12. Quality Assessment Scores form.
IdQuality Assessment CriteriaScore
A1What is the relevance of the paper according to the
conference/journal where it was published?
A2What is the relevance of the citation according to its
related citations?
A3How clearly is the problem of the study described?
A4How clearly is the research context stated?
A5How explicitly are the contributions presented?
A6How explicitly are the insights and issues for future
work stated?
Average Evaluation
Table
Table 13. RQ1 Results.
Table 13. RQ1 Results.
RQ1Which are the Impacted Application Domains?
Q1.1What kind of network technologies are simulated?%n
MANET14%16
WSN8%10
Wireless Networks12%14
Mobile Networks15%18
Networking27%32
VANET17%18
Other7%9
Q1.2Which are the related application fields?
Medicine
Physics
Astronomy
Other
Q1.3What kind of problem is addressed?
Security11%11
Assessment33%34
Network Protocol26%27
Physical Layer6%6
Performance24%24
Table
Table 14. RQ2 Results.
Table 14. RQ2 Results.
RQ2How Much Is This Approach Extensible?
Q2.1Is it a new extension or module?%n
Yes17%20
No83%100
Q2.2Does it use an external module or extension?
Yes31%37
No89%83
Table
Table 15. Years distribution.
Table 15. Years distribution.
20092010201120122013201420152016201720182019
2028771923271510

Share and Cite

MDPI and ACS Style

Campanile, L.; Gribaudo, M.; Iacono, M.; Marulli, F.; Mastroianni, M. Computer Network Simulation with ns-3: A Systematic Literature Review. Electronics 2020, 9, 272. https://doi.org/10.3390/electronics9020272

AMA Style

Campanile L, Gribaudo M, Iacono M, Marulli F, Mastroianni M. Computer Network Simulation with ns-3: A Systematic Literature Review. Electronics. 2020; 9(2):272. https://doi.org/10.3390/electronics9020272

Chicago/Turabian Style

Campanile, Lelio, Marco Gribaudo, Mauro Iacono, Fiammetta Marulli, and Michele Mastroianni. 2020. "Computer Network Simulation with ns-3: A Systematic Literature Review" Electronics 9, no. 2: 272. https://doi.org/10.3390/electronics9020272

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop