1 Introduction

Technology has enabled people to stay connected and has become essential to human life. As a result, mobile devices such as smartphones, tablets, and wearable devices have increased their computational power, and companies are exploring new ways to connect users by providing new interfaces and experiences [37, 44].

Among the many new ways of communication, Augmented and Mixed Reality (AR/MR) has been widely discussed and developed due to its benefits in many different fields, such as industrial support [178], healthcare assistance [174], teaching and training [97], smart cities [164], or marketing and sales [78].

New AR/MR applications have been mainly developed to run on mobile devices (smartphones and tablets) or head-mounted displays (HMDs) [25]. These devices are used for their mobility. Furthermore, although many developments consist of a local process without external information, a tendency in applications that require connectivity has been reported by [116]. Therefore, AR/MR technology is expected to be one of the most relevant applications in the next generation of mobile networks [154].

While using a mobile connection, many AR/MR applications face the challenge of transmitting 3D data and videos for mobile devices and HMDs [12, 85]. Furthermore, unlike video players, latency in AR/MR is crucial to promoting user immersion and reducing MR sickness [27].

The development of 5G networks have changed how people interact with technology. Many authors have discussed using AR/MR over 5G networks and beyond, which can significantly enhance the user experience by enabling seamless and high-quality interaction with digital content in real-time. However, as 5G is still not available worldwide, previous surveys have not had the opportunity to evaluate mobile network limitations in commercial use. Additionally, no research studies have proposed the limits AR/MR can achieve by using 5G and what will be feasible in future networks.

In order to tackle the challenges presented by mobile AR/MR, this paper aims to identify where AR/MR has been studied and developed in 5G networks. The results will be compared with the current worldwide availability of 5G networks, allowing for an evaluation of the results achieved on commercially available networks. Additionally, this paper will address the issues with the current mobile network for AR/MR, explore how 5G networks can contribute to AR/MR dissemination, and highlight the use cases that have been developed using AR/MR and 5G. Finally, based on the identified limitations of 5G, this work proposes a technological roadmap for AR/MR in the 5G era and beyond. These objectives will be achieved through a survey of the state of the art in AR/MR in future mobile networks.

The rest of this paper is structured as follows. Section 2 defines AR/MR and 5G networks. Section 3 presents previous publications arguing for AR/MR in 5G networks. Section 4 presents the research methodology. Section 5 compiles all data obtained from the research and discusses the results. Finally, Section 6 presents some final considerations and proposes the future of MR in 5G networks.

2 Definition and background

Here we present the main AR/MR and 5G Networks definitions.

2.1 Augmented and mixed reality (AR/MR)

Before defining AR/MR, it is necessary to introduce Extended Reality (XR), an umbrella term encompassing AR, VR, and MR technologies [65]. These technologies extend our reality by blending virtual and real environments, creating a fully immersive experience. In AR, digital information and virtual objects are overlaid in the real world [13]. This overlapping is achieved through an estimation process that uses markers, visual cues trained beforehand to be recognized later in the camera stream [76], or characteristic points from the scenario, referred to as markerless tracking [48]. Users can use technology through AR headsets or handheld devices while interacting with and seeing what is happening in front of them.

On the other hand, MR takes the best qualities of AR to create immersive environments where virtual and real objects coexist and can interact with each other in real-time. MR requires an MR headset, such as Microsoft HoloLens, and more processing power than VR or AR.

In contrast to AR and MR, a VR experience fully immerses users in a simulated digital environment [169]. Users must wear a VR headset to view a virtual environment. In VR, most sensory information is computer-generated. Figure 1 shows the XR taxonomy used in this paper.

Fig. 1
figure 1

Extended Reality refers to all real-and-virtual environments generated by computer technology and wearables (figures adapted from [34, 63, 99], and [156])

Full size image

AR/MR technologies offer a wide range of benefits in various fields, including education, healthcare, and entertainment [20]. One of the key benefits of these technologies is their ability to enhance learning and training experiences. For example, medical students can use AR/MR to visualize and manipulate 3D models of organs and structures in real-time, which can enhance their understanding of complex anatomical structures [15, 90]. Similarly, AR/MR can train employees in various industries, such as manufacturing and logistics, by providing them with realistic simulations and training scenarios [97, 168, 177].

Another benefit of the technology is its potential to improve customer experiences and design. AR/MR can be used to create interactive and personalized experiences for customers, such as virtual try-on experiences for fashion products, or virtual tours of real estate properties [122] or realize collaborative activities of design [21].

Ensuring a consistent overlap of objects is one of the main challenges of MR. MR systems must estimate the virtual object’s position and orientation (pose) in real-time [14, 88].

Markers are one of the most common methods used to estimate object poses. They are identified by cameras and compared with previously defined patterns [76]. However, markers are not always the best solution due to MR usage requirements. Markerless methods can be applied in such situations, which can consist of sensor-based and positioning techniques or vision-based methods.

Sensor-based and positioning techniques are generally more straightforward and less computationally expensive than vision-based methods [132]. Some of the most common sensors used for tracking include inertial and magnetic sensors [130]. However, such techniques provide a coarse pose estimation and are not recommended for solutions that require high accuracy [132].

Vision-based techniques are generally more complex and less mature than sensor-based techniques and marker-based tracking [88]. Natural Feature Tracking (NFT) and Simultaneous Localization and Mapping (SLAM) are some of the most commonly adopted markerless tracking methods. In NFT, characteristic points from the image are detected in real-time by the MR system, and the poses of virtual objects are calculated from such points [48]. In the SLAM method, a probabilistic feature-based map is constructed to estimate a real-time camera pose and a position of interested features [40]. In MR systems, SLAM is usually implemented via Parallel Tracking and Mapping (PTAM), where tracking and mapping occur separately, allowing the execution of both tasks in different processing cores [77].

2.2 5G Mobile networks

Fifth-generation mobile networks (5G) propose changing from an operator- and service-centric concept to a user-centric concept [45]. Some requirements of 5G are: 1000 times higher mobile data volume per area; 10 to 100 times higher number of connected devices; 10 to 100 times higher user data rate; 10 times longer battery life for low-power massive machine communication; and five times reduced end-to-end latency [108].

Different architectures were proposed to achieve the network requirements, such as Multi-tier architecture, Centralized-radio access network (C-RAN), and Cognitive Radio Network (CRN) [47, 53, 109]. Figure 2 illustrates a general 5G network architecture.

Fig. 2
figure 2

General 5G network architecture (adapted from [28, 46, 53, 109], and [47]).

Full size image

2.2.1 A multi-tier architecture

The Multi-tier architecture is a hybrid network composed of a macro cell in the upper layer and smaller cells working under macro cells’ supervision in the lower layer [1]. The macro cell encompasses all types of smaller cells of different sizes, such as Femtocell (which raging from 10 to 20 meters for a few users); Picocell (a range of 200 meters and capacity from 20 to 40 users); Microcell (with up to 2 kilometers range for more than 100 users); and Macrocell (with a range up to 35 kilometers and capacity for many users) [109].

The service in the proposed hybrid network is delivered by macro base stations (MBS) and small base stations (SBS). Both are connected via backhauls [158].

The MBS has several antennas spread across the macro cell that connects to the base station through an optical fiber. These antennas are used to identify users belonging to each cell. Aiming to communicate with the MBS and SBS, antennas can be deployed on buildings’ rooftops and vehicles of the transportation system, for instance.

Users with their equipment to access the network, called UE (User Equipment), can communicate via SBS or WiFi, or mmWave, reducing the load over the MBS [46].

In this architecture, detecting and recovering defective cells in the network is extremely important, especially in dense architectures with multiple layers. For this purpose, three self-reconfiguration architectures are proposed [160]:

  • Centralized: a dedicated server is responsible for detecting defective cells by collecting user behavior data. If an abnormal event is noticed in a cell, the server can reconfigure it through the information collected about the system. However, such a configuration requires a high level of communication on the network and high computational cost, which limits its scalability.

  • Distributed: each SBS is responsible for detecting failures in smaller cells in its neighborhood through the analysis of the signals received and information collected about users when passing through the domain of one cell to another (i.e., handover). If any of these nearby cells fail, SBS increases transmission power and hence the access cover for the users of the faulty cell.

  • Local Cooperative: an architecture based on distributed triggering and cooperative detection. Each SBS is responsible for collecting information about users, and if an abnormality is detected, a control message is sent to the dedicated server, i.e., a distributed trigger. Once activated, the server makes a decision based on the information received by all SBS, i.e., a cooperative detection. Thus, we have an approach that does not require communication between smaller cells and offers high accuracy and low latency in its execution.

2.2.2 Centralized-radio access network (C-RAN) architecture

The current architecture of the radio access network (RAN) consists of MBS formed by antennas, a radio unit (Remote Radio Head - RRH), and a Baseband Unit (BBU), which can be located up to 40 kilometers away from the RRH [28].

The centralized radio access network (C-RAN) is an architecture that breaks this paradigm by proposing that a dedicated BBU no longer serves each RRH. Instead, the RRHs are served by a cluster of BBUs in a cloud system.

When virtualizing network mechanisms, the BBUs’ operations can be centralized while their functionalities can be virtualized in central servers [28, 95].

2.2.3 Cognitive Radio Network (CRN) architecture

Ad-hoc wireless networks are composed of self-organized devices and deployed without human support. Although these networks can support different wireless transmission technologies, their operations are mostly limited from 900MHz to 2,4GHz [4]. The operating bands are increasingly congested with the growth in the number of devices, especially in large urban centers. However, other frequency bands licensed to operators, in the 400 to 700 MHz range, are underused for transmission [4, 61].

A Cognitive Radio Network (CRN) is formed by a set of cognitive radio nodes called secondary users (SU), which exploit the frequency spectrum opportunistically, seeking its best use. SU can analyze several different channels, looking for those not allocated by primary users (PU). Thus, when available, the channels can be used by other services, optimizing the availability of frequency bands and reducing interference among other users [135].

A possible CRN-based architecture for 5G networks would be establishing a relationship in which smaller cells would act as SU. Thus, they would communicate with the macro cell using a licensed frequency band while providing service to the UE through opportunistic access, interfering to a minimum in the activities of the macro cell [109].

3 Related works

Aiming to identify reviews concerning AR/MR application on 5G networks, a search in ACM, IEEE Xplore, Scopus, and Science Direct libraries was accomplished in December 2022 using the same search string proposed in the research methodology (Section 4). The results of this search, along with their primary conclusions, are presented in Table 1. It is important to note that a comparison between these findings and those presented in the current paper can be made, as shown in the final row of the table.

Table 1 Comparison between related surveys and the current paper
Full size table

The related work from [106] reviewed the state of the art in VR and AR technologies and how the 5G network supports virtual applications demands. The authors also present expectations for the future of AR applications and challenges for 5G.

In [116], the AR/MR is discussed in mobile networks before 5G, and the authors highlight that the limited networking and computing capability hinder the technology’s practical application. Then, the authors compared possible network architectures in 5G and how the AR/MR can be benefited from each one. Finally, the paper discusses the challenges of AR/MR on the 5G networks.

In [55], the problem of latency and its impacts on the quality of service (QoS) and quality of experience (QoE) for different technologies are discussed, which includes AR. The authors focus on the importance of 5G networks for future ultra-reliable, low-latency AR applications and the network’s open issues and challenges.

Recent articles have discussed the potential benefits of 5G and beyond, with AR being one of the technologies that could reap the rewards of these networks. In one such article, [175], the authors describe an overview of 5G and an introduction to 6G. Unlike previous authors, they anticipate an increase in AR applications only with 6G networks due to higher data rates, lower latency, more efficient spectral efficiencies, increased energy efficiencies, and improved network capacities.

AR has been considered a core technology [94], bringing potential towards holographic communication together with 6G. In [157], the authors discuss the technology and how it can change communication as mobile networks become more reliable while pointing out the importance of new signal processing techniques to address security in the AR future world.

Another review concerning AR challenges in 5G is presented in [68]. The paper discusses edge computing for AR, its benefits, and challenges regarding offloading tasks. The authors also highlight the trade-off between high accuracy tracking and low latency; the latency tends to decrease as the accuracy increases.

In [140], the authors discuss mobile AR applications and the current and future network architectural options to support such applications. The authors also describe technical requirements such as communication, mobility and energy management, service offloading and migration, security, and privacy.

Finally, in [3], authors examine the data rate and the latency necessary to implement AR/MR applications using a mobile network. Their analysis concluded that public 5G networks could not support applications for wide areas or with high interactions. However, 6G and next-generation WiFi systems will allow AR/MR as an efficient communication tool.

Unlike the studies mentioned above, in this paper, we review from before 5G until new developments over 5G networks. The results propose a road map of the technology based on the network limitations and challenges, which was not presented before. Our analysis extends 5G, and we point out applications that will probably be feasible only in 6G networks. Additionally, we discuss some opportunities for future AR/MR devices beyond 5G.

4 Research methodology

This survey was conducted by three authors from two different institutions whose research domains are: Extended Reality, Computer Networks, and Distributed Systems. The methodology used to conduct this survey was based on the guidelines proposed by [23], which consists of three phases:

  • Planning: in this phase, it is defined the research questions and the review protocol;

  • Conducting: this phase consists of identifying previous related papers, selecting and evaluating the primary studies, and extracting the interest data;

  • Reporting: finally, this phase defines the communication strategy to publish the results.

4.1 Search strategy

The strategy to define the search string consists in identifying what has been published concerning AR/MR and 5G and beyond networks and understanding the AR/MR systems that use mobile networks. Therefore a combination of the technologies’ names was implemented to create the following search string:

("augmented reality" OR "mixed reality") AND ("mobile network" OR "6G" OR "5G" OR "4G" OR "LTE" OR "3G")

The search was accomplished in ACM, IEEE, Science Direct, and Scopus databases. First, we searched the string in the title, abstracts, and author’s keywords. The results were filtered to select only manuscripts published in journals and written in English.

The first author realized the screening of the identified papers. After categorizing the papers as accepted or rejected, the second and third authors reevaluated the groups to validate the decision according to the criteria defined in this survey protocol.

The software StArt (State of the Art through Systematic Reviews) was employed to support this survey. StArt provides a protocol to conduct a survey [79] relevant results are reported in [59] and [155]

4.2 Research questions

The research questions and the motivation that this survey aims to answer are presented in Table 2.

Table 2 Research questions and motivations
Full size table

4.3 Inclusion and exclusion criteria

We considered only articles and surveys for this study and extended written in English and published in journals from previously cited databases until the end of 2022. We excluded manuscripts that do not consider AR/MR or do not consider AR/MR using a network. Moreover, non-academic publications such as commercial literature, reports, and posters were also excluded.

4.4 Selection procedure

After accomplishing the search strategy, 401 articles were found. However, 87 of those appeared in more than one database and were considered duplicate studies.

After eliminating duplicated manuscripts, a total of 314 publications were screened and assessed to determine if they addressed AR/MR and mobile networks. Among them, 109 manuscripts were identified as relevant for data extraction. Figure 3 outlines the selection process across the databases, highlighting the reasons for the rejection of manuscripts based on the inclusion and exclusion criteria.

Fig. 3
figure 3

Selection procedure to define primary studies

Full size image

4.5 Data extraction

Once the papers were selected, the next step was reading the entire work to extract the information that answered the questions of the present study. The following collected data from the articles were extracted:

  • Title;

  • Publication Year;

  • Published Journal;

  • Research category (article or survey);

  • Country of work of the first author;

  • Authors’ affiliation;

  • Application context (mobile network before 5G, using 5G or beyond 5G);

  • Network Infrastructure;

  • Challenges or limitations of the application in the mobile network;

  • Advantages in the use of the mobile network compared with previous generations;

  • For the applications that used 5G and beyond networks, If the development used a commercial network or a testbed/simulator.

Finally, a digital form to record the information of the 109 primary studies was prepared. This phase also adopted the stArt tool to compile all the information. The results were validated for the other two authors according to their main research backgrounds.

5 Bibliometric results

Firstly, we separated the selected manuscripts into three groups. A group of papers concerning applications running until 4G network, publications arguing about AR/MR in 5G, and research of AR/MR in 6G. Figure 4 shows the distribution of each group per publication year. As it is possible to note, the interest in AR/MR applications on mobile networks increased significantly from 2019, when the first 5G networks became commercially available.

Fig. 4
figure 4

Identified publications per year

Full size image

Secondly, we identified the publication’s source. Figure 5 shows where the selected papers were published. Among the selected articles, 14 journals had more than one published paper. The top three journals by the number of published papers were IEEE Access (11 papers), IEEE Internet of Things Journal (9 papers), and IEEE Transactions on Broadcasting (6 papers).

Fig. 5
figure 5

Journal where articles were selected

Full size image

Finally, we analyzed authors’ affiliation Fig. 6 shows institutions that had more than one article selected, and the links represent that these institutions worked in partnership for publications.

Fig. 6
figure 6

Authors’ affiliations and institutions co-authorship

Full size image

Although most authors’ affiliations are research and education centers, the telecommunication and healthcare industries also appear as industrial segments researching AR/MR in mobile networks. Figure 7 shows the industrial domains of the authors and co-authors.

Fig. 7
figure 7

Authors affiliations domain

Full size image

6 Mixed Reality main difficulties in pre-5G networks

The evolution of mobile network generations had raised the expectation that 4G technology would facilitate the development of AR/MR applications using mobile networks. This expectation was fostered primarily due to the upstream traffic flowing at higher transmission rates and lower latency than the previous network generation, allowing rich context data to reach back-end servers to be processed [166]. A few applications were presented, such as poster presentations [12] user guidance [166].

However, with the 4G deployment, many AR/MR expectations were not fulfilled, and issues have remained open. Two significant challenges for AR/MR applications using mobile networks are web browsers and external processing.

The authors highlighted that most applications requiring a web browser had not achieved the minimum computational capacity to track and render information at the necessary frame rate. The primary reason is the computing inefficiency of scripts like JavaScript in a mobile web browser for dense computing tasks [116, 117]. Additionally, web browsers raise concerns about a lack of standards for capturing, processing, and generating holograms [115, 116].

Considering applications that do not require a web browser, the network latency and bandwidth were the most mentioned issues [55]. Although the 4G made feasible, the connection between AR/MR systems and information located externally the device [32, 104], the network could not offer latency lower than 15 frames per second (fps), data transfer rate over 100 Mbps and packet error rate under 10-5 [54, 67, 85, 92]. For this reason, those applications used network connections only to receive and send data, maintaining all the graphic computing processes locally.

7 The augmented and mixed reality in 5G networks

The advent of 5G has the potential to significantly transform AR/MR applications, as it enables users to have greater mobility [94]. As a result, several authors have proposed changes in software development, including using diverse architectures based on the application’s purpose. In addition, as devices move towards more ergonomic designs with better energy consumption, the AR/MR user experience is expected to evolve. However, achieving this transformation in AR/MR involves overcoming several challenges to provide the best experience for future users.

7.1 Architectures for augmented and mixed reality

Analyzing the selected manuscripts, we identified possible architectures to operate AR/MR applications in 5G networks. Regarding the core processing functionality location, the authors segregated the architectures into three groups: local, edge, and cloud.

7.1.1 Local architecture

The local architecture consists of applications that process the data within the device or in a server located in the local network. In such an architecture, the 5G core network updates the software, backup, or transfers information among devices [73].

This architecture allows the benefit of using the same security and privacy policies and standards as the internet. Moreover, it is less complex to be implemented as data transfer can be based on standard protocols. Therefore, compared to previous mobile networks, 5G allows data transfer to be faster and more extensive.

As a disadvantage, it requires the devices to have a high capacity to process all the information and power capacity for continuous use. These challenges are some of the issues that HMD and mobile devices may face in 4G networks to run AR/MR applications.

7.1.2 Edge architecture

The edge architecture operates upon two different modes. First, as a client-server model where the server is hosted at the network edge executing computationally intensive tasks, such as pre-processing, AR tracking, and rendering the augmentations [18, 103, 119, 140]. In addition, it is possible to operate as a cache located in the SBS and the users’ devices, especially to cache background scenes. The caching strategy can be facilitated by distributed caching schemes rather than downloading the content from the SBS, so that content can be shared using D2D communications [30, 144].

This architecture requests the remote cloud server when the application is not deployed in the edge server or cache [86, 115]. Computation-intensive tasks that are difficult to be satisfied at the edge are then assigned to the cloud server [84]. On the other hand, moving the services from the remote cloud to the network edge reduces the communication delay and the bandwidth usage of core networks [50, 52, 69, 84, 116, 163]. Furthermore, this architecture’s energy consumption is reduced compared to local processing due to the reduction of processes that occur in the device [10, 30, 172]. The energy saved is outstanding for complex applications, not being so different for low fps uses [41].

In edge-based architectures, new challenges arise concerning security and privacy. Different heterogeneous network elements are collocated in the edge infrastructure, making the conventional privacy and security mechanisms inapplicable. Also, the data offloading over wireless channels may not be secure since malicious eaves-droppers can capture computation tasks [113]. Furthermore, when edge caching and computing are used, an individual’s information contents, such as visited locations, may be shared with other users causing privacy and security concerns [136, 144].

It is crucial to provide flexible and shared infrastructure to reduce the dense SBS and deployments of MEC Points of Presence and investments in network operations, especially in dense urban zones [35, 96, 153]. Furthermore, the challenge is to face the high number of possible edge servers and the incoming user requests with applications that require low latency. Therefore, it is essential to comply with the applications’ latency thresholds and reduce the latency variance among users in the same session [5].

The size of the cache, cache management, and computing costs are all open issues that require further study. As processing tasks and cached content become more complex, it is important to correctly define which content should be cached and where it should be cached, as well as determine what should be processed in the edge server and which server it should be processed on [39, 83, 144]. Access to the edge server or cache also requires specific algorithms to ensure fair access and avoid user competition for accessing the edge resources, which can lead to reduced quality of experience (QoE) [62, 123, 124, 136, 144].

7.1.3 Cloud architecture

A cloud-based architecture system operates through the client-server model. where computationally intensive tasks are executed on a remote server. Such an architecture has appeared in pre-5G AR/MR applications [115, 116]. Besides the latency reduction in 5G, cloud architecture is recommended for those applications that do not require low latency [41, 140].

Compared to local architecture, a cloud can also offer energy saving to the user device, as only pre-processing tasks are accomplished in the device [144]. Also, application management is more straightforward as most of the software is deployed on the server side, avoiding accessing each device for updates.

On the other hand, cloud architecture is centralized, so the application may be unavailable whenever the server fails. Thus, this architecture highly depends on the infrastructure conditions [140]. Further, as in edge architecture, critical information is transferred outside the user device, so privacy and security issues must be considered [136, 144].

7.2 Augmented and mixed reality developments using 5G

We conducted a study of selected articles in which authors discussed 5G and beyond networks to identify the origin of the research. Our aim was to determine if the published studies originated from countries where 5G networks have been deployed. Figure 8 shows the countries with 5G networks worldwide and the results from the selected studies. The information regarding network availability is up to June 2022 [22].

Fig. 8
figure 8

5G networks availability and AR/MR studies using 5G (adapted from [22]).

Full size image

The map categorizes the countries into three groups:

  • 5G networks launched: Countries where 5G networks are commercially available from standalone or non-standalone infrastructure (networks that still depend on LTE network);

  • 5G technology deployment: Countries where operational tests have been performed, but the service is still not available;

  • Investment in 5G: Countries where investments have been made but no network infrastructure has been deployed.

Comparing the countries with any available 5G network and the origin of research, we found that countries publishing about 5G and beyond have already launched their networks. The only exceptions were studies conducted in Morocco [43], Egypt [107], Pakistan [118], and India [19, 26, 31, 54, 55, 121]. In these publications, authors conducted surveys, published conceptual papers, or emulated 5G using commercial simulators, such as EdgeCloudSim.

Regarding publications from countries with available 5G networks, 70% of the studies used a testbed or emulators to validate the AR/MR application. However, this scenario has gradually changed, and in 2022, almost 40% of the identified publications tested the applications on a commercial 5G network. Figure 9 shows the change over the years.

Fig. 9
figure 9

Use of commercial 5G or testbeds per year

Full size image

7.3 Augmented and mixed reality use cases using 5G

With the deployment of 5G networks, many studies have been published discussing the use case, benefits, and constraints. Table 3 presents the use cases we found and their network characteristics. This table did not include studies proposing general AR/MR applications. Instead, we grouped the manuscripts according to their primary area. For each study, we consider:

  • User mobility: if the user is static or on displacement. We consider under displacement if the user movement is faster than walk;

  • Navigation system: we analyze if the application is suitable for indoor or outdoor contexts;

  • Suitable context: if the application is used in an urban or rural zone;

  • User interaction: if the application allows single or multiple-user interaction in the same scene;

  • Communication architecture: if the application operates in client-server or D2D model;

  • Processing architecture: for a client-server application, one of the 5G architectures was employed.

Table 3 AR/MR use cases in 5G networks
Full size table

Most studies we surveyed consider scenarios where users are static or on short or slow movement. This characteristic is presented in both indoor and outdoor navigation systems. In these applications, users are not crossing through base stations (i.e., handovers). Another relevant fact is that most of the studies occurred in testbeds, which tends to simplify the network architecture. Therefore, there is a gap in the literature to exploit scenarios where users frequently change their base stations (i.e., fast handovers) while using AR/MR applications.

The Client/Server model has been predominantly deployed over D2D communications in the communication architecture. Another relevant found was that most applications propose an edge architecture due to the potential for latency reduction. Moreover, such an architecture has been considered a trend in 5G and beyond networks to offload complex tasks.

Finally, we analyze how users interact with the proposed systems. Only one study considers a browser application [127]. While all the other applications are standalone systems, it is still a gap to answer if browser standards will be feasible and reliable in the future of AR/MR, using 5G and beyond networks. Also considering user interaction, one study presented an analysis of haptic AR/MR over 5G, [102] highlighted open issues considering this interaction.

AR/MR developments with 5G networks tend to offload some tasks to an edge server or cache to provide a better user experience. Also, those applications can be used for industrial, commercial, or consumer use cases. However, the literature still has a gap in understanding how those systems will behave in scenarios of high demand, fast handovers, and coverage instability. Finally, we could not find evidence of web AR/MR use in the 5G networks, as most studies proposed standalone applications.

7.4 Challenges to augmented and mixed reality in 5G era

From the studies we surveyed, the AR/MR efforts in 5G have been centred on a few challenges. Figure 10 illustrates a word cloud of challenges.

Fig. 10
figure 10

Main identified challenges word cloud

Full size image

7.4.1 Data traffic and network infrastructure

One of the most significant challenges for 5G network deployment is traffic estimation and infrastructure cost. The need to install numerous antennas to cover large areas can increase the mobile network cost and overall energy consumption, as it requires more edge servers to remain in operation [71, 159]. Therefore, an effective deployment algorithm for solving the hyper-dense deployment problem of 5G networks has become a critical research topic, as it is essential to meet the network requirements [91].

Understanding AR/MR in 5G networks is vital for correctly scaling investments in infrastructure. Remote rendering of AR scenes requires powerful GPUs to stream the AR experience in real-time [10]. For example, in South Korea, 5G data traffic is projected to increase to 6340 PB by 2025, serving 26502 million users, with most of this content associated with AR/MR [138]. This trend is seen worldwide as AR/MR devices become cheaper [111].

Infrastructure issues can create bottlenecks in any of the 5G system layers, degrading the overall end-user experience [60]. Therefore, important countermeasures are necessary, such as implementing policies to ensure the appropriate allocation of resources and support to the layers of 5G systems. Balancing computational demands of both service subscribers and providers is also essential to ensure proper system functionality [60, 62, 102, 127].

Finally, 5G networks must ensure reliability in a dense environment and allow for user mobility, as mobile users can change their positions during the computation offloading time [56].

7.4.2 Security and privacy

Security and privacy standards were the second most challenging mentioned. The security challenge for AR/MR can be faced on three levels: input, data access, and output [131]. Therefore, AR/MR architectures are susceptible to security issues in different layers, such as communication and data processing.

The data input level concerns protecting data acquired from the user device. Therefore, input protection should ensure that sensitive information collected from cameras and sensors (e.g., environment elements further than the AR/MR interest context, people’s faces) should be removed before offloading the data [140, 157]. As such, countermeasures such as computer vision algorithms and machine learning techniques capable of capturing only relevant objects can be applied.

The data access security level concerns the protection of data transferred among users and servers. Such a level should be implemented in servers or devices, in cases of device-to-device (D2D) communication [53, 140]. The data access should prevent individual private information from being shared with other users [55, 136, 144]. Maintaining network privacy and potential anonymity will challenge the next 6G network architecture due to the dynamic behavior of devices joining/leaving mobile networks [157]. Some approaches to deal with the data access security level include the use of blockchain integrated with edge computing to achieve secure authentication and collaboration with trusted distributers [139], as well as cryptography techniques with machine learning [57].

The output level concerns the protection of received data. As this communication may occur from public networks, ensuring that the received data are trustworthy is crucial. Spurious data may display the unintended result, improper positioning tracking, harming the users [17, 131, 140, 167]. Thence, providing security countermeasures in the user devices is imperative to ensure the privacy of users [120].

In summary, security and privacy challenges in AR/MR systems include ensuring system availability under multiple attacks, device and data anonymity, mutual authentication between users or servers, and unlinkability [167]. Countermeasure mechanisms have to consider data integrity and security while maintaining low latency for the end-user [68].

7.4.3 Latency and bandwidth

In the context of AR/MR in 5G networks, latency and bandwidth are critical factors for a successful user experience. Latency is a significant challenge as it encompasses both device and MEC service latency and reliability [92].

The latency requirement of AR/MR cannot be achieved by a 5G network, particularly in cloud architectures, when the core network is far from the user [141]. Moreover, the response time increases with the complexity of virtual objects and accuracy in tracking [68, 74, 114, 136, 137].

Recent studies have shown that even in 5G, latency-sensitive tasks such as telesurgery are still problematic due to the critical response time and unacceptable connection breakdowns [161].

To address these issues, it is necessary to achieve a per-user data rate of gigabit per second for AR/MR applications to meet the quality level of user experience. As encoding and decoding data are time-consuming processes, the transmitted data cannot be compressed, making high bandwidth essential for the optimal functioning of AR/MR applications [43].

7.4.4 Efficient resource allocation

Efficient resource allocation is another significant challenge in AR/MR systems due to the presence of devices with heterogeneous services and platforms. The allocation of communication and computational resources is a complex task, as it must balance the requirements of different devices and services. Previous studies have shown the need for efficient resource allocation in 5G networks [86, 92, 113, 134, 171, 172].

Transmission and synchronization of heterogeneous data streams in AR/MR systems may face challenges due to packet loss, higher packet rate, and variable delay [136, 143]. To overcome these challenges, technologies such as dynamic time division duplexing can offer a flexible configuration of the downlink and uplink channels ratio to adapt to the asymmetric traffic of AR/MR applications [142].

Another critical aspect of efficient resource allocation is the need for ultra-reliable low-latency communication (uRLLC) networks to provide full coverage with no dead zones and an availability of around 100% [107]. This ensures that AR/MR applications can operate with high reliability and availability.

Finally, as future AR/MR applications are prone to allow users’ mobility, flexible edge server collaboration [56, 75, 112, 126] and new techniques to improve streaming [29, 110] are required to provide seamless and uninterrupted services to users.

8 The future of augmented and mixed reality beyond 5G

Efforts to evolve networks beyond 5G are underway in research centers worldwide, with significant efforts focused on exploring new communication mechanisms to integrate networks across space, air, oceans, and land to achieve seamless global coverage of integrated information [94, 107].

Studies on 6G envision a network generation supported by autonomous devices, such as terrestrial and aerial autonomous vehicles, to implement mobile edge servers, with data forwarding to cloud servers in the absence of support for compute-intensive tasks at the edge [54, 150]. Algorithms should be designed to optimize offloading policies and consider the impact of the current offloading decision on the execution of subsequent tasks [81, 103].

Although 5G networks can improve QoS from enhanced mobile broadband and ultra-reliable low-latency communications, there are no dedicated service definitions to best support AR/MR related tasks [9, 42, 149].

Current TCP and UDP protocols cannot guarantee a low upper bound of end-to-end latency and are not designed to utilize available network assets optimally. Therefore, they cannot meet the QoS requirements for AR/MR applications [145, 148]..

For the future generation, a paradigm change to information-centering networking is expected to enable content, cache, and routing optimization and increase QoS for inter-personal communication for AR/MR applications [2, 75]. Furthermore, the 6G might face the challenge of supporting adaptive and customizable connectivity, e.g., using AI to predict the next AR/MR application pose [7].

6G networks are expected to offer GHz to THz frequency, increasing bandwidth and supporting data offload for real-time mapping, tracking, and object recognition in parallel tasks for AR/MR systems [26, 43]. This increase in latency and bandwidth will allow for a high-fidelity AR/MR interaction and even ubiquitous holographic information interaction [87, 105].

8.1 A roadmap for the future of augmented in mixed reality in mobile networks

This section proposes a development roadmap for AR/MR over 5G and beyond networks based on the studies we found. Figure 11 shows the expected evolution of the network technology.

Fig. 11
figure 11

A roadmap for AR/MR regarding 5G and beyond networks

Full size image

As discussed in the previous sections, standard AR/MR support from 5G networks is to offload complex tasks, especially using edge servers or caches. Due to the complexity of data and privacy protection, the initial part of complex processes, e.g., tracking, can be offloaded. At the same time, pre-processing images and private data can be accomplished on the user’s devices.

With the increment of 5G deployments and the development of dense networks, more high computational effort tasks can be offloaded. As a consequence, the devices can achieve a reduction in energy consumption. In this scenario, a new opportunity for AR/MR headsets development is available, as the existing AR/MR hardware suffers from ergonomic aspects and battery autonomy [33, 66, 165, 173].

More ergonomic devices bring an opportunity for AR/MR to be one of the communication tools in the future. These applications can use D2D communication, allowing communication among users in the same space and interactions among users and smart things.

Additionally, studies of multisensorial experiences over 5G have identified that the network can achieve satisfactory QoE in stable scenarios [36, 152, 176]. However, there is no evidence that this scenario could be realized in commercial 5G networks. First, there is a lack of protocols to deliver multisensorial data [101]. Also, due to the network limitations to deal with changeable and unpredictable network conditions. [36].

We identified in 6G studies that the focus of such a generation network for AR/MR applications would be enhancing users’ social interactions. From 6G networks, it is expected that users can interact simultaneously with each other together with the virtual environment [58]. This communication can encourage the creation of supporting technologies and facilities such as AR/MR-based web browsers. Furthermore, the web-based AR/MR approach overcomes the cross-platform and extensive provisioning limitations inherent in both device and app-based AR/MR applications [120].

With the higher bandwidth, lower latency, and higher reliability of 6G networks, digital twin models can be visualized and interacted with in real-time, using AR/MR [24]. This will enable users to experience and manipulate digital twin models, leading to enhanced collaboration [6]. Moreover, 6G technology will enable digital twins to be used in more complex and dynamic systems, such as smart cities, autonomous vehicles, and advanced manufacturing [100].

Finally, with massive coverage of 6G networks and cross-platform AR/MR navigation mode, a new web surfing approach much more immersive and haptic sensitive may be available to users. Thus, future devices will be able to receive and send a large amount of data, increasing the user experience to a new level by exploring users’ senses (e.g., sight, hearing, and touch). Consequently, the multisensorial experience would make the virtual environment more realistic and integrated into the real scene.

9 Conclusions

This paper complements previous AR/MR surveys by compiling what and how is expected from AR/MR over 5G and beyond networks. Furthermore, the survey analyzes studies to expose how the related technologies have been combined with 5G networks and future generations of mobile networks.

While a comprehensive search for studies on AR/MR applications in mobile networks was conducted, it is important to acknowledge that there may still be a possibility that a few articles were missed during the literature review process. Additionally, the exclusion of conference proceedings may have impacted the overall comprehensiveness of this review. Moreover, it is crucial to note that this survey was conducted until the end of 2022, and any developments beyond that period were not considered. Therefore, these temporal limitations should be considered when interpreting this study’s results.

The development of AR/MR applications that impose QoS requirements from mobile networks has been proposed since the 4G advent. However, before 5G, such a development could not meet the latency and bandwidth requirements of nowadays. Therefore, the previous mobile networks would be best recommended for applications without high fps requirements.

From the first deployments of 5G networks, authors identified the possibility of offloading part of high computational resource tasks to be processed in more powerful servers. If those servers are located on the network edge, then edge-based architectures can meet AR/MR latency requirements. This can help organizations to develop AR/MR applications that offer improved performance and responsiveness.

Additionally, the possibility of offloading high computer demand tasks brings new opportunities to AR/MR hardware manufacturers. Hardware, especially HMD, should become smaller, lighter, and more ergonomic if the high computational effort tasks will no longer proceed internally. These future devices can stimulate AR/MR use in people’s daily life.

However, the proposed architecture brings new challenges concerning traffic estimation, infrastructure cost, security, and privacy. Understanding these challenges can help telecommunication companies plan and develop effective infrastructure to make AR/MR ubiquitous and accessible.

Furthermore, this paper has identified an important gap in the current research, highlighting the lack of studies discussing the use of AR/MR applications on 5G networks in scenarios with high demand, fast handovers, and coverage instability. Despite the extensive research on the topic, no studies published until the end of 2022 have addressed these specific scenarios, which presents a valuable opportunity for future research.

Additionally, we analyze the tendencies toward AR/MR for future generations of mobile networks. Finally, a roadmap of the technologies is suggested based on the studies we found and the development tendencies we identified.

Based on the roadmap, it is expected that in a dense 5G network scenario, users will be able to use AR/MR in displacement scenarios, offload most of the complex processing and take advantage of D2D AR/MR communications.

Also, the roadmap analysis indicates that although AR/MR internet research and multisensorial experiences have been conducted in the 5G scenario, these use cases would probably be feasible beyond 5G networks. This is expected due to the lower latency and higher bandwidth that future networks will offer and the development of new protocols to support them.