RealityMedia: immersive technology and narrative space

1 Introduction

The theory and practice of VR narrative are usually understood in relation to and as developing out of the tradition of 3D videogames and other interactive digital forms, and these in turn are often understood on the model of the film as visual narrative or perhaps even oral storytelling. Film and storytelling are appealing as models because of their apparent immediacy, reaching the viewer or listener directly through their sense of sight or hearing. And VR too has often been promoted as also being immediate, the medium to end all mediums, reaching the user directly through their senses. There is another tradition through which VR narrative can be approached: as a new form of writing or inscription (Gnanadesikan, 2009). Inscribing signs in a surface such as clay, wood or stone is probably the oldest form of writing, and writing on monuments and buildings continues to this day. Monumental inscription involves deploying text in space, often for public display, e.g., inscriptions listing soldiers who have died in a war. In a sense writing in any medium, including ink on the pages of a printed book, is spatial, and the affordances and conventions of the medium determine or at least suggest the order in which the text is to be read. In this tradition, constructing VR texts can be understood as a process of inscribing elements in space, and consuming the text becomes a process of “reading” the space.

This is the approach we take in this paper. We explore VR as a new writing space in the long tradition of inscription. VR texts can be multimedia, including not only written words, but also images, video, audio, and even interactive elements. Because VR is inherently spatial and immersive, we have looked in particular to the other examples in which artifacts and texts are organized in space: the library and in recent centuries the museum. In our current project we are focusing on the museum as a model, because a museum typically contains various media forms including written text, images, and often videos. Museums are thus writing spaces with multiple forms of inscriptions, and, as various theorists have noticed, they may contain a complex of interwoven narrative lines: stories that may be embodied in a single object on display supported by wall copy or may extend throughout a gallery or throughout an entire theme-oriented museum (e.g., The Yad Vashem Holocaust History Museum in Jerusalem or The National Center for Human and Civil Rights in Atlanta).

In considering a VR space as affording a new kind of writing space, we see that, as opposed to the linear space of a printed book, the narratives embodied in the physical space of a museum can be analyzed on three distinct levels: (1) the architecture of the space itself, (2) the collection as a whole, and (3) the individual artifacts. The levels can present multiple narrative threads. From this perspective, the task of curation/writing becomes how to organize and present elements at each level in order to facilitate the “reading” of the various threads by the visitor/user/reader. How can VR address this task, both by remediating strategies familiar from physical museums and the various media contained within them and by offering new affordances not available to brick and mortar, glass and steel museums?

To address these questions, we have designed and implemented a testbed called RealityMedia. This digital environment is a counterpart to the printed book Reality Media (MIT Press 2021) written by three of us (Bolter et al., 2021, Engberg, and MacIntyre). This book is not fiction; it is a non-fiction essay on the history of media. We will be using the term “narrative” in a broad sense to include not only fictional stories but also non-fiction that conveys its ideas in a discursive form, e.g., biographies, essays, and other kinds of theoretical or historical works. All these kinds of texts, including Reality Media, are narratives in the sense that they convey ideas to be consumed by the reader in a particular order. As we noted above, VR is often seen as the new form of (fictional) film or videogame. We want to explore whether VR can also be used to represent a broader range of texts. Reality Media uses typical conventions to organize and present its verbal ideas in the 2D space of a printed book: e.g., paragraphs, pagination, headings, chapters, and an index and reference list. The digital counterpart remediates the printed book in two forms: as a set of web pages and as a fully immersive VR space. (To explore both, visit https://realitymedia.digital.). The web pages serve as a conceptual bridge between the printed book and the VR space; they also link the user directly into the VR space, implemented in Hubs, a browser-based WebXR platform. To facilitate meaningful reading experiences among the users, we used VR to organize traditional media content and explore innovative VR narrative techniques. The 3D VR version remediates the physical book and also the museum space. It consists of a series of “galleries,” many of which resemble rooms in a conventional museum, although some of the galleries recall other built structures (e.g., the amphitheater). The galleries present 2D content in the form of text, images and videos, while all the galleries are accessed from a central rotunda via teleporting portals.

In addition to the 2D content, RealityMedia also makes use of a set of expressive media forms facilitated by the VR medium:

The meaningful arrangement of 2D content in the 3D architecture of each room;

3D visualizations of the ideas contained in the printed book in the form of manipulable “force graphs”;

Panoramic spheres and complete immersive experiences that are embedded in galleries but visited as independent world spaces (e.g., a virtual trip to the surface of Mars, a virtual Acropolis, and a recreation of a well-known pit experiment).

The virtual galleries offer the same themes and often examples found in the printed version; however, the content balance is different. The VR version contains more images and much less text and features multimedia, such as audio and video not available in a printed book.

Our research objective is to explore the meaning-making process afforded by spatial narratives in VR—to test whether VRfacilitates traditional ways of weaving complex, multiple narrative strands and provides new opportunities for leveraging space. To accomplish this, we conducted a user study with 20 participants. Our results highlight how the three semantic levels (space, collection, and artifacts) can work together to constitute meaningful narrative experiences in VR. Our findings can provide useful insights for researchers, storytellers, and game developers to leverage space and narrative devices that can carry over into fictional narratives. We seek to identify promising storytelling strategies of VR to constitute a new kind of “immersive book,” and we acknowledge at the outset that the remediation of the museum is only one of many possible avenues toward that goal.

Here are the questions we address within the Storytelling in Virtual Reality research topic:

What earlier media and media forms can serve as models for conveying narrative and non-narrative information in VR?

Can we use the spatial, immersive, and interactive affordances of VR to convey narrative and non-narrative more effectively than in these earlier media. And if so, how?

How do the readers/users interact with the immersive book? What are their perceptions of the spatial narratives in VR?

2 Related work

We are exploring how visitors conceptualize and make meaning from their experience of our immersive gallery space both in itself and in relation to a website that links to and contextualizes that space. Meaning-making in this context involves understanding and assimilating this new experience into the visitors’ web of prior concepts and experiences. As we have said, the 3D version of RealityMedia resembles a museum or exhibition hall. It remediates a museum in the sense that visitors understand how to appreciate its spaces as they would a set of galleries in a museum, examining exhibits in turn. It could also be said to remediate a 3D adventure videogame that consists of a series of space to explore, but its relationship to the videogame is less important because the visitor will soon realize that there are no non-player characters, no objects to amass for later use, no puzzles to solve, and no winning state to achieve. The remediation of the museum likely remains foremost in the visitors’ minds, and visitors come to RealityMedia with a tacit, if not explicit, understanding of how museums can embody narratives and discursive arguments. As Chronis (2012) points out, “upon entering the experiential space of the museum, visitors bring along their existing knowledge and experiences related to the museum presentation.”

2.1 Museum narratives

In recent decades, museums have been reimagining themselves from being a repository of artifacts to providing a place for visitor engagement and meaning-making (Hooper-Greenhill, 2000, p.152; McCall and Gray, 2014; Nielsen, 2014). As a result, curators have begun to consider how to effectively facilitate the construction and transmission of these narratives to visitors. To achieve this, the museum delivers narrative through three spatial organizational levels, architecture, gallery, and individual object (Witcomb, 1994).

At the architectural level, the messages of the museum are transmitted through various sensory channels, such as sight, touch, and movement (Bennett, 1995). The visual and haptic qualities of the architecture, from its form, and lighting to material, function as a medium of communication with visitors and become an essential part of the interpretation (Psarra and Grajewski, 2000; Schorch, 2013; Lu, 2017). The Yad Vashem Holocaust History Museum is a notable illustration of this, with its geometric form of architecture, exquisite lighting, and the constant use of concrete material together recounting the historical times during the Holocaust (Lu, 2017). Similarly, research has shown that the castle-like architecture of the Museum of Scotland aligns with the message of the medieval traditions of Scotland (Psarra and Grajewski, 2000). Furthermore, the spatial layout of the museum shapes the visitor’s kinesthetic experience and, thus, the progression of the narrative (Bennett, 1995). For example, a configuration that regularly alters the directions constrains the amount of information visitors obtain at once (Hiller, 2003). Studies on visitor itineraries in space often discuss the degree of integration as it influences how visitors read the content (Psarra, 2005; Hiller and Tzortzi, 2006; Tzortzi, 2007; Wineman and Peponis, 2010; Tzortzi, 2014). According to Psarra (2005), a more integrated space allows for an easy transition between areas, potentially leading to a weaker narrative as it enables multiple interpretations.

At the gallery level, groups of exhibit objects are spatially brought together as a collection and make statements on overarching themes such as historical events (Hooper-Greenhill, 2000, p.124). This gallery-level story is conveyed directly through written descriptions as well as through the selection of objects, their arrangement, and the juxtaposition (Hooper-Greenhill, 2000; Wolff et al., 2013). Also, the positioning and size of the gallery spaces express either central or peripheral messages (Psarra and Grajewski, 2000). For instance, galleries that can be directly accessed, have much distant visibility and connection can be said to be privileged (Tzortzi, 2007). Furthermore, the storyline at this level not only helps to connect the themes of the other galleries but also links the underlying stories of the individual object (Wolff et al., 2013).

More detailed and concrete stories are conveyed at the individual exhibit level. For this, the exhibit objects and labels serve as the fundamental elements and structure of the narrative (Noy, 2020). These interact with the visitor to convey inscribed ideas and memories (Hooper-Greenhill, 2000). Through various multimedia, such as exhibit objects, images, text, video, and audio, a narrative hierarchy is constructed to create conceptual relationships, suggesting a viewing sequence (Wolff et al., 2013). In organizing the hierarchy, curators often borrow techniques from the book, such as hooks and overall tone, to engage visitors (Dean, 1994). In summary, narratives crafted at these three levels of spatial organization (architecture, gallery, and individual object) shape the visitor’s experience and contribute to the museum’s overall message.

Several research studies in the field of VR have utilized the structure of the museum to craft and communicate a message (Walczak, Cellary, and White, 2006; Pan et al., 2009; Kersten et al., 2017). These studies have highlighted the unique visitor experiences made possible by VR technology, such as the ability to freely navigate and closely examine exhibits (Kersten et al., 2017). The concepts underlying these experiences were pioneered in the virtual worlds of Second Life, a popular platform in the early 2000s, to construct immersive spaces for multiple users. Urban, Marty, and Twidale’s (2007) review identified over 150 museum-like artifacts on Second Life, attesting to the platform’s widespread use. These early experimental applications provided innovative ideas and insights into current VR museum practices, such as augmenting 2D artworks into 3D (Huang and Han, 2014), managing a vast amount of content (Sookhanaphibarn and Thawonmas, 2009), and exploiting the unlimited virtual space (Urban, Marty, and Twideale, 2007). However, the primary focus of many of these studies was on translating existing museum exhibits into virtual environments, which differs from the approach taken by RealityMedia. Furthermore, these studies prioritized delivering novel experiences and did not directly address the issue of how the space itself can add to the narrative.

RealityMedia draws on the structure of museums to thread multiple interconnected narratives by leveraging exhibition techniques in virtual spaces. This approach is advantageous because it allows visitors to apply their pre-existing knowledge and experiences while visiting the museum (Chronis, 2012). While the museum or gallery is primarily a physical structure, the relationship to a VR space such as RealityMedia is metaphorical. Media theorist Janet Murray’s concept of the digital medium as being inherently spatial is particularly relevant here, as VR and 3D dynamic and interactive computer graphics exemplify the ability of computers to embody spatial metaphors (Murray, 1997). As a narrative space, a museum conveys stories through a combination of text, images, artifacts, and sometimes video and audio. Therefore, the literature on media convergence and transmedia storytelling can provide valuable insight into the process of meaning-making in this context.

2.2 Media convergence and transmedia storytelling

Since at least the 2000s, Jenkins and many others have identified digital media convergence as a kaleidoscope of changing relationships and experiences across legacy and new media. One popular view has been that the digital medium is becoming a metamedium, absorbing most or all important earlier media (e.g., Manovich, 2013). But the digital medium is not all encompassing. It is true that the computers and other digital devices now serve as platforms for the presentations of earlier media such as books, television, film and recorded music. But as Jenkins, (2006).

Printed words did not kill spoken words. Cinema did not kill theater. Television did not kill radio. Each old medium was forced to coexist with the emerging media. That’s why convergence seems more plausible as a way of understanding the past several decades of media change than the old digital revolution paradigm had. Old media are not being displaced. Rather, their functions and status are shifted by the introduction of new technologies.(14).

Large audiences still consume film and books in the more or less traditional ways in addition to digital streaming in multiple formats. As these formats multiply, convergence might as well be called divergence.

There have been different views about how transmedia storytelling should function. Jenkins’ view was that the meaning making should take place across various media rather than simply within each medium.

A transmedia story unfolds across multiple media platforms, with each new text making a distinctive and valuable contribution to the whole. In the ideal form of transmedia storytelling, each medium does what it does best—so that a story might be introduced in a film, expanded through television, novels, and comics; its world might be explored through game play or experienced as an amusement park attraction. (95–96).

This requires a kind of active engagement of the kind evinced by a committed fan culture.

Transmedia storytelling refers to a new aesthetic that has emerged in response to media convergence—one that places new demands on consumers and depends on the active partici-pation of knowledge communities. Transmedia storytelling is the art of world making.” (20–21).

World making in Jenkins vision of transmedia can engage not only the producers, but the fans as well. A less radical vision is that the various media elements may be enjoyed independently. Ferrandiz speaks of a “calculated balance between stories that stand on their own in a single medium—that is, they are not impossible to understand for an occasional audience—and stories that are fragments of a more comprehensive whole capable of attracting a more involved transmedial audience” (Ferrandiz, 2014, 20). In practice, it seems often to be the case that there is a preferred medium, typically film or television. The original Matrix film and its sequels were generally enjoyed by large audiences who never went on to the other transmedial artifacts. This is also true of the endless Star Wars sequels and spinoffs and of the sprawling Marvel franchise as well. From an industry perspective, the point of repurposing content across media is simply to increase the brand value and revenue.

If we think of museums as sites of transmedial narrative, Jenkin’s view suggests that visitors should understand the exhibits and galleries as more than the sum of their parts, making new meanings out of their relationships. Science museums and history museums often weave individual exhibits or whole collections into a story. A good example is the National Center for Civil and Human Rights in Atlanta, Georgia, whose “exhibitions feature the papers and artifacts of Dr. Martin Luther King, Jr; the history of the civil rights movement in the United States; and stories from the struggle for human rights around the world today” (National Center for Civil and Human Rights, 2022). The conservative view would suggest that visitors focus on consuming the objects and their stories, one by one. We suspect that most visitors think of most museums that way, perhaps especially art museums that tend to categorize their objects by eras and countries or regions. An exception would be special exhibitions in art museums, where curators are clearly trying to convey an unified story to the visitors.

In this context transmedia theory prompts us to consider whether and how RealityMedia (its objects, exhibits, space) is unified as a narrative/discursive experience. Jenkins’s view would seem to suggest that an experience like RealityMedia should consist of a mutually reenforcing series of narratives at the level of the exhibits (through audio interpretations, text on the walls, and video) and at the level of the spatial layout and relationships of the galleries.

2.3 Remediation

Remediation as a theory begins with the observation that media forms are always interdependent and that therefore new media forms are never entirely new. In order to be intelligible as a medium of communication and expression, any new technique or technology must relate to earlier or existing forms in our media economy (Bolter and Grusin, 1999). Remediation operates across whole media: for example, broadcast television remediated radio for mid-20th century America by adding the visual dimension to the economic and cultural paradigm of radio. Many of the same networks that had dominated radio now become television networks, and many of the most popular radio programs were refashioned for television. Remediation also operates at the level of genres and individual media artifacts. For example, detective films remediated detective novels; detective television series remediated both films and novels. At the level of individual artifacts, a film such as the original Matrix borrows from and refashions the manga comic tradition, the dystopian sci-fi tradition in novels (and other films), the video game, and so on. This example shows that one media form or artifact can (and perhaps usually does) remediate several others. Such borrowings, homages, refashionings and improvements are characteristic of our complex media economy.

We can think of remediation as transmedial adaptation, in the sense that transmedia is the term for the general practice, and the theory of remediation describes the motives behind the practice from the perspective of the producers and the way the practice is received by the audience. For example, the makers of a video game such as the Last of Us (Naughty Dog, 2013) borrow techniques of storytelling, character development, and visual style from film in order to appropriate the sense of authenticity that we associate with (certain genres of) film. At the same time, video games that remediate film are making an implicit claim to improve upon the earlier medium by enabling the player to participate in and influence the progress of the story: the claim is that player interaction leads to a new and deeper sense of authenticity. The viewer or player of a new media artifact may not be aware of all the remediations that the makers had in mind and may even find different affiliations that were not part of the makers conscious intent.

This leads us to the question: what does our application, RealityMedia, remediate; what media or media forms does it recall or seek to emulate or improve upon? In addressing that question we can think of remediation in terms of the creators and then the consumers. As indicated above, we as the creators set out to remediate the information spaces as the digital remediation of both the museum and the printed book. In the process of making the digital spaces, however, it became increasingly obvious that the immersive space we were creating resembled above all an exhibition or museum. As we sought to introduce more interactive elements, we were aware that our immersive book has elements of the videogame. Indeed given the history of virtual reality, it would be almost impossible to ignore the relationship to the videogame. Videogames are still the most economically important and culturally influential applications of VR. But what about the users of our application? Do they understand the application in the same terms as we do? Specifically, do they experience it as a book, as a museum, as a game? Do they recognize elements of each of the prior media? This is one of the key questions that we address in our user study. In Section 3 we explain in greater detail how we as the creators view the remediations in RealityMedia; then in the user study described in Section 4 we examine whether users viewed the remediations in the same way.

3 RealityMedia: digital remediation of the museum

To explore the spatial narrative in VR, we designed and implemented a testbed called RealityMedia using a customized version of Mozilla Hubs, a WebXR platform. RealityMedia is a complement to the printed book Reality Media, providing information about the history of reality media and how augmented and virtual reality are taking their places in contemporary media culture. The scope of our study covers one chapter of the printed book “Presence” and the corresponding immersive spaces. In RealityMedia, the information spaces are the digital remediation of a museum. The application combines new media technologies with content from legacy media and explores new storytelling techniques to help engage the users in meaning-making.

Its information space is organized as a collection of galleries, many of which resemble the rooms of a conventional museum, although a few of the galleries recall other built structures (e.g., the amphitheater). These nested spaces or “spaces within spaces” relate to many game spaces which create a simple representation of a complex world (Schell, 2008). In addition, as opposed to the linear space of a printed book, we designed the spatial narrative on three distinct levels in the 3D space (with text, images, audio, and video) as follows.

3.1 Architecture level

The rotunda embodies the whole argument and suggests how the various galleries are connected in a circle of key ideas. In contrast to the way the book and website present the ideas, both hierarchical and generally linear, the immersive book gives the individual users considerable latitude to follow their own course throughout the space. There is no specific reading direction, although a sequential flow of writing space is suggested by the design of the space itself. As we have noted, most of the galleries look like museum galleries, which colors how the user perceives them. The layout of each is organized around themes—each room engages with one or a few related themes within the large theme of the gallery. Wall texts function as in museums to direct and also to provide elucidation, assisting the visitor in “reading the room” with an emphasis on certain themes and issues.

For example, an amphitheater in RealityMedia contains exhibits from the history of such media; its tiered structure suggests the increasing sophistication of the technologies employed to reproduce the real. The amphitheater structure itself dates back to the Renaissance memory theaters of Giulio Camilo and others (Figure 1).

FIGURE 1. Architecture level in RealityMedia. All the galleries are accessed from a central rotunda via teleporting portals (left). The amphitheater in ‘What are Reality Media’ room contains exhibits from the history of such media (right).

3.2 Gallery level

As indicated, each gallery presents a major theme or topic from the book: e.g., the history of reality media, presence and aura as media concepts, the concept of “total VR” (what is current called “the metaverse”), privacy and security in AR and VR, and so on. All the galleries are accessed from a central rotunda via teleporting portals, and the themes are proclaimed on the portals themselves. In addition, as the user enters each gallery, they can listen to an audio description of the theme. The relationships among ideas presented in each gallery are portrayed in the form of 3D force graphs (Figure 4), with which the user can interact. These elements are designed to prepare the user to read the various exhibits in light of the corresponding theme.

For example, our test gallery, Presence and Aura, consists of three rooms (Figures 2, 3). The first room shows that presence in VR has been a key concept for computer scientists since the 1990s, who have the concept in various ways, e.g., as transportation to a different visual world, as a feeling of immersion in a visual/auditory environment, as (photo)realism, and so on (Lombard and Ditton, 1997). The second room shows how presence manifests itself in VR experiments and applications, including a recreation of the well-known VR ‘Pit experiment’ conducted at the University of North Carolina in the 2000s (Meehan et al., 2002). Chapter 5 of the printed Reality Media goes on to argue that presence is related to important concepts in media studies: Walter Benjamin’s aura (Benjamin, 1935) as well as the distinction between place and space, first elaborated by the humanistic geographer Yi-Fu Tuan (1977). The third room then illustrates these concepts and their connection to presence through wall texts, video examples and a portal to a model of the Acropolis and Parthenon, a classic instance of the aura of ancient Greek art and architecture.

FIGURE 2. Gallery level in RealityMedia: from left: our test gallery, the Presence and Aura gallery; and the Pit experiment room.

FIGURE 3. The floor plan of the Presence and Aura Gallery.

3.3 Individual artifacts

To allow users to get inside, to inhabit AR and VR, we sought to design novel individual artifacts for RealityMedia (Figure 4), such as the 360-degree panoramic spheres or panoballs, which provide complete immersive experiences that are embedded in galleries but visited as independent world spaces. Presence in VR is usually conceived as forgetting that the medium is there. The idea is that if the user can be enticed into behaving as if they were not aware of all the complex technology, they feel presence. Following this, we designed the surrounding scene of each panoball to provide a sense of presence related to individual perception of the world (Lombard and Ditton, 1997). This includes transportation (a virtual trip to the surface of Mars), immersion (an immersive tour of Van Gogh’s Yellow House) and realism (a real 360° panorama view of Piedmont Park, Atlanta). In our VR world, the reader/user can find and walk into these panoballs and also listen to the audio description by stepping onto the audio pad. The 3D force graphs constitute an interactive visualization of the main ideas in each space, with each node representing one idea. Nodes with commonality are drawn to each other and color-coded to make it easier for the users to recognize them as being similar or different. The user can physically manipulate the graph by pulling out each thread and to see the connections more clearly. By clustering the data and interacting with the force graph, the user is not only seeing the visualization of ideas, but they are making sense of a narrative. Our portal is a good example of remediating the traditional hyperlink. It adopts the 3D spatial metaphor; instead of clicking on a line of text on the wall, users can walk directly into the portal to teleport from one place to another. Most of the exhibits, however, still take the form of text, images and videos arranged on the walls, and the arrangement will be immediately familiar to visitors from their prior experience of museums and galleries.

FIGURE 4. Interactive individual artifacts in RealityMedia: from left: 360-degree panoramic sphere, force graph, and a portal, all taken from the Presence and Aura Gallery (Figure 3 above).

Our expectation was that this three-tiered structure would make it relatively easy for the user to understand the place of each exhibit in the larger thematic Universe of our immersive book. In addition, there were at least two ways to access each gallery. We expected that most users would enter from the rotunda, but it was also possible to enter from a web page specially devoted to that gallery’s theme. Those who chose to read the web page first would enter the gallery with a further textual elaboration of the unifying theme. They might be more inclined to view the gallery itself as a remediation of the discursive narrative of the printed book, Reality Media, from which the web page texts were adapted.

4 User study

Our study focused on providing a deeper context for designing VR narrative by exploiting the immersive space as affording a new kind of book. We designed RealityMedia to experiment digital remediations of traditional narrative devices and the spatial, immersive, and interactive affordances of VR. We aimed to extend our knowledge on how various narrative formats can facilitate the process of meaning-making among users. As mentioned in the previous section, RealityMedia’s principal object of remediation is the museum and consists of three distinct levels of narrative threads: the architecture of the space itself, the collection, and the individual artifact. To this end, we opted for qualitative research methods to understand and evaluate user experience with the proposed system and its immersive forms of expressive ideas and narrative techniques.

4.1 Participant recruitment

We conducted user studies with 20 university students within the age range of 18–37 years in 2022. Our inclusion criteria included those without vision impairment, located in the US, and with prior experience using a VR device (Google Cardboard, Samsung GearVR, Oculus Quest, HTC VIVE, etc.). To gain more specific insights into RealityMedia and our research question, we sought to cover the broad spectrum of the target group. We recruited participants with varying levels of skills in VR, consisting of 9 (45%) beginners, 6 (30%) intermediate VR hobbyists and 5 (25%) advanced VR experts; 4 owning a VR headset. In addition, we included participants from a range of backgrounds and life contexts, including ethnicity, gender and the field of education. Among participants, 12 were Asian (60%), 6 were Non-Hispanic white (30%), and 2 were Hispanic (10%). Our effort towards recruiting a diverse participant pool was to open doors to understanding a mixture of goals, problems, ideas and user scenarios for the system.

4.2 Study design

Our study proceeds in two phases: the observations made during the VR experience using a task-based protocol and the semi-structured interviews with participants. This research also included user logging and video recordings on Meta Quest headset, which were used for later recollection. All user study sessions were conducted in a university laboratory. Over 2 months, 20 people were interviewed, lasting on average 1 h (30-min VR experiment and 30-min interview). Two researchers facilitated the experiment. While the moderator conducted 1:1 interviews and moved the discussion along, the experimenter led the VR experiment and took notes during the sessions. The study was approved by the University ethics committee.

At the beginning of the user study, the researcher provided a brief overview of the purpose of the study and introduced RealityMedia. Next, we gave instructions on using the Meta Quest 2 VR controllers and helped participants put on the VR headset. Participants could choose either to have a seated experience on a swivel chair or to stand up and walk around. Participants were asked to think aloud or verbalize their thoughts while completing the tasks. The tasks were designed for the participants to explore the information spaces of RealityMedia and interact with the novel narrative devices. The tasks include “ (user enters the Arrival Hall) You are inside the 3D space now. Step onto the podium that says ‘How to navigate in the gallery’ and listen to the audio description,” “Enter the portal to read more about aura,” and “You can click and drag to interact with the 3D force graph.” All VR sessions were cast to a screen for monitoring and recorded to capture the detail of the user’s interaction within the narrative spaces. We assisted those who struggled to complete the tasks in VR or those who asked for help with system errors or general controller issues. We encouraged the participants to communicate at any time to report on bugs, errors, questions, and concerns related to the study.

Following the VR experience, we held in-depth semi-structured interviews with participants to gain insights into the interaction experience and users’ expectations towards future systems. The interview consisted of these themes: 1) Overall evaluation of system usability and user experience, 2) User’s engagement with various forms of 3D content 3) User’s perception of using VR as a storytelling or discursive narrative medium. We asked questions to understand users’ mental models: “How comfortable do you feel using VR?,” “How frequently do you use VR?,” “Have you had any experience with Mozilla Hubs?,” “What was your first impression of RealityMedia?,” “How do you think using RealityMedia would benefit you?.” In addition, we asked questions to get more specific feedback on the prototype and identify design suggestions or ideas they may have: “Tell us what you thought about the overall experience with RealityMedia,” “What was the most memorable content within the 3D book? Why?,” “Was there anything surprising or unexpected about RealityMedia?,” “What three things would you fix in this immersive book? What changes would you make to help fix those problems?,” ‘What are your thoughts on using VR as a storytelling medium?.’

The resulting observation notes, interviews and research data were fully transcribed and anonymized by the researcher who conducted the session. Data was analyzed thematically (Braun and Clarke, 2006) to identify patterns of information using the Atlas. ti analysis software. Two of us independently read the transcripts numerous times to familiarize ourselves with the data before the coding process. The first of us derived the initial set of codes, which was audited by the other to confirm that the codes reflected the data accurately. The finalized collection of codes were grouped into categories and themes through multiple rounds of iteration. All authors discussed and refined the themes until a consensus was reached.

4.3 Findings

4.3.1 Users’ perception of remediations

In many ways, our findings resonate with the previous literature on remediation, underlining how the older media shapes a user’s perception of a new media form, in our context, the information spaces in VR (Bolter and Grusin, 1999). This includes the interplay of new and earlier technologies in shaping new media practices and the impact of prior experiences using these technologies to make sense of the practice in the new media (Jensen et al., 2018; Luik et al., 2019). For example, in explaining what helped them to understand the architecture, many participants referred to existing media paradigms and metaphors (e.g., browser tab, hypertext, games, museum, and Zoom conference call).

In addition, the participants spoke about various ways that RealityMedia can be perceived. The prototype was readily described not as a “VR book” but as remediations of traditional expressive media (e.g., the evolution of a printed book, videogames and the museum). For example, P1 described his experience with RealityMedia as “basically going around the museum, but it felt like playing a game,” referencing the characteristics and non-linear narratives of video games. In most cases, participants described RealityMedia as virtual showcases incorporated into the open museum space and discussed their expectations as museum visitors. Participants were curious to experience the VR technology in the context of reading and expected to enjoy something new and interesting compared to the traditional media. Others, however, described it as being “almost like a textbook” in the sense that it contains a linear quest-based story and covers a wide range of ideas and details.

I’ve played some video games, and it is like a different world. There’s a whole story in it and you’re like, “go to this world”, and you can’t go to the other worlds until you’ve completed the first one […] Some video games have more linear progressions to them, a very forced environment that you have to go through. Some other games are very open world. This one felt more open world to me. Because it was not forcing me through a story, I could kind of explore things as I wanted. (P3).

It did not feel like a book. Books, to me, do not feel like that. Books are more linear. I think the museum or the exhibit metaphor makes more sense. (P7).

Different remediations of expressive media were identified across three levels of narrative thread: individual artifact, gallery and architectural. As noted above, individual artifacts (e.g., texts, images, videos, etc.) were all familiar from earlier media. While at the gallery level, the Acropolis immersive model resembled for some users the “Assassin’s Creed Odyssey” video game where players can freely explore a large open world set in Greece. At the architectural level, the ability to navigate and interact with the environment was often described as a remediation of existing VR games such as “Half-life: Alyx”.

4.3.2 System evaluation and interactions

All participants actively engaged in the immersive environment and reported that RealityMedia was easy to use in terms of navigation and functions. For example, P2 stated that “I felt like I knew what was going on. I kind of had a sense of how the space was structured.” All the participants succeeded in carrying out the tasks, and reported no symptoms of nausea or motion sickness during and after the VR experience. The mean time the participants spent to complete the task in our system was approximately 27 min.

For many participants, RealityMedia was appealing precisely because it encompassed a variety of media forms, including 2D textual and visual information and 3D interactive content. Participants mentioned that by having access to rich forms of ideas, they were able to uncover a deeper understanding of the themes in the galleries by making connections with discursive and experiential forms of information.

Well, I have gone through virtual museums before, but I’ve never done this kind of thing. I think this is a very rich way to do it. Like there are stacks, the 3d spheres, and there’s videos, I’ve never done something to that extent - more than just walking and seeing. This one was a little bit more. There was some more information retrieval that was happening, which was nice. (P7).

Based on the results, the combination of rich interactivity and conventional experience of a museum space was identified as important in enhancing user experience in VR. Participants emphasized that VR served as an engaging medium for storytelling, seeing it as an opportunity to be transported to different story worlds and look at things from different angles. For instance, one participant mentioned that the transportation experience of walking into the panoballs (spheres with panoramic views on the inside) from the static room gave a feeling of immersion.

It is interesting that you’re not just stuck looking at the perspective. In other media, if you’re watching a video, the director of the video would determine what you’re looking at. But here, you get to decide what you’re looking at and look at it from different angles [.] you can look around and explore the space and then come back and start paying more attention if you want (P17).

Still, it became evident that there is a need to improve the design to blend the complex/multiple narrative strands into a coherent and seamless VR experience. Some participants underlined the importance of balancing “the serious” and “the fun” content in order to maintain the user’s interest in the virtual museum space. Furthermore, the participants were most intrigued when the narrative device was provided in the natural 3D spatial context, and they could physically manipulate the remediated artifacts (e.g., force graph and portal).

In addition, most participants described the 3D Pit experiment as their most memorable content in RealityMedia during the interview (Figure 5): “The thing that would stick with me, I would put the Pit experiment as [.] it makes my body more, more remembering of it (information).”(P15), “I was actually interacting with something inside that world instead of just reading and consuming information. I was taking part in moving, and my actions were influencing something in that environment. So it felt more memorable, because I could see the consequences of my actions, like moving the ball and dropping it.”(P13). The implication here is leveraging the spatial metaphors to weave together the 2D and 3D elements, elevating the user’s sense of embodiment in VR.

FIGURE 5. Images from the user study presenting the VR environment of the Pit experiment. A user interacting with the virtual ball to throw at a target on the ground of the pit.

Because everything or a lot of the things in VR, including most of this exhibit relies really heavily on spatial metaphor and feeling like you’re in space. Like you’re in this exhibit, you’re moving things around, you have that mobile of connections, where you can kind of see how things are connected by manipulating them. It is a challenge when that gets broken. (P2).

4.3.3 The new spatial narrative

The user study revealed that the spatial dimension of RealityMedia played a crucial role in shaping the way participants explored and comprehended the content. Several participants noted that RealityMedia relied extensively on spatial metaphors (P2, P4, P7). More specifically, for one participant (P4), the execution of physical movement within the virtual space was a key to a tangible understanding of the content, as opposed to simply reading from a traditional book. Also, P7 described RealityMedia as “feeling like walking … to get new information”, and added that the space itself scoped the breadth of content and guided their exploration of it.

In RealityMedia, we utilized several devices that leveraged virtual space to present or organize the content. First, the participants interacted with the 3D force graphs to see “how things are connected by manipulating them” (P2), creating their own interpretations. Likewise, the other two devices, portals and panoballs, facilitated a distinctive meaning-making experience by instantly altering the surrounding environment. The shift caused by these devices gives the participants a feeling of “teleportation” (P1) or “time travel.” (P12) Furthermore, it aided the participants in reconnecting with the concept they were exploring, as demonstrated by participant P5’s account of their experience of being teleported to Mars through the panoball on “Presence as realism”.

In the spheres, it did not feel as much real to me. But I think that was the point of it. Because that was at least in the spheres, from what I understood was to give you a feel for realism, or how it would look on Mars and things like that. I do not think the idea there was to feel like I was present on Mars. It was just to give me that experience. (P5).

During the transitions, the visual styles of the virtual spaces also influenced participants’ perceptions of the space. For instance, the stark contrast between the photorealistic and the illustrated textures marked a noticeable change in the environment (P19, P20).

The experience of spatial change was particularly strong when participants spawned to unexpected locations such as the center of the new environment after entering the portal or panoball. For P16, the spawn point even facilitated the new understanding as it created compelling visual effects; P16 stated, “I love how it just spawned me near the biggest pillar. And it just blew my mind that I was looking at VR structure, but like, it is really big.” However, the deviation from the unexpected location could cause disorientation and a disconnection from the narrative, as exemplified by one subject (P10) who shared their struggles when they spawned into an unexpected location, noting that it caused disorientation as they were “not going back to the exact same space.” Another subject (P2) expressed confusion about whether the spawn point would work just like “exiting the door” or transport them in the center of a new location.

However, this issue of disorientation caused by spawn points was mostly mitigated using the previews of new spaces which prepared users for transition, maintaining the narrative immersion. For instance, portals provided a visual cue to the new space through a preview of the title and location, as one participant (P3) stated “I kind of just knew while I saw the title, I saw what was above. I was like, this is where I want to go.” Similarly, P20 mentioned, “the process of walking through, and you can kind of see a peek into the next world and stepping through.” Similarly, the visual cues provided by the panoballs aided smooth transitions as they feel natural and “does not break the immersion.” (P13).

One interesting observation during the VR protocol was that participants often engaged in experimentation with their movements when they noticed a change in the environment, to attain a better understanding of it. For example, after transitioning to a new place, several subjects questioned if they could fly (P1) or if they could walk faster (P20), indicating a desire to examine the new environment and test the limits of the physics laws. This was notably apparent in the Pit Experiment, where more interactive elements were offered, leading to a more physical engagement with the space. For instance, in the Pit experiment, P15 spoke about their high expectations about physics in this gallery.

Overall, the user study has shown the impact of spatial metaphors on users’ understanding of the content presented in RealityMedia. However, it also highlighted the potential problems that can arise from using these metaphors, such as disorientation and disruption to the narrative immersion. We acknowledge both the potential and challenges of VR storytelling and will further explore these in detail in the discussion section.

5 Discussion

The principal finding in our study is that users did indeed understand RealityMedia as a remediation of the museum and that they found the experience satisfying in those terms. The finding is important because it promotes the museum or gallery as a plausible model for digital narrative. As noted in Section 2.1, the museum has a long history of presenting ideas through the disposition of objects and media in three-dimensional physical space. Users who have experience with museums or similar exhibitions, which is to say almost all users, will be able to transfer their understanding of how to read such spaces to virtual experiences such as RealityMedia. There are, in fact, many VR applications that use this model, in which the user is invited to walk through a virtual environment that looks like a gallery and contains exhibits; however, the narrative or discursive power of this paradigm has not been fully appreciated. In basing our VR experience on a book that is itself a complex discursive presentation of ideas, we sought to use the space itself to inform the narrative.

5.1 Authorial control and user agency

Our view is that virtual space itself can contribute to digital narrative in important ways and address traditional issues in digital narrative. Discussions of VR and narrative tend to focus on two aspects: immersion and interactivity. VR space provides a sense of immersion, which can add (powerfully) to the impact of the experience. But it is through the user’s interaction with objects and characters that the narrative itself comes to be. The space is literally the background. The user’s experience of the space is passive; their interactions are active. In RealityMedia, the architecture of the whole space and the layout of each gallery constitute a balance between a suggested flow of ideas and navigational freedom for the user. Referring back to the Presence and Aura gallery (Figure 3), we see that the three rooms suggest a progression from the definition of presence in room 1, to examples of presence in VR and applications in room 2, to the historical and media theoretical context in room 3. Within each room there are also some suggested paths (e.g., clockwise around room 1, but the user is always free to deviate from suggested orders or to ignore them altogether. The path of the user through the space is itself a narrative decision, which is partly constrained --the user cannot walk through a wall--but also partly open. In other words, the configuration of the space itself is constitutive of the digital narrative, and traversing the space is as much a part of the interaction as encountering any particular object that responds to user clicks or other interactive affordances. Appreciating that movement itself in VR environments is agency should be taken into account in any debate on the issue of authorial control and user agency.

In our study, participants reported experiencing high levels of user freedom and control and that having a sense of agency was very satisfying. While the participants acknowledged this as a strength, several asked for more detailed directions in exploring the virtual spaces and further guidance on engaging with the VR book. Some participants mentioned that RealityMedia lacks flow rules or an obvious order.

Having a recommended order to work through them would be helpful, because then somebody who has not read the book can be like, “Oh, okay, I kind of know how I’m going to experience this and what the trajectory is”, or somebody who has read the book can come in and be like, “oh, I want more info about this particular section. So I’m gonna go right there”. (P2).

Some participants reported that they walked up to what caught their eye first rather than following a map for the materials. Thus for these users, not having a specific order and sequence might be a little confusing if the story needs a specific order. These are, of course, classic issues in hypertext and hypermedia narrative. In the case of RealityMedia, however, the architecture of the space ensured that users would never be completely lost or could relatively easily recover their sense of place by returning to the rotunda. Also, some participants noted that the presentation and architecture layout facilitated genuine interactions and motivated serendipitous discovery.

In this context, we argue that VR can contribute to the tension between authorial control and user interaction, echoing an old debate on the author’s intentions versus reader interpretations (Murray, 2004). Because our genre (nonfiction book/collection) is not as strongly linear, the problem is less pronounced in contrast to fictional narratives. Yet, we note some challenges for future VR narrative: shaping the layout of a virtual story world, for it to be both meaningful to the users and the producers. How do the users’ experiences relate to the author’s intentions in VR narrative? How can we design VR experiences that are seamless and frictionless, helping users to feel autonomous and in control and make meaningful choices along the way? Part of the answer, we maintain, lies in appreciating that the space itself is a narrative element, and that spatial layout can itself accommodate the multiplicity of narrative or discursive threads. But we also acknowledge that different users have different capacities and preferences in spatial navigation in VR as in the physical world. Hence VR spaces need to be designed to facilitate users who prefer a clearer path through the content as well as those who prefer more freedom.

5.2 The opportunities and challenges of storytelling in VR

The process of designing RealityMedia and conducting user studies has underlined several opportunities for delivering non-fiction and potentially fiction narratives in VR. One such opportunity is to exploit the space to devise forms that are challenging or impossible to achieve through traditional media, thus creating a unique architecture-level narrative. Several prior works have demonstrated this potential, such as enabling users to visit the reconstructed historical sites (Khorloo et al., 2022) or building impossible spaces with overlapping virtual environments to facilitate natural locomotion (Suma et al., 2012; Fisher et al., 2017). In line with this, our study explored VR narrative devices to enhance the experience for users by enabling them to reimagine the space they are inhabiting. However, there is still much untapped potential there. The opportunities to manipulate space through a medium are certainly not new. At least since the 1910s, Hollywood cinema has depended on manipulating the viewer’s perception of space as a key device for visual narrative. Film directors routinely employ camera angles and shot compositions to control not only the audience’s perception of filmic space but their understanding of the narrative itself (Bordwell et al., 1985, p.1–84). However, in VR, users view and move through the space in a more personalized way, which in turn gives designers more freedom to use the structure and layout of the virtual space to unfold the narrative. Again, we do not argue that our gallery designs are the only or even the most expressive structures possible. They are only a beginning.

Another opportunity for storytelling in VR comes from embracing a wide range of remediation strategies by incorporating various media forms and genres. While VR narrative has primarily focused on remediating entertainment media such as film, television, and videogames, we have found that adopting more diverse media types can greatly enhance the overall narrative experience. By integrating a mixture of mediums, such as books, physical museums, and websites, each medium can add a unique layer to the narrative, rendering the overall experience of RealityMedia more diverse and multidimensional. The challenge here is in selecting the appropriate remediation strategies and organizing them effectively to achieve the narrative goals. This is critical because, as seen in section 4.3.1, the medium or mediums that users identify significantly shape their experience and understanding of the narrative, as they draw from their existing knowledge of it to consume the new experience.

While virtual reality creates a new design space for narratives, it also poses new challenges. First, from the study, we observed user disorientation which resulted from the deviation from the users’ expectations of where they would spawn in the virtual space. It aligns with the findings that abruptly transporting users to a different location was detrimental to wayfinding (Bowman et al., 1997) and imposed disorientation (Bowman et al., 1999). To minimize this issue, strategies such as having clear landmarks (Jansen-Osmann, 2002) and maps (Darken and Sibert, 1996) can be adopted especially when teleporting the user to a new scene.

Another challenge comes, ironically, from renewed popular interest in VR as “the metaverse” after the first wave of hype for VR in the 1990s. The popular assumption of the metaverse is that VR will function as a computing medium for a digital second life (Xu et al., 2022), so that everything, including legacy media content, must be delivered in a 3D format. Our project aligns with this expectation as we have focused on presenting the contents of a physical book in a VR form. However, we also propose that the metaverse should not be limited to designing everything in 3D, but rather it should find a balance between the legacy and the emerging media. In RealityMedia, we have preserved traditional media formats, such as discursive content, in many parts of the prototype. As demonstrated in section 4.3.2, participants appreciated having access to both traditional and immersive forms of information as it allowed them to acquire a more comprehensive understanding of the concepts. Future research should focus on finding the optimal balance between 3D and traditional media to ensure that the overall experience remains seamless and engaging for audiences.

6 Conclusion

This work addresses the opportunities and challenges of using VR as a new writing space for non-fiction narratives, building upon the longstanding practice of inscription. For this, we have developed a test platform RealityMedia using the museum as a model to spatially embody the narratives on three distinct levels, architectural, gallery, and individual exhibit.

The findings from our study reveal that the participants successfully engaged with narrative threads across all three levels. Furthermore, we uncovered the diverse ways in which RealityMedia was perceived and how it transformed the subjects’ perceptions of space, resulting in a unique understanding of the material. In discussing the results, the study may shed new light on the debate between authorial control and user agency in digital media, by suggesting that navigation in VR environments can be seen as a form of agency and spatial configuration as an authorial control. This study is also in line with the renewed interest surrounding the metaverse, providing an early exploration of leveraging VR for non-fictional narratives, which demonstrated the promise of VR in this field and even for fictional narratives.

We acknowledge a limitation in this research that could guide the directions for future research. The present study did not substantiate whether the subjects fully assimilated the concepts presented in RealityMedia. Future work could conduct pre- and post-tests to inspect the knowledge acquisition. Another avenue of future work is to examine the potential of VR beyond the current museum model and of our narrative devices for fictional narratives. Lastly, we have not examined the social VR experience in RealityMedia. While running the RealityMedia on the webXR platform had its limitations, such as in the number of high-quality image textures and models, it also presented an opportunity, which is an easy integration of a multi-user experience. In the future, we aim to explore the potential of combining social VR experiences with RealityMedia where multiple users navigate the VR space to engage in meaning-making together.

Data availability statement

The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by the Georgia Institute of Technology Institutional Review Board (IRB). The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author contributions

The authors confirm their contribution to the paper as follows: study conception and design: SJ, JP, ME, BM, and JB; data collection: SJ and JP; analysis and interpretation of results: SJ, JP, and JB; draft manuscript preparation: SJ, JP, ME, BM, and JB. All authors contributed to the article and approved the submitted version.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References