Keywords

1 Introduction

Evolving technologies such as smart cities, humanoid robots, domestic Internet of Things (IoT) and autonomous vehicles are creating a vision of near future environments, i.e. Physical-Cyber Environments [1]. These range from digitally augmented objects (tangible user interfaces) to interactive virtual spaces (MS Hololens) to the Internet of Things vision and Physical-Cyber Environments (P-CE). These emergent technologies together form a paradigm of future interactions where the digital and physical properties of objects are mixed together and merge fictional scenarios with working prototypes.

For instance, conducting user-based research into autonomous vehicles, a product that is yet to properly exist in an accepted day-to-day form, is difficult as it involves researching a ‘future’ product, which is yet to fully come into being [2]. Controlled experiments and classic research methods in Human-Computer Interaction are considered to have limitations in their application to P-CE suggesting that newer methods for researching user interactions and future technologies are needed [3].

The design of near future products and environments must involve the collaboration and expertise of various disciplines and stakeholders, such as: artists, software engineers, interaction and product designers, business managers and policy makers. Each of them brings their own internalised assumptions and thought processes, making understanding and discussion between the various parties potentially problematic. Tools and methods are needed to aid productive dialogue between those involved in such processes. For instance, Shedroff and Noessel [4] suggest that lessons can be learned from the interfaces written about in science fiction and employed in the development of real world interfaces. They speak of a two-way influence on design, one relating to real world design, influencing hybrid science fiction interfaces and the other based in science fiction influencing real-world interface design by inspiration, expectation, social context and the innovation of new paradigms. Science Fiction Prototyping (SFP) is a method that allows engineers, designers or futurists to think about the technologies they are developing from a human perspective, linking the imaginations of product developers and teams to future users and usage. SFPs are generally short literary works of fictions, which are grounded in scientific facts. The purpose of these stories is to start conversations about the implications, effects or ramifications that technology may have on the future [5].

However, the literature on SFP highlights a number of shortcomings in the use and application of this methodology. In its current format, SFP does not fully support the ideation process with respect to the development of concepts. In part this is due to the individual, exhaustive nature of creating a written SFP. Furthermore, the results are quite limited reducing to a classic wall full of colored sticky notes, few storyboards, a story world and a fancy video at most. Complex environments, such as C-PE need prototypes to be properly explored and render a vision of the future technology that is tangible and accessible to those participating in such design discussions. Low-fidelity prototyping (e.g. paper prototypes, storyboards and narratives) is just a first step towards developing a system and achieving working prototypes can allow users to inspect new forms of interactions; recently, Schmidt [6] noted in a critique to extensive use of low fidelity prototyping to test user experience (sketching, storyboarding, videos, etc.) that «making functional prototypes is a source of inspiration, understanding and reflection».

This work aims to build upon the idea of SFP, to introduce the use of proto-tools – multi-purpose digitally augmented artefacts with no constraints and seemingly limitless uses (e.g. digitally augmented surfaces and objects) – that can aid the creation of future fictional scenarios and corresponding high-fidelity prototypes for exploring the design of interactions in Physical-Cyber Environments.

2 Challenges

A growing challenge to PC-E design practice is the need to bridge the communication gap between various professions, designers, other stakeholders and end user groups involved in the design process. Even within sub departments of organisations people have “unique perspectives” of aims and tasks causing conflict. Another aspect is time to be effective and productive. It can take years to be effectively submerged in a new culture. For example, Myra Strober [7] discovered that at Bio X, an interdisciplinary science centre at Stanford, it took two years of weekly meetings to learn the culture and habits of mind of each other’s disciplines. Therefore, time effective solutions need to be explored to begin to address some aspects of developing productive communication between various and diverse collaborators.

For instance, the use of fictional scenario-based design and rapid prototyping methodologies might break the mental image of future scenarios, biased by participants’ background. The use of a generative session, or workshop, allows for the understanding of latent and tacit knowledge and while bounded in time by sessions, a fictional design workshop could be complemented by digitally augmented prototypes instead of low-fidelity prototypes (e.g. storyboards). Digitally augmented prototypes can materialise ideas to stimulate participants reflection in action and at the same time create a common language to talk about problems and solutions [8].

The main challenge then shifts to introduce digitally augmented technologies in design fiction workshops that can be used by participants to prototype solutions without requiring any specific technical background, i.e.: designers, stakeholders and end user groups.

3 The Quest for Digitally Augmented Artefacts

Ethnography suggests that collaboration can be enabled by shared representation, but these externalised representations add to cognitive processing [9]. Externalisation of individual’s thoughts and ideas via representation in artefacts can aid communication of those thoughts and ideas. The question then becomes: how to inspect others’ thoughts or ideas in an effective and productive way to inform the design of future cyber-physical environments.

Our hypothesis is that digitally augmented artefacts spontaneously created by people to support their own vision of such cyber-physical environments can offer a window into their cognitive and creative process. Our hypothesis is consistent with research in distributed cognition and the use of artefacts to externalise cognitive models [10, 11]. Furthermore, the theory of embodied interaction considers the materiality of tools as one of the most critical cognitive resource for human activity [12, 13]. Considering cognitive artefacts as a glimpse on participants’ inner design mechanisms has some limitations studied by [14]: especially in terms of artefacts being too bounded to the context or scenario proposed in the design session (context bias). We propose to address such limitation by stimulating rapid prototyping of Cyber-Physical environments through the use of proto-tools allowing the articulation of artefact at “basic level” of generality following Rosch’s contribution in prototype theory [15] and Schon’s reflection in action [8]. The aim is to use proto-tools to inspect design brainstorming outcomes and reflect on them.

In our vision a proto-tool is a rapid protyping environment used to assign behavior to smart artefacts, i.e. smart objects that offer features like connectivity, sensors, actuators, and embedded software [16]; by exploiting the dual nature of smart objects (physical and digital) and leveraging on human’s natural ability of interpreting and manipulating objects in the real world, proto-tools aid participants to render fictional scenarios into functional prototypes to be used as a source of inspiration, understanding and reflection [6]. The rapid prototyping environments are used to assign specific meaning and behaviour to smart objects without requiring programming expertise but instead providing a technology powerful enough to render a working prototype of the fictional scenarios.

4 Prototyping Cyber-Physical Environments with Proto-Tools

From autonomous vehicles and Internet of Things, to futuristic augmented experiences in museums Physical-Cyber Environments (P-CE) are emerging as a relevant paradigm in future interactions where the digital and physical properties of smart artefacts are mixed together to render fictional scenarios into working prototypes.

For example, in domains like Cultural Heritage (CH), smart artefacts can be installed in museums, archaeological parks and exhibitions to create smart visit experiences, i.e. scenarios where visitors acquire CH content by interacting with the surrounding environment and smart artefacts included in it. Nevertheless, such interactions among people and smart artefacts in an augmented environment are technically challenging for museum curators and other stakeholders involved in design visits. In Desolda et al. [31] researchers introduced a tool called EFESTO-5W for simplifying the creation of Event-Condition-Action (ECA) rules combining smart object events/action (see Fig. 1). Some studies have shown that the EFESTO-5W composition paradigm effectively guides curators and non-technical users in establishing the behavior of multiple smart objects.

Fig. 1.
figure 1

An example of service attributes assigned to a smartphone for future use by visitors as planned by curators without involving any programming.

Full size image

In [32] a tangible user interface block-oriented programmable object (TUIBOPO) framework was introduced as an extension of the TUIO protocol providing further interaction capabilities for multi-device environments. The TUIO protocol, as Kaltenbrunner et al. (2015) stated, “is an attempt to provide a general and versatile communication interface between tangible tabletop controller interfaces and underlying application layers. It was designed to meet the needs of tabletop interactive multi-touch surfaces, where the user is able to manipulate a set of objects and draw gestures onto the table surface with the fingertips.”

TUIBOPO can be considered a framework for implementing block-oriented programmable objects that simplifies the implementation of such objects with physical and digital properties (Fig. 2).

Fig. 2.
figure 2

Block-oriented programming with tangibles. Users are assigning digital properties and functionalities to a smartphone assembling blocks instead of by programming.

Full size image

Those are just few examples of the emerging platforms providing capabilities that are compliant with our definition of proto-tools and that can be exploited to design and implement high-fidelity prototypes in fictional design scenarios.

5 From Physical-Cyber Environments Design Fiction to Working Prototypes

The world that we inhabit is full of tangible objects, objects of different weights, shapes, textures, colours, temperatures and even tastes and scents; some are stationary while others are portable. We pick up books, play musical instruments, wear clothes, sit on furniture and engage with all manner of artefacts in our day-to-day existence. With the emergence of smart materials and the IoT [16], we are having to re-think how we engage with a range of smart, interconnected objects, both day-to-day and specialist devices. This reconsideration of purpose is reflected in the emergence of a range of approaches to the design of TUIs.

Developing and optimising tangible interfaces, with their mix of physical active elements combined with variably configured displays, presents significant challenges compared to design processes targeted at traditional, screen-centric keyboard/mouse driven UI systems. Sketching, perhaps the most widely used design tool is obviously used to efficiently communicate concepts to other members of a design team but also, and perhaps more importantly in terms of the current study, is a central methodology in design research. In this role, it can be considered as an early-stage, minimal fidelity, prototyping methodology. “… sketching is thinking. … Its purpose is to test and verify a concept through a communicable representation”. [17] This design-research aspect of sketching is problematic in a hybrid digital/physical domain since the elements that make up tangible interface systems tend to require configuration and linking in application-specific ways that mitigate against the flexibility and low investment investigations typical in more conventional, paper-based sketching. Blackwell et al. [18] explicitly follow the sketching paradigm in their ‘Tangible-Prototyping Workshop’ methodology, but extend the range of sketching resources to include familiar 3D materials such as foam boards, modelling clay, pipe cleaners, string etc. Their aim is to be able to rapidly produce tangible ‘sketches’ that that provide maximum opportunity for creative exploration and evocative experiences and which support a range of analyses of the resultant ‘solid diagram’. Their approach is severely limited in terms of the functionality of the prototypes produced, but a number of other projects such as that of Nam (2005) [19] and ‘Sketch-a TUI’ [20] have sought to address this problem through approaches based on ‘interactive sketching’, extensions or augmentations of traditional sketching which seek to retain the spontaneity of drawing while using AR and sensing to implement some of the functionality of reactive objects.

In contrast to the underlying sketch paradigm of these projects, where there are (at least in principle) very few preconceptions about the ontology of the tangible artefact being designed, many projects have attempted to facilitate rapid TUI prototype development by providing collections of selected functional elements which may then be assembled by the design team to construct a prototype. These frameworks or toolkits typically consist of (or enable) a number of physical elements (‘iStuff’ [21], ‘Makey Makey’ [22]) or small active networked devices designed to add functionality to physical objects (‘Smart-Its’ [23], ‘CookieFlavour’ [24], ‘Amerino’ [25], ‘Kniwwelino [26]) together with a software architecture which handles configuration and intra-object communication in a way that does not require specialist coding skills. An alternative, but related approach seeks to use AR to resolve the difficulties of configuring and connecting the physical elements of a TUI by providing a prototyping framework which is partly [27] or wholly [28] virtual. A third approach, perhaps acknowledging the ludic nature of parts of early-stage design processes as well as the openness and flexibility required of systems designed to stimulate the imagination, uses frameworks such as Lego [29] originally intended for play. While it might be argued that this necessarily reduces the power of the prototyping process, turning it from an exploratory, unbounded process to one where creativity becomes at least to some extent a matter of selection, in practice provided a sufficiently large number of varied elements are included in the toolkit or they are sufficiently generic and open in their potential configurations, results can be useful at least in terms of exploring the materiality of tangible interface elements.

However, such toolkits are at best a collection of casings and interface mechanisms such as buttons, dials, sliders and so on which typically have low fidelity in terms of the physical materiality and haptic identity of tangible interface elements. As the ‘TUIkit’ project [30] has suggested, evaluating and optimizing material and physical qualities is a hugely important part of the TUI design process, yet, “in the design process of tangible interfaces, little attention has gone to the meaning of material qualities for TUIs, and materials are often chosen more for their availability or technical qualities, rather than for their benefits to the user” and then only comparatively late in the design cycle. So how do we develop design tools to enable us to create high-level cyber-physical systems, that require a tight linkage between the physical and the digitally functional? Developing such tools is a design problem in its own right. In order to think about some challenges that we face, we offer a couple of scenarios, conceptual designs that can aid us in our design-thinking. In our first scenario first we propose a system that is a simple tangible device that we call the Palm Stone (PS). The PS is a small stone like object that sits in the palm of the hand, the user carries around with them and it responds, in different ways to certain activities in the environment. For example our PS gets warm when our son returns home from school, it vibrates gently when our friends are playing an online game that we play together, it vibrates with a certain rhythm for a diary date and it has a range of preferences that can be set to trigger it when certain events, or environmental factors are sensed. Theoretically it appears as though this is a simple case of ‘if this then that’, but practically making a small prototype of such a system would need us to program the actuators, devise vibration systems, add heating elements, networking protocols would need to be dealt with and the physical form would need to be made. It could be done with a low power Arduino system, soldered together and cased, but imagine if there was a simple system that allowed us to add a series of triggers to a ready made system that would then allow us to quickly prototype and take out the system into ‘the wild’ [33] gather feedback and refine the system. This would be a much more useful proposition for a range of possible users; from designers to people trying to understand consumer behaviour.

Now imagine if we were able to take such technologies and employ them in domains and places such car interfaces where different people might want different physical interface properties, or want different degrees of haptic feedback, want the interface to sense pollution, or react to voices and so on. Currently having to develop something on this scale would need a whole range of programming, engineering and 3D design skills. Developing proto-tools to help support the development of cyber-physical systems would be a huge leap forwards as a way to quickly create tangible prototypes even about near future technologies.

6 Conclusion

Cyber-Physical environments are becoming a relevant area of research considering the evolution of technology and near-future visions of digitally and physically augmented objects, which can offer unbounded possibilities especially in terms of impact and interaction with the society,

In order to design for Cyber-Physical environments and therefore smart objects operating in such scenarios we need to involve a wide range of stakeholders, from engineers and social scientists to citizens and thus we need methods and tools that will allow participation from different group of backgrounds.

Those methods, for instance fictional design scenarios, normally involve low level prototyping in form of storyboards or videos but we think that currently there are more possibility to refine those prototypes and produce higher level artefacts.

We introduce the concept of proto-tools which are an emergent type of environments that can be used to rapid prototype smart objects in Cyber-Physical environments with little or no technical background required. Those proto-tools will provide a way to overcome the limitation of low fidelity prototypes capabilities when rendering fictional scenario environments. For instance, we can imagine a multi-disciplinary group of participants in the automotive sector, like: engineers, policy makers, drivers being able to prototype a dashboard of an autonomous car and render it, instead of with a bunch of sketches and storyboards, with a mixed reality application programmed via a proto-tool where a physical box representing the dashboard frame could be extended with augmented reality projection of controls made interactive by haptic controllers. Such scenario can then be explored thoroughly revealing user experience issues and opportunities at a more complex scale overcoming the limitation of a low fidelity prototype.