Stereovision and augmented reality for closed-loop control of grasping in hand prostheses

Figure 1 summarizes the operation of the system and shows how the tasks and responsibilities are shared among different functional blocks. The user wears AR glasses with embedded stereo cameras. He/she triggers the operation of the semi-autonomous controller via a simple myoelectric interface. The semi-autonomous controller selects grasp type and aperture size appropriate for the target object by processing the visual information retrieved by the stereo cameras. Accordingly, the controller sends a preshaping command to the hand. The selected grasp type and hand aperture size are visually fed back to the user through the AR glasses (AR objects). The user can employ the AR feedback to adjust the automatic decisions through the myoelectric interface. Eventually the controller implements hand closing to grasp and hand opening to release the object, in response to user myoelectric commands. The presented control is best described as semi-automatic since the user triggers the automatic operation but can also override/correct all of the decisions of the autonomous controller.

Zoom In Zoom Out Reset image size

The system comprises the following components: (1) a standard laptop computer (8GB RAM, i5@2.73 GHz) equipped with a USB data acquisition card (DAQ 6210, National Instruments, US), (2) AR glasses (iWear920AR, Vuzix, UK), (3) a two-channel analogue myoelectric (EMG) amplifier (anEMG2, OTBioelettronica, IT) and (4) a robotic hand prosthesis prototype (IH2 Azzurra, Prensilia, IT).

The robotic hand was a left-handed, commercial version of the SmartHand [6]. It consists of four fingers and a thumb actuated by five electrical motors. Allowed motions are flexion/extension of the thumb, index, middle and ring–little finger as a couple and the rotation of the thumb opposition space. The hand includes encoders on each motor and an electronic controller that implements position control, by receiving commands sent over a serial bus from the laptop. The hand is mounted on a custom splint made from thermoplastic material and strapped firmly using Velcro straps and elastic bands to the subjects' forearm, so that the robotic hand is positioned directly beneath and parallel to the subjects' hand. Four different grasp types were implemented: palmar, lateral, bidigit pinch and tridigit pinch grasp.

Each of the two channels of the EMG amplifier includes a second-order band pass Butterworth filter (10 and 500 Hz). The amplifier gain is adjusted individually for each subject to obtain good signal-to-noise ratios (visual inspection). Two pairs of standard Ag/AgCl EMG electrodes (Neuroline 720, AMBU, USA) are placed over the left-hand flexor and extensor muscles, with a reference electrode on the wrist. The amplifier outputs are acquired by a data acquisition card (1 kHz sampling rate) connected to the laptop.

AR glasses are worn by the subjects. The glasses include stereo cameras embedded into the glass frames (resolution: 640 × 480 pixels, refresh rate: 30 Hz). The cameras are therefore directed towards the scene the user is looking at. The video stream acquired from each camera is sent to the laptop via a USB port. The image frames are processed and then projected via a VGA port at the two panels embedded in the glass frames. The panels implement a stereoscopic display and therefore allow the user to see a single 3D image of the scene in front of him/her (i.e., as if he/she is looking without the glasses). The scene contains both real and virtual objects (AR feedback).

The control flow is implemented as a finite state machine in which transitions between the states are triggered by a simple two-channel myocontrol interface. A high-level representation of the state machine is shown in figure 2(a) and illustrated using snapshots in figure 2(b) and a detailed algorithm is given in table A1 in the appendix.

Zoom In Zoom Out Reset image size

The user could start operating the system by directing his/her sight and thus the glasses to the target object (figure 2(b), panel 1). After recognizing the target object, the system would acknowledge to the user the recognition of the target in the form of an overlay covering the object (object selection AR feedback) (figure 2(b), panel 2) and the user could then issue the command for opening the hand. This action would trigger the semi-automatic control and the system would analyze the scene (computer vision) to determine the geometrical model and properties (size and shape) of the target object. Based on this analysis, the system would estimate the grasp type and aperture size appropriate for grasping the object and send a preshape command to the hand (figure 2(b), panel 3). The component for stereovision analysis and subsequent grasp type and size selection based on the artificial vision is named stereovision grasp decoder. The AR feedback about the automatically selected grasp type and aperture size would then be presented to the user (figure 2(b), panel 4). The grasp-type AR feedback is presented as a visual icon, i.e., an image in the upper right corner of the subject's field of view. The aperture-size AR feedback is presented in the form of a virtual box placed just next to the target object, with one side parallel to the support plane and the other side aligned with the front side of the object. The size of this AR box is proportional to the current hand aperture size as read from the hand sensors. The user could therefore assess whether the hand was appropriately sized to grasp the target object by simply comparing the size of the virtual box to the size of the actual target object. The user could exploit this feedback to adjust the aperture (e.g., aperture too small/large) or to restart the decision process from the beginning (e.g., wrong grasp type) by issuing specific EMG commands. If the user judged the grasp type and aperture as correct, then he/she could bring the hand into the object vicinity and issue the EMG command to close it (figure 2(b), panels 5 and 6), manipulate the object as wished and finally reopen the hand in order to release the object (and restart). The described control system was running online and the average time needed to respond to a preshape command was less than 1 s (using the equipment described in the previous paragraph). This time includes command detection, stereovision processing and sending of the preshape commands to the hand.The online control software running on the laptop was implemented in Matlab 2012 and organized as a set of individual modules shown in figure 3 and described in the following. For more detailed information, see the supplementary data document attached to this paper (available from stacks.iop.org/JNE/11/046001/mmedia).

Zoom In Zoom Out Reset image size

Myoelectric control module. This module implements the online acquisition and processing of the EMG signals. The inputs for the processing are the EMG signals picked-up from the flexor and extensor muscles of the hand. The output is a 4-bit code indicating four possible user-generated commands (flexion/extension × high/low). Four commands are generated by using two-level thresholding of the RMS calculated over a 200 ms window (with 50% overlap) of raw EMG, and 'the first signal wins' strategy was used to resolve the eventual co-activation. This 4-bit code is fed into the state machine (figure 3) to trigger the state transitions and into the prosthesis control module to adjust the grasp properties (type and aperture size) (figure 3). Figure 4 demonstrates how myoelectric processing was used to command the system throughout the whole operation sequence (figure 2).

Zoom In Zoom Out Reset image size

Computer vision module. This module is responsible for the bidirectional communication with the AR glasses, i.e., acquisition of the images from the stereo cameras and control of the stereoscopic displays, and also for the processing of the image data. The inputs for processing are the two images from the stereo camera pair, and the current preshape of the hand (grasp type and aperture size) supplied by the preshape control module. The first input is used to estimate the properties of the target object (control) and the second to construct the AR feedback object (sensory feedback). The module implements a state-of-the-art computer vision processing pipeline comprising the following steps (figure 5): (1) depth estimation (figures 5(a) and (b)), (2) object segmentation, (3) 3D point cloud generation, (4) fitting of a geometrical model (box, cylinder, sphere and line) through the point cloud (figure 5(c)), and (5) construction of the virtual objects (AR feedback) and their projection into the real scene (figures 5(c) and (d)). The outputs from the module are the target object properties (shape and size) and stereo images with the embedded AR feedback. The size of the object was given in centimeters. The object properties are input for the preshape control module, whereas the images are sent to the stereoscopic panels of the AR glasses to be presented to the user. Depth estimation is implemented using an efficient large-scale stereo matching (ELAS) algorithm, as described in [47], which is applied on the rectified grayscale image pair. The cloud point is computed from the disparities using triangulation in combination with the extrinsic and intrinsic properties of the camera system, which are identified during calibration [48]. The RANSAC algorithm [49] with application specific modifications is used for point cloud processing and object modeling.

Zoom In Zoom Out Reset image size

Preshape control module. The inputs for this module are the object properties estimated by the computer vision module and the current state of the hand preshape provided by the prosthesis control module, whereas the outputs are the automatically selected grasp type and size. The grasp type was selected using cognitive-like processing represented by a set of IF-THEN rules similar to the ones described in [50] and [35], while the grasp size was a continuous variable calculated as the object size estimate plus heuristically adopted extra margin of 0.5 cm. In essence, the rule base implemented a mapping from the object properties into an appropriate grasp type (i.e., IF OBJECT SHAPE is X and OBJECT SIZE larger/smaller than Y THEN use GRASP TYPE Z). Palmar grasp was used for wide cylindrical and spherical objects, lateral for thin box objects, tridigit and bidigit for smaller objects etc. The computer vision module together with the preshape control module constitutes the stereovision grasp decoder.

Prosthesis control module. This module is responsible for the low level communication between the system and the hand prosthesis (i.e., sending commands and receiving sensor data over the serial bus with a sample rate of 20 Hz). The inputs are desired grasp type and aperture size (automatic control) and the user commands for the correction of the hand aperture (manual control) and the output is the current preshape state determined by reading the prosthesis sensors. The prosthesis actuators are position driven. The position commands that should be sent to the hand in order to implement a grasp of certain type and size are specified using lookup tables. The latter are prepared beforehand by measuring the response of the hand to position commands covering the available range of motion.

The control system was tested by able-bodied subjects performing a simple grasping task. Thirteen healthy subjects (29 ± 4 yr) volunteered in the experiments after signing the informed consent form approved by the local ethics committee. The subjects were comfortably seated on an adjustable chair in front of a table where a workspace was organized (see figure 6). Three positions were marked on the workspace: the initial (rest) position for the hand, the initial position for the target object and a position to which the objects had to be transported after being grasped. The workspace was organized and the system was mounted so that the robot hand was outside the user's field of view while he/she was looking at the target object.

Zoom In Zoom Out Reset image size

At the beginning of the session, the subject was given instructions about the system operation and experimental protocol and then he/she was allowed to briefly familiarize with the system (less than 10 min). Twenty objects of different sizes and shapes (see table 1) were used as the targets and were presented to the subjects in a randomized order. Subjects were asked to reach, grasp, transport and release each of the target objects by operating the artificial hand using the semi-autonomous control loop explained in the previous paragraphs. The subjects were instructed to place the hand back in the initial position after the completion of each trial. Subjects operated the hand within four different experimental conditions with 5–10 min rest between the conditions.

• (1)  
  Automatic control with AR feedback (AUTO-AR). In this condition, the subjects could not correct the decisions made by the system/stereovision grasp decoder (refer to table A1 in the appendix). Two series of 20 trials were performed. This condition was used to assess the baseline performance when the control was fully automatic. The AR feedback did not play a useful role in this condition but was displayed to the subjects so they could familiarize with it.
• (2)  
  Semi-automatic control with AR feedback (SEMI-AR). In this condition, the system operated according to the full control sequence (without restrictions). Therefore, in addition to triggering the automatic control the user was able to correct the system decisions (grasp type and size) by relying on the AR feedback (refer to table A1 in the appendix). The subjects could decide if and how much to correct. They were instructed neither to look at the hand nor to move it from the initial position until the corrections were completed. This meant that all the adjustments had to be accomplished by relying exclusively on the AR feedback. The test included one series of 20 trials and the goal was to compare the performance of semi-automatic control with respect to the fully automatic control (AUTO-AR).
• (3)  
  Semi-automatic control with AR feedback and random errors (SEMI-AR-RE). The control was the same as in the SEMI-AR condition but the computer vision module was programmed to introduce random errors (± 2–5 cm) in the estimated hand aperture size. This error was added only after the grasp type was determined. As a result the estimated grasp type was (mostly) correct but the hand aperture was surely wrong forcing the subject to readjust the hand based on the visual information provided by the AR feedback as in the SEMI-AR condition (i.e., without looking at the hand). Again, it was up to the subject to decide when and how much to correct. This scenario included two series of 20 trials and the goal was to test the general perception and usefulness of the AR feedback by forcing the subjects to rely on it while finely adjusting the aperture size in practically every trial.
• (4)  
  Semi-automatic control with random errors and without the AR feedback (SEMI-VIS-RE). The system operated identically as in the SEMI-AR-RE condition but with the AR feedback turned off. The subjects now had to bring the hand close to the object and correct the aperture by directly comparing the hand to the object. This condition was tested in one series of trials and was used as the control condition for the SEMI-AR-RE.

The correct automatic control of the prosthesis and the task accomplishment were correlated but not in an absolute sense. Indeed, the task could be accomplished although a wrong grasp and/or size were selected (e.g., lateral grasp to pick up a bottle or hand fully opened to grasp a small object). To take this into account, we adopted two sets of outcome measures (see table 2), the first to evaluate the performance of the automatic or semi-automatic control of grasping (GC1–4) and the second to assess the actual accomplishment of the task (TA1–2). Additionally, in order to further quantify the stereovision grasp decoder performance and compare different operating modes, we have also measured the time from the moment when the first hand preshaping command was issued till the moment when the first closing command occurred, which is in the further text referred to as the time needed for the task accomplishment (TTA, see figure 4). Note that 100% − (TA1 + TA2) represents a per cent of the cases in which both grasp type and size were correct but the user still failed to complete the task. Put differently, this outcome measure includes the trials in which the automatic or semi-automatic grasp control was successfully accomplished, but the subjects failed the task afterwards, due to some other reasons. This could be, for example, an unintentional myoelectric triggering while approaching the object or a slippage of the object from the hand due to the prosthesis construction. We grouped these cases together into a common category of myoelectric/prosthesis control failure. As explained above, although this measure does reflect the overall performance of the conglomerate system, it is not related to its core features, which are automatic and semi-automatic grasp control. In the case of the semi-automatic control (SEMI-AR, SEMI-AR-RE, SEMI-VIS-RE), the outcomes GC2–4 were measured after the subject had fine-tuned the system, whereas in the case of the automatic control (AUTO-AR), they were the values estimated by the stereovision grasp decoder. In the SEMI conditions, we also registered if the user corrected the controller decisions (aperture size, grasp type). We assumed that the user corrected the preshape if he/she corrected either the grasp type or the aperture size. We used a two-sided t-test for dependent samples (repeated measures) for pair-wise comparison between the conditions. A p-value of 0.05 was selected as the threshold for statistical significance.

The overall results for the performance of the stereovision grasp decoder (1560 trials in total) demonstrate that stereovision can be used to estimate the grasp relevant properties in a set of target objects with different shapes and sizes. The experimental task showed that after only a few minutes of training, the subjects were able to operate the system successfully and use the fully automatic control of grasping.

The autonomous operation was not flawless but the subjects were able to correct most of the errors by exploiting the AR feedback to fine-tune the decisions of the stereovision grasp decoder (semi-automatic control). The subjects successfully understood the meaning of the AR feedback, perceiving correctly the information about the grasp type and size. Also, subjects understood the bidirectional control loop, the concept of control sharing and they operated the system smoothly through different phases comprising the full control cycle (i.e., TTA was only few seconds). They successfully used the AR feedback information about the prosthesis state as well as controller decisions to make appropriate corrective actions when needed, bringing the control of grasping very close to an ideal performance (∼100%). Thus, all of this demonstrates the feasibility of the overall conceptual solution and its components, such as stereovision grasp decoder and AR bidirectional communication interface. In addition, the rate of failure in the task accomplishment due to incorrect grasp control (TA2) was consistently very small (∼2–3%) in all the SEMI-AR conditions. Importantly, the transition from automatic to semi-automatic control was almost effortless and consisted in the introduction of a few additional simple myoelectric commands to adjust the decisions of the stereovision grasp decoder. The closed-loop system demonstrated very good robustness when the user was challenged by the introduction of the intentional errors into the automatic control (SEMI-AR-RE). The errors were present in every trial and the error size was such that in most of the cases it would have been impossible to accomplish the grasp without a correction. However, this did not influence the overall performance.

Finally, the feasibility of the AR feedback was further confirmed by the last test (SEMI-AR-RE versus SEMI-VIS-RE) which demonstrated that the subjects were able to exploit the novel AR feedback as easily as the normal visual feedback. The subjects perceived the virtual object embedded into the real scene and were able to correctly compare its size against the size of the real objects as easily as when comparing the size of the real objects directly (hand opening versus object size). It is important to emphasize that the scenarios SEMI-AR-RE and SEMI-VIS-RE do not necessarily reflect the expected use of the AR and normal visual feedback in the potential real application. It is likely that trained subjects will learn to rely more on the feed-forward control and use feedback only when necessary.

When analyzing the task completion success rate across the experimental scenarios it is notable that the behavior of the conglomerate system was also affected by failures unrelated to the automatic/semi-automatic control of grasping, but to the myoelectric control and prosthesis design (myoelectric/prosthesis control failure rate (figures 7 and 9). This depended on several factors: (1) individual ability of the subject to generate and control EMG signals as well as dexterity in prosthesis handling, (2) intrinsic properties of the prosthetic device and its mounting system and (3) a simple method to process the EMG signals. Therefore, these errors could be minimized by providing the prospective users with more training so that they can generate more consistent muscle activations while manipulating the prosthesis, by improving the prosthesis mechanical design (e.g., silicone coating over the fingers to prevent object slipping), and by implementing more robust methods for myoelectric control. These errors are general issues common to any prosthetic system and, as explained before, they do not reflect the performance of the core feature of the novel approach (semi-automatic control of grasping).

Computer vision analysis was based on the standard state-of-the-art methods. The performance in the automatic control of grasping achieved during the experimental sessions indicates that the computer vision module was relatively robust and reliable. The robustness is further illustrated in figure 10, which demonstrates the system operation in a rich, cluttered environment. The figure depicts several scenes in which the subject successfully selected and grasped four target objects. It is worth noting that in both cases the scene included many other objects in addition to the target. Nevertheless, the system correctly located and segmented the target object as shown by the semi-transparent green overlay representing the pixels allocated to the object (figure 10, left). The segmentation, modeling, size estimation and grasp type selection (figure 10, right) were all correct despite the richness of the scene and the fact that the target object was partially occluded by the object(s) placed in front of it.

Zoom In Zoom Out Reset image size

The analysis of the scene by using stereovision is much more powerful and robust compared to the use of a single camera [35], but still suffers from well-known limitations like pixel correspondence problems [51]. This was also registered in our experiments. During the object targeting phase, the computer vision module sometimes failed to segment out the target object or there was a 'spillover' from the area of the target into the neighboring objects or the background. However, this was quite a rare event (<3%) and it did not significantly influence the performance due to the following countermeasures: (1) the 'snapshot and analysis' was repeated continuously during targeting and an occasional failure would be most often corrected in the very next cycle; (2) the users could rely on the AR feedback to assess the quality of segmentation and avoid triggering when large errors were evident.

The system developed is an illustrative example of how an artificial controller can be enriched with an additional, non-conventional information source (stereo camera pair) and high level processing (cognitive like reasoning) to achieve fully automatic control of the functions that are conventionally the responsibility of the user (e.g., hand preshaping). In this scheme, the user is able to 'release' predefined 'motor programs' performing relatively complex functions, instead of continuously monitoring the task. This substantially simplifies the myoelectric interface, which only needs to implement a simple triggering mechanism, thereby reducing the burden on the user. This can be advantageous especially in the case of modern dexterous prosthetic hands and/or full arm prostheses. The presented control concept scales smoothly with the system complexity. In fact, the more complex the system is, the higher and more obvious will be a discrepancy between the necessary user efforts and required system functionality. For example, in the case of an entire upper limb prosthesis, the stereovision control could be used both to preshape the hand and to navigate the arm to reach and grasp the selected target object (stereovision servoing [52]). The complex 'preshape and reach program' could be triggered via a simple myoelectric command.

The idea to close the loop 'through' the user by providing feedback about the state of the prosthesis is not novel [53]. However, the current work proposes a fundamentally novel interface to accomplish this goal. Compared to the 'classical' methods of electrical or direct mechanical stimulation, the most important advantage of the AR feedback is that it can exploit the high bandwidth and flexibility that is available within the visual communication channel. This is reflected both in the sheer volume of information that can be communicated to the subject as well as in the form in which that information can be presented. In the current work, the AR feedback was projected into the scene, next to the target object and a virtual box was used to implement 'artificial proprioception', i.e., the size of the box was proportional to the actual aperture size of the hand. As stated before, the system operation and experimental paradigm were not specifically designed to test the usefulness of the AR feedback and/or the potential advantages of the AR versus normal visual feedback. Instead, the aim was only to test the general feasibility since the AR interface has been used in this context for the first time, i.e., testing if the AR interface can be used as a component for implementing user corrections within a novel control system. However, even this first basic implementation demonstrates some possible benefits. Namely, as explained in the experimental protocol, in SEMI-AR and SEMI-AR-RE scenarios, the users operated the hand (i.e., correcting grasp type and size) without seeing it, i.e., the hand was outside the field of view and the subjects were looking at the object to be grasped. Note that this corresponds to the way grasping is performed in daily life by able-bodied subjects. Apart from promoting the normal use of the hand, artificial proprioception through AR can have some obvious advantages in certain situations, e.g., when the hand is partially occluded by some other object or is not completely visible from the user perspective, which is often the case during grasping (especially in cluttered environments). Moreover, the potential utility of AR feedback is even more evident when taking into account that the feedback could be much more sophisticated; for example, a full graphical model of the hand/prosthesis could be used to communicate complete and very detailed information in an intuitive way (e.g., position, contact and force of each individual finger). The flexibility and bandwidth of AR feedback were exploited in some other fields (e.g., robot-assisted surgeries [54]), but not in prosthetics which is a context with unique requirements. How to best utilize the potential of AR feedback in daily life prosthetic or clinical use is a very interesting and relevant question that will be addressed in the studies to follow.

To minimize the interference of AR feedback with the ongoing visual or other cognitive tasks, AR feedback could be simplified (e.g., a 2D object) and moved within the peripheral part of the visual field. The optimal form and positioning of AR feedback is an important question that will be addressed in future studies. It is also possible to combine several representations to best reconcile the needs of different tasks (e.g., unobtrusive, simple feedback most of the time and detailed feedback during specific, more demanding tasks).

A further novelty that was proposed in this work is the use of feedback not only to acknowledge to the user about the state of the device but also to allow him to monitor the decisions of the artificial controller. This gives the opportunity to the subject to supervise the automatic operation and take over control when needed, effectively implementing a bilateral communication between the user and controller. In the semi-automatic framework, the control is therefore shared between the two agents (user and controller) and the optimal integration of the two control loops (manual and automatic) will be an important question to address in future research.

An important feature of the presented concept and corresponding implementation is a modular design. Strictly speaking, both stereovision and AR feedback could be used on their own, independently of each other. For example, the stereovision could implement automatic control, while the corrections are performed using conventional visual feedback (as in the SEMI-VIS-RE condition). Or, the AR could be used to provide feedback about the prosthetic device that is controlled 'manually' using a classic myoelectric control. The placement of the sensor element is also flexible. In the current implementation, the cameras were embedded within the front glasses, but a miniature sensor could be placed on the sides or above the glasses, for example, into the glass frame, or even incorporated into the hand or a prosthetic socket. Truly wearable and cosmetically acceptable, mobile solutions for the AR, such as specialized glasses and even contact lenses, are being developed and some models (e.g., Google Glass project [46]) are already available in limited quantities and are expected to become widely accessible very soon. This, together with the development and availability of fast processing cores within small form factor embedded systems, can provide the necessary technological framework for the practical implementation of the presented control concept. From the technical standpoint, the developed system relies on state-of-the-art technologies that are in the focus of current research efforts and are also fast developing. Although some of the components might not be ready for immediate practical application in daily life, this might change in the very near future. Stereovision is not the only method to implement the proposed control concept. The necessary information for the control of grasping is an estimate of the 3D shape of the target object and any technology providing a similar output (i.e., 3D point cloud) could be plugged into the control scheme in a straightforward manner. For example, the active methods based on IR depth sensing derived from Microsoft Kinect are more robust than passive stereovision. In general, this technology is developing very fast, evolving towards solutions that will soon become low cost and physically small, and thereby very convenient for practical applications (e.g., Prime Sense [55], Creative Senz3D [56]). Finally, advanced scene processing [57] and grasp planning methods [58] that were investigated intensively in robotics research could be used to improve the prosthesis preshape module, but the candidate solutions could be only those approaches that can cope with the constraints of this specific application (e.g., incomplete input information, responsiveness to user decisions, ad hoc scene etc).

Importantly, the presented approach and its components are a rather general solution that could be applied with relatively minor adjustments in many contexts. In principle, it could be used with any assistive system that includes a multi-grasp end effector (e.g., electrical stimulator, hand exoskeleton, hand prosthesis). To port the control scheme (figure 3) to a different device, the prosthesis control module would have to be replaced with a module providing a low level interface to that specific system. All the other components could remain unchanged or, in the worst case, they would have to be somewhat adapted to support the new application (e.g., available grasps). Alternatively, only a part of the functionality could be ported. For example, stereovision grasp decoder and myoelectric control could be used for grasp type and size selection in EMG-triggered FES. Similarly, AR glasses could be used not for control as in this study, but only to provide AR feedback, supplementing the impaired sensations in patients suffering from a neurological condition (e.g., stroke). As explained below, stereovision can be extended to the control of orientation, and in this case, the scope of potential applications is even larger: a rehabilitation robot for assistance in reaching, full arm prosthesis, a hybrid system combining a reaching robot and an FES device for grasping. Finally, the presented control system could indeed be used in daily life for prosthetic or orthotic applications, which is similar to the scenario presented in the current study, but also only during a limited time period, for therapy (e.g., functional electrical therapy, robotic rehabilitation) or training (e.g., simplifying myoelectric training for a complex prosthesis).

The presented system is an example of the sensor fusion approach to the control of prosthetic devices [34]. In the current system, control was implemented by combining visual information and myoelectric commands. This approach could be extended further. The 3D information about the scene and the target object could be integrated with information about the pose of the prosthesis provided by inertial measurement sensors placed on the device, resulting in an adaptive system that can reconfigure automatically depending on the side and angle of 'attack'. For example, the hand aperture could readjust depending on the side the user is approaching the object or in the case the hand is equipped with wrist rotator and/or flexion/extension, the system can control the hand orientation. The future steps will be system improvement, in terms of more functions and practicality, followed by an evaluation in the specific context and on the actual target population, i.e., amputees or patients, depending on the selected application. The goal of these tests will be to assess the usability, acceptance and finally the efficiency of the proposed interface.

We believe that there are many potential benefits of using the AR interface versus normal visual feedback or tactile stimulation, and investigating this will be the topic of the studies to follow. For example, as a separate study, we plan to perform an in-depth analysis of the performance and possible applications of proposed AR feedback concepts. Specifically, one of the goals would be to compare the regular open-loop (visual inspection) and closed-loop (AR feedback) prosthesis control in more complex real-life scenarios (e.g., cluttered environment with occlusions, adjusting the hand while reaching for the object). The current AR interface could also be easily extended to feedback additional signals. Specifically, the AR channel could be used for the closed-loop control of the grasping force or prosthesis orientation. For example, the grasping force could be visualized as a vertical bar in the peripheral visual field with the height of the bar being proportional to the force amplitude. These developments are also something that we plan to address in future studies, as we believe they are the key for unlocking the full potential of the AR feedback interface.

We are well aware that there are drawbacks that could jeopardize some of the potential applications. For example, when applied in prosthetic or orthotic scenarios, the system could provide simple and effective control of a complex device, but at the expense of adding additional components. In the current version, AR glasses with stereo camera pair have to be worn by the subject. For the user of a prosthesis, this could be an additional nuisance that could further compromise the already sensitive process of device acceptance. However, as always, this ultimately depends on the cost benefit ratio: if the user gets a complex multi-degree of freedom prosthesis that can react to a simple command by automatically reaching and grasping for the desired object, we could expect that he/she would likely be much more eager to wear an additional component. In fact, the control scheme proposed here was filed as a joint patent [59] with our industrial partner (Otto Bock Healthcare GmbH, Vienna, AT), one of the leading manufacturers of prosthetic equipment. Importantly, there are some recent technological developments that could significantly improve the system appearance, thereby overcoming the present drawbacks. Finally, this issue is less likely to be a serious obstacle when considering the potential short term system application (therapy or training). There are studies reporting the application of virtual reality equipment in patients with paralysis [60] as well as in amputees [61]. In conclusion, this study presented a set of novel methods and demonstrated their feasibility using a first developed prototype. The next steps are to further improve the system (ongoing work) and then to benchmark the novel approach against the current state-of-the-art prosthetic systems (semi-automatic versus manual myoelectric control, normal visual and/or tactile feedback versus AR feedback).