Nathan Semertzidis
ABSTRACT
The subjective experience of emotion is notoriously difficult to interpersonally communicate. We believe that technology can challenge this notion through the design of neuroresponsive systems for interpersonal communication. We explore this through "Neo-Noumena", a communicative neuroresponsive system that uses brain-computer interfacing and artificial intelligence to read one's emotional states and dynamically represent them to others in mixed reality through two head-mounted displays. In our study five participant pairs were given Neo-Noumena for three days, using the system freely. Measures of emotional competence demonstrated a statistically significant increase in participants' ability to interpersonally regulate emotions. Furthermore, participant interviews revealed themes regarding Spatiotemporal Actualization, Objective Representation, and Preternatural Transmission. We also suggest design strategies for future augmented emotion communication systems. We intend that work gives guidance towards a future in which our ability to interpersonally communicate emotion is augmented beyond traditional experience.
Supplemental Material
- Bai, Zhen, Blackwell, Alan F, and Coulouris, George. 2015. Exploring expressive augmented reality: The FingAR puppet system for social pretend play. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems ACM, 1035--1044.Google Scholar
Digital Library
- Beck, Luna, Kumschick, Irina R, Eid, Michael, and Klann-Delius, Gisela. 2012. Relationship between language competence and emotional competence in middle childhood. Emotion 12, 3, 503.Google ScholarCross Ref
- Bene?, Bedrich and Hartman, Christopher. 2006. Autonomous boids. Computer Animation and Virtual Worlds 17, 3--4, 199--206.Google ScholarCross Ref
- Bermejo-Martins, Elena, López?Dicastillo, Olga, and Mujika, Agurtzane. 2018. An exploratory trial of a health education programme to promote healthy lifestyles through social and emotional competence in young children: Study protocol. Journal of advanced nursing 74, 1, 211--222.Google ScholarCross Ref
- Bies, Alexander J, Blanc-Goldhammer, Daryn R, Boydston, Cooper R, Taylor, Richard P, and Sereno, Margaret E. 2016. Aesthetic responses to exact fractals driven by physical complexity. Frontiers in human neuroscience 10, 210.Google Scholar
- Boehner, Kirsten, Depaula, Rogerio, Dourish, Paul, and Sengers, Phoebe. 2005. Affect: from information to interaction. In Proceedings of the 4th decennial conference on Critical computing: between sense and sensibility ACM, 59--68.Google ScholarDigital Library
- Boehner, Kirsten, Depaula, Rogério, Dourish, Paul, and Sengers, Phoebe. 2007. How emotion is made and measured. International Journal of HumanComputer Studies 65, 4, 275--291.Google ScholarDigital Library
- Brasseur, Sophie, Grégoire, Jacques, Bourdu, Romain, and Mikolajczak, Moïra. 2013. The profile of emotional competence (PEC): Development and validation of a self-reported measure that fits dimensions of emotional competence theory. Plos one 8, 5, e62635.Google ScholarCross Ref
- Braun, Virginia and Clarke, Victoria. 2012. Thematic analysis.Google Scholar
- Cauchard, Jessica R, Zhai, Kevin Y, Spadafora, Marco, and Landay, James A. 2016. Emotion encoding in human-drone interaction. In 2016 11th ACM/IEEE International Conference on HumanRobot Interaction (HRI) IEEE, 263--270.Google ScholarCross Ref
- Chen, Min, Zhou, Ping, and Fortino, Giancarlo. 2016. Emotion communication system. IEEE Access 5, 326--337.Google ScholarCross Ref
- Ciarrochi, Joseph, Scott, Greg, Deane, Frank P, and Heaven, Patrick Cl. 2003. Relations between social and emotional competence and mental health: A construct validation study. Personality and Individual Differences 35, 8, 1947--1963.Google ScholarCross Ref
- Delgado-Mata, Carlos, Martinez, Jesus Ibanez, Bee, Simon, Ruiz-Rodarte, Rocio, and Aylett, Ruth. 2007. On the use of virtual animals with artificial fear in virtual environments. New Generation Computing 25, 2, 145--169.Google ScholarDigital Library
- Denham, Susanne A, Blair, Kimberly A, Demulder, Elizabeth, Levitas, Jennifer, Sawyer, Katherine, Auerbach--Major, Sharon, and Queenan, Patrick. 2003. Preschool emotional competence: Pathway to social competence? Child development 74, 1, 238--256.Google Scholar
- Faltaous, Sarah, Haas, Gabriel, Barrios, Liliana, Seiderer, Andreas, Rauh, Sebastian Felix, Chae, Han Joo, Schneegass, Stefan, and Alt, Florian. 2019. BrainShare: A Glimpse of Social Interaction for Locked-in Syndrome Patients. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems ACM, LBW0155.Google ScholarDigital Library
- Fan, Jianyu, Thorogood, Miles, and Pasquier, Philippe. 2016. Automatic soundscape affect recognition using a dimensional approach. Journal of the Audio Engineering Society 64, 9, 646--653.Google ScholarCross Ref
- Freedberg, David and Gallese, Vittorio. 2007. Motion, emotion and empathy in esthetic experience. Trends in cognitive sciences 11, 5, 197--203.Google Scholar
- Frey, Jérémy, Grabli, May, Slyper, Ronit, and Cauchard, Jessica R. 2018. Breeze: Sharing Biofeedback Through Wearable Technologies. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems ACM, 645.Google ScholarDigital Library
- Grnfelde, M?ra. 2018. The Four Dimensions of Embodiment and the Experience of Illness. AVANT. Pismo Awangardy Filozoficzno-Naukowej, 2, 107127.Google Scholar
- Hecht, Heiko, Welsch, Robin, Viehoff, Jana, and Longo, Matthew R. 2019. The shape of personal space. Acta psychologica 193, 113--122.Google Scholar
- Hodzic, Sabina, Ripoll, Pilar, Lira, Eva, and Zenasni, Franck. 2015. Can intervention in emotional competences increase employability prospects of unemployed adults? Journal of Vocational Behavior 88, 28--37.Google ScholarCross Ref
- Homan, Richard W, Herman, John, and Purdy, Phillip. 1987. Cerebral location of international 10-- 20 system electrode placement. Electroencephalography and clinical neurophysiology 66, 4, 376--382.Google Scholar
- Howell, Noura, Chuang, John, De Kosnik, Abigail, Niemeyer, Greg, and Ryokai, Kimiko. 2018. Emotional Biosensing: Exploring Critical Alternatives. Proceedings of the ACM on HumanComputer Interaction 2, CSCW, 69.Google ScholarDigital Library
- Howell, Noura, Devendorf, Laura, Tian, Rundong Kevin, Vega Galvez, Tomás, Gong, Nan-Wei, Poupyrev, Ivan, Paulos, Eric, and Ryokai, Kimiko. 2016. Biosignals as social cues: Ambiguity and emotional interpretation in social displays of skin conductance. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems ACM, 865--870.Google ScholarDigital Library
- Howell, Noura, Devendorf, Laura, Vega Gálvez, Tomás Alfonso, Tian, Rundong, and Ryokai, Kimiko. 2018. Tensions of data-driven reflection: A case study of real-time emotional biosensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems ACM, 431.Google ScholarDigital Library
- Ibáñez, Jesús. 2011. Showing emotions through movement and symmetry. Computers in Human Behavior 27, 1, 561--567.Google ScholarDigital Library
- Introna, Lucas D. 1996. Notes on ateleological information systems development. Information Technology & People 9, 4, 20--39.Google ScholarCross Ref
- Kemp, Charles, Xu, Yang, and Regier, Terry. 2018. Semantic typology and efficient communication. Annual Review of Linguistics 4, 109--128.Google ScholarCross Ref
- Kitson, Alexandra, Dipaola, Steve, and Riecke, Bernhard E. 2019. Lucid Loop: A Virtual Deep Learning Biofeedback System for Lucid Dreaming Practice. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems ACM, LBW1322.Google ScholarDigital Library
- Kitson, Alexandra, Schiphorst, Thecla, and Riecke, Bernhard E. 2018. Are you dreaming?: a phenomenological study on understanding lucid dreams as a tool for introspection in virtual reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems ACM, 343.Google ScholarDigital Library
- Koelstra, Sander, Muhl, Christian, Soleymani, Mohammad, Lee, Jong-Seok, Yazdani, Ashkan, Ebrahimi, Touradj, Pun, Thierry, Nijholt, Anton, and Patras, Ioannis. 2011. Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing 3, 1, 18--31.Google Scholar
- Kotsou, Ilios, Nelis, Delphine, Grégoire, Jacques, and Mikolajczak, Moïra. 2011. Emotional plasticity: Conditions and effects of improving emotional competence in adulthood. Journal of Applied Psychology 96, 4, 827.Google ScholarCross Ref
- La Delfa, Joseph, Baytas, Mehmet Aydin, Wichtowski, Olivia, Khot, Rohit Ashok, and Mueller, Florian Floyd. 2019. Are Drones Meditative? In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems ACM, INT046.Google ScholarDigital Library
- La Delfa, Joseph, Jarvis, Robert, Khot, Rohit Ashok, and Mueller, Florian'floyd'. 2018. Tai Chi In The Clouds: Using Micro UAV's To Support Tai Chi Practice. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts ACM, 513--519.Google ScholarDigital Library
- Leijnen, Stefan and Van Veen, Fjodor. 2016. ArkaNet: Investigating Emergent Gameplay and Emergence.Google Scholar
- Liu, Yisi, Sourina, Olga, and Nguyen, Minh Khoa. 2010. Real-time EEG-based human emotion recognition and visualization. In 2010 international conference on cyberworlds IEEE, 262--269.Google ScholarDigital Library
- Lo, Shao-Kang. 2008. The nonverbal communication functions of emoticons in computer-mediated communication. CyberPsychology & Behavior 11, 5, 595--597.Google ScholarCross Ref
- Mann, Steve. 2001. Wearable computing: Toward humanistic intelligence. IEEE Intelligent Systems 16, 3, 10--15.Google ScholarDigital Library
- Martínez, Ruiz Xicoténcatl. 2019. Interpreting Poetics, Cognitions and Aesthetic Emotion. In Time for Educational Poetics Brill Sense, 71--75.Google Scholar
- Mikolajczak, Moïra, Avalosse, Hervé, Vancorenland, Sigrid, Verniest, Rebekka, Callens, Michael, Van Broeck, Nady, Fantini-Hauwel, Carole, and Mierop, Adrien. 2015. A nationally representative study of emotional competence and health. Emotion 15, 5, 653.Google ScholarCross Ref
- Nunes, Francisco, Verdezoto, Nervo, Fitzpatrick, Geraldine, Kyng, Morten, Grönvall, Erik, and Storni, Cristiano. 2015. Self-care technologies in HCI: Trends, tensions, and opportunities. ACM Transactions on Computer-Human Interaction (TOCHI) 22, 6, 33.Google ScholarDigital Library
- Pelowski, Matthew, Markey, Patrick S, Forster, Michael, Gerger, Gernot, and Leder, Helmut. 2017. Move me, astonish me? delight my eyes and brain: The Vienna integrated model of top-down and bottom-up processes in art perception (VIMAP) and corresponding affective, evaluative, and neurophysiological correlates. Physics of Life Reviews 21, 80--125.Google ScholarCross Ref
- Pelowski, Matthew, Markey, Patrick S, Lauring, Jon O, and Leder, Helmut. 2016. Visualizing the impact of art: An update and comparison of current psychological models of art experience. Frontiers in human neuroscience 10, 160.Google Scholar
- Picard, Rosalind W., Vyzas, Elias, and Healey, Jennifer. 2001. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis & Machine Intelligence, 10, 1175--1191.Google ScholarDigital Library
- Roo, Joan Sol, Gervais, Renaud, and Hachet, Martin. 2016. Inner garden: An augmented sandbox designed for self-reflection. In Proceedings of the TEI'16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction ACM, 570--576.Google ScholarDigital Library
- Rude, Stephanie, Gortner, Eva-Maria, and Pennebaker, James. 2004. Language use of depressed and depression-vulnerable college students. Cognition & Emotion 18, 8, 1121--1133.Google ScholarCross Ref
- Schutte, Nicola S, Malouff, John M, Bobik, Chad, Coston, Tracie D, Greeson, Cyndy, Jedlicka, Christina, Rhodes, Emily, and Wendorf, Greta. 2001. Emotional intelligence and interpersonal relations. The Journal of social psychology 141, 4, 523--536.Google ScholarCross Ref
- Semertzidis, Nathan Arthur, Sargeant, Betty, Dwyer, Justin, Mueller, Florian Floyd, and Zambetta, Fabio. 2019. Towards Understanding the Design of Positive Pre-sleep Through a Neurofeedback Artistic Experience. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems ACM, 574.Google ScholarDigital Library
- Sliwinski, Jacek. 2019. Mindfulness and HCI. In Handbook of Research on Human-Computer Interfaces and New Modes of Interactivity IGI Global, 314--332.Google Scholar
- Spehar, Branka, Clifford, Colin Wg, Newell, Ben R, and Taylor, Richard P. 2003. Universal aesthetic of fractals. Computers & Graphics 27, 5, 813--820.Google ScholarCross Ref
- Springham, Neil and Huet, Val. 2018. Art as relational encounter: An ostensive communication theory of art therapy. Art Therapy 35, 1, 4--10.Google ScholarCross Ref
- Street, Nichola, Forsythe, Alexandra M, Reilly, Ronan, Taylor, Richard, and Helmy, Mai S. 2016. A complex story: Universal preference vs. individual differences shaping aesthetic response to fractals patterns. Frontiers in human neuroscience 10, 213.Google Scholar
- Tarvainen, Mika P, Hiltunen, Jaana K, Ranta-Aho, Perttu O, and Karjalainen, Pasi A. 2004. Estimation of nonstationary EEG with Kalman smoother approach: an application to event-related synchronization (ERS). IEEE Transactions on Biomedical Engineering 51, 3, 516--524.Google ScholarCross Ref
- Häuslschmid, Renate, and Hussmann, Heinrich. 2019. A Review & Analysis of Mindfulness Research in HCI: Framing Current Lines of Research and Future Opportunities. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems ACM, 457.Google Scholar
- Thorogood, Miles, Fan, Jianyu, and Pasquier, Philippe. 2019. A framework for computer-assisted sound design systems supported by modelling affective and perceptual properties of soundscape. Journal of New Music Research, 1--17.Google ScholarCross Ref
- Tishby, Naftali, Pereira, Fernando C, and Bialek, William. 2000. The information bottleneck method. arXiv preprint physics/0004057.Google Scholar
- Verbeek, Peter-Paul. 2015. Beyond interaction: A short introduction to mediation theory. Interactions 22, 3, 26--31.Google ScholarDigital Library
Index Terms
- Neo-Noumena: Augmenting Emotion Communication
Recommendations
Neo-Noumena
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing SystemsCommunicating emotional experiences is something core to being human, yet also notoriously difficult. With this considered, we acknowledge previous work in bio-responsive and neurofeedback systems which facilitate the externalization of subjective ...
AffectivelyVR: Towards VR Personalized Emotion Recognition
VRST '20: Proceedings of the 26th ACM Symposium on Virtual Reality Software and TechnologyWe present AffectivelyVR, a personalized real-time emotion recognition system in Virtual Reality (VR) that enables an emotion-adaptive virtual environment. We used off-the-shelf Electroencephalogram (EEG) and Galvanic Skin Response (GSR) physiological ...
A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system
Highlights- A novel LSTM based method is implemented for emotion recognition using EEG signals.
AbstractPositive and Negative emotions are experienced by the majority of individuals in their day-to-day life. It is important to control access of negative emotions because it may lead to several chronic health issues like depression and ...
Comments