Marco Cristani
Alessandro Vinciarelli
ABSTRACT
One of the main findings of cognitive sciences is that automatic processes of which we are unaware shape, to a significant extent, our perception of the environment. The phenomenon applies not only to the real world, but also to multimedia data we consume every day. Whenever we look at pictures, watch a video or listen to audio recordings, our conscious attention efforts focus on the observable content, but our cognition spontaneously perceives intentions, beliefs, values, attitudes and other constructs that, while being outside of our conscious awareness, still shape our reactions and behavior. So far, multimedia technologies have neglected such a phenomenon to a large extent. This paper argues that taking into account cognitive effects is possible and it can also improve multimedia approaches. As a supporting proof-of-concept, the paper shows not only that there are visual patterns correlated with the personality traits of 300 Flickr users to a statistically significant extent, but also that the personality traits (both self-assessed and attributed by others) of those users can be inferred from the images these latter post as "favourite".
- I. Arapakis, J.M. Jose, and P.D. Gray. Affective feedback: an investigation into the role of emotions in the information seeking process. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, pages 395--402. ACM, 2008. Google Scholar
Digital Library
- M. Baldauf, S. Dustdar, and F. Rosenberg. A survey on context-aware systems. International Journal of Ad Hoc and Ubiquitous Computing, 2(4):263--277, 2007. Google ScholarDigital Library
- F.J. Bernieri and J.S. Gillis. Judging rapport: Employing brunswik's lens model to study interpersonal sensitivity. In J.A. Hall and F.J. Bernieri, editors, Interpersonal Sensitivity. Theory and Measurement. Lawrence Erlbaum, 2001.Google Scholar
- J.C. Biesanz and S.G. West. Personality coherence: Moderating self--other profile agreement and profile consensus. Journal of Personality and Social Psychology, 79(3):425--437, 2000.Google ScholarCross Ref
- E. Brunswik. Perception and the representative design of psychological experiments. University of California Press, 1956.Google ScholarCross Ref
- A. Burdick, J. Drucker, P. Lunenfelds, T. Presner, and J. Schnapp. Digitial Humanities. MIT Press, 2012. Google ScholarDigital Library
- D. Comaniciu and P. Meer. Mean shift: a robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5):603 -- 619, 2002. Google ScholarDigital Library
- J.W. Condon and W.D. Crano. Inferred evaluation and the relation between attitude similarity and interpersonal attraction. Journal of Personality and Social Psychology, 54(5):789, 1988.Google ScholarCross Ref
- R. Cowie. The good our field can hope to do, the harm it should avoid. IEEE Transactions on Affective Computing (to appear), 2013. Google ScholarDigital Library
- R. Datta, D. Joshi, J. Li, and J. Wang. Studying aesthetics in photographic images using a computational approach. In Proceedings of the European Conference on Computer Vision, volume 3953 of Lecture Notes in Computer Science, pages 288--301. Springer Verlag, 2006. Google ScholarDigital Library
- J. Elster. Explaining Social Behavior. Cambridge University Press, 1997.Google Scholar
- D.C. Evans, S. D. Gosling, and A. Carroll. What elements of an online social networking profile predict target-rater agreement in personality impressions. In Proceedings of the International Conference on Weblogs and Social Media, pages 45--50, 2008.Google Scholar
- P. F. Felzenszwalb, R. B. Girshick, and D. McAllester. Discriminatively trained deformable part models, release 4. http://www.cs.brown.edu/~pff/latent-release4/, 2010.Google Scholar
- R. Fidel. Human Information Interaction. MIT Press, 2012. Google ScholarDigital Library
- S. Fitzgerald, D.C. Evans, and R.K. Green. Is your profile picture worth 1000 words? Photo characteristics associated with personality impression agreement. In Proceedings of AAAI International Conference on Weblogs and Social Media, 2009.Google Scholar
- World Economic Forum. Personal data: the emergence of a new asset class. Technical report, World Economic Forum, 2011.Google Scholar
- C.M. Georgescu. Synergism in low level vision. In Proceedings of the International Conference on Pattern Recognition, pages 150--155, 2002. Google ScholarDigital Library
- D. Johnson and J. Gardner. Personality, motivation and video games. In Proceedings of the Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction, pages 276--279, 2010. Google ScholarDigital Library
- N. Jojic and A. Perina. Multidimensional counting grids: Inferring word order from disordered bags of words. In Proceedings of Uncertainty in Artificial Intelligence, pages 547--556, 2011.Google Scholar
- C.M. Judd, L. James-Hawkins, V. Yzerbyt, and Y. Kashima. Fundamental dimensions of social judgment: Unrdestanding the relations bteween judgments of competence and warmth. Journal of Personality and Social Psychology, 89(6):899--913, 2005.Google ScholarCross Ref
- A.M. Kaplan and M. Haenlein. Users of the world, unite! The challenges and opportunities of social media. Business Horizons, 53(1):59--68, 2010.Google ScholarCross Ref
- M. Kosinski, D. Stillwell, and T. Graepel. Private traits and attributes are predictable from digital records of human behavior. Proceeedings of the National Academy of Sciences, 110(15):5802--5805, 2013.Google ScholarCross Ref
- K. Krippendorff. Reliability in content analysis. Human Communication Research, 30(3):411--433, 2004.Google Scholar
- Z. Kunda. Social cognition: Making sense of people. The MIT Press, 1999.Google ScholarCross Ref
- J. Leskovec, L. Adamic, and B. Huberman. The dynamics of viral marketing. ACM Transactions on the Web, 1(1):5, 2007. Google ScholarDigital Library
- M.S. Lew, N. Sebe, D. Chabane, and R. Jain. Content-based multimedia information retrieval: State of the art and challenges. ACM Transactions on Multimedia Computing, Communications, and Applications, 2(1):1--19, 2006. Google ScholarDigital Library
- P. Lovato, A. Perina, N. Sebe, O. Zandoná, A. Montagnini, M. Bicego, and M. Cristani. Tell me what you like and I'll tell you what you are: discriminating visual preferences on Flickr data. In K.M. Lee, Y. Matsushita, J.M. Rehg, and Z. Hu, editors, Proceedings of the Asian Conference on Computer Vision, volume Lecture Notes in Computer Science 7724. Springer Verlag, 2012. Google ScholarDigital Library
- J. Machajdik and A. Hanbury. Affective image classification using features inspired by psychology and art theory. In Proceedings of the ACM International Conference on Multimedia, pages 83--92, 2010. Google ScholarDigital Library
- F. Mairesse, M. A. Walker, M. R. Mehl, and R. K. Moore. Using linguistic cues for the automatic recognition of personality in conversation and text. Journal of Artificial Intelligence Research, 30:457--500, 2007. Google ScholarDigital Library
- G. Marchionini. Human--information interaction research and development. Library & Information Science Research, 30(3):165--174, 2008.Google ScholarCross Ref
- K.V. Mardia and P.E. Jupp. Directional Statistics. Wiley Series in Probability and Statistics. Wiley, 2009.Google Scholar
- M. Minelli, M. Chambers, and A. Dhiraj. Big Data, Big Analytics. Wiley, 2013.Google Scholar
- C. Nass and S. Brave. Wired for speech: How voice activates and advances the Human-Computer relationship. The MIT Press, 2005. Google ScholarDigital Library
- R.E. Nisbett and T.D. Wilson. The halo effect: Evidence for unconscious alteration of judgments. Journal of Personality and Social Psychology, 35(4):250--256, 1977.Google ScholarCross Ref
- F.J. Olhorst. Big Data Analytics. Wiley, 2013.Google Scholar
- A. Oliva and A. Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, 42(3):145--175, 2001. Google ScholarDigital Library
- M. Pantic and A. Vinciarelli. Implicit Human-Centered Tagging. IEEE Signal Processing Magazine, 26(6):173--180, 2009.Google ScholarCross Ref
- A. Perina and N. Jojic. Image analysis by counting on a grid. In Proceedings of the International Conference on Computer Vision and Pattern Recognition, pages 1985--1992, 2011. Google ScholarDigital Library
- A. Peterson Bishop, N.A. van House, and B.P. Buttenfields, editors. Digital Library Use. MIT Press, 2003.Google ScholarDigital Library
- B. Rammstedt and O. P. John. Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. in Journal of Research in Personality, 41:203--212, 2007.Google ScholarCross Ref
- S. Ray. Multiple instance regression. In Proceedings of the International Conference on Machine Learning, pages 425--432, 2001. Google ScholarDigital Library
- D. Rosenberg. Data before the Fact. In L. Gitelman, editor, Raw data is an oxymoron, pages 15--40. MIT Press, 2013.Google Scholar
- G. Saucier and L.R. Goldberg. The language of personality: Lexical perspectives on the five-factor model. In J.S. Wiggins, editor, The Five-Factor Model of Personality. 1996.Google Scholar
- K.R. Scherer. Personality markers in speech. In Social markers in speech, pages 147--209. Cambridge University Press, Cambridge, 1979.Google Scholar
- K. Sohn, D.Y. Jung, H. Lee, and A.O. Hero. Efficient learning of sparse, distributed, convolutional feature representations for object recognition. In IEEE International Conference on Computer Vision, pages 2643--2650, 2011. Google ScholarDigital Library
- J. Suler. The psychotherapeutics of online photosharing. International Journal of Applied Psychoanalytic Studies, 6(4):339--344, 2009.Google ScholarCross Ref
- H. Tamura, S. Mori, and T. Yamawaki. Texture features corresponding to visual perception. IEEE Transactions on Systems, Man and Cybernetics, 8(6), 1978.Google ScholarCross Ref
- R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58:267--288, 1994.Google Scholar
- J.S. Uleman, L.S. Newman, and G.B. Moskowitz. People as flexible interpreters: Evidence and issues from spontaneous trait inference. In M.P. Zanna, editor, Advances in Experimental Social Psychology, volume 28, pages 211--279. Elsevier, 1996.Google Scholar
- J.S. Uleman, S.A. Saribay, and C.M. Gonzalez. Spontaneous inferences, implicit impressions, and implicit theories. Annual Reviews of Psychology, 59:329--360, 2008.Google ScholarCross Ref
- P. Valdez and A. Mehrabian. Effects of color on emotions. Journal of Experimental Psychology General, 123(4):394--409, 1994.Google ScholarCross Ref
- P.A. Viola and M.J. Jones. Robust real-time face detection. International Journal of Computer Vision, 57(2):137--154, 2004. Google ScholarDigital Library
- J. Weiser. Phototherapy techniques: Exploring the secrets of personal snapshots and family albums. Jossey-Bass San Francisco, 1993.Google Scholar
- C.Y. Yaakub, N. Sulaiman, and C.W. Kim. A study on personality identification using game based theory. In Proceedings of the International Conference on Computer Technology and Development, pages 732--734, 2010.Google ScholarCross Ref
- V. Yanulevskaya, J. Uijlings, E. Bruni, A. Sartori, E. Zamboni, F. Bacci, D. Melcher, and N. Sebe. In the eye of the beholder: employing statistical analysis and eye tracking for analyzing abstract paintings. In Proceedings of the ACM international conference on Multimedia, pages 349--358, 2012. Google ScholarDigital Library
- N. Yee, N. Ducheneaut, L. Nelson, and P. Likarish. Introverted elves & conscientious gnomes: The expression of personality in World of Warcraft. In Proceedings of the Annual Conference on Human Factors in Computing Systems, pages 753--762, 2011. Google ScholarDigital Library
Index Terms
- Unveiling the multimedia unconscious: implicit cognitive processes and multimedia content analysis
Recommendations
Unconscious word processing engages a distributed network of brain regions
A briefly exposed visual stimulus may not be consciously perceived if it is preceded and followed by a dissimilar visual pattern or mask. Despite the subject's lack of awareness, prior behavioral studies have shown that such masked stimuli, nevertheless,...
Unconscious learning versus visual perception: Dissociable roles for gamma oscillations revealed in meg
Oscillatory synchrony in the gamma band (30-120 Hz) has been involved in various cognitive functions including conscious perception and learning. Explicit memory encoding, in particular, relies on enhanced gamma oscillations. Does this finding extend to ...
Unconscious and conscious processing of color rely on activity in early visual cortex: A tms study
Chromatic information is processed by the visual system both at an unconscious level and at a level that results in conscious perception of color. It remains unclear whether both conscious and unconscious processing of chromatic information depend on ...
Comments