Understanding How Well You Understood – Context- Sensitive Interpretation of Multimodal User Feedback

Abstract

Human interlocutors continuously show behaviour indicative of their perception, understanding, acceptance and agreement of and with the other’s utterances [1,4]. Such evidence can be provided in the form of verbal-vocal feedback signals, head gestures, facial expressions or gaze and often interacts with the current dialogue context. As feedback signals are able to express subtle differences in meaning, we hypothesise that they tend to reflect their producer’s mental state quite accurately.

To be cooperative and human-like dialogue partners, virtual conversational agents should be able to interpret their user’s evidence of understanding and to react appropriately to it by adapting to their needs [2]. We present a Bayesian network model for context-sensitive interpretation of listener feedback for such an ‘attentive speaker agent’, which takes the user’s multimodal behaviour (verbal-vocal feedback, headgestures, gaze) as well as its own utterance and knowledge of the dialogue domain into account to form a model of the user’s mental state.