Logics based on qualitative descriptors for scene understanding

Abstract

An approach for scene understanding based on qualitative descriptors, domain knowledge and logics is proposed in this paper. Qualitative descriptors, qualitative models of shape, colour, topology and location are used for describing any object in the scene. Two kinds of domain knowledge are provided: (i) categorizations of objects according to their qualitative descriptors, and (ii) semantics for describing the affordances, mobility and other functional properties of target objects. First order logics are obtained for reasoning and scene understanding. Tests were carried out at the Interact@Cartesium scenario and promising results were obtained.

Introduction

In an envisaged future in which we live in smart homes and intelligent robots populate our world we imagine an everyday phone conversation with our home ambient intelligent vision system (or robot), asking it “Is everything fine?” which the system happily confirms. However, if everything is fine or not at home from the human perspective is not a simple observable fact for a machine. It requires cognitive interpretation. So that an artificial intelligent agent can reach such a conclusion in alignment with the human, the agent needs to be able to observe its environment and interpret the information perceived in a cognitive manner. Let us continue with the example and consider that our robot detects a patch of water on the floor—should that be considered to be normal? Clearly, for a human, the interpretation of such observed information varies, depending on where and when the patch of water was recognized: a patch of water on the bathroom floor close to the shower can be a typical outcome of normal use, while a patch of water on a parquet flooring next to a window after a rainy day is a clear indication of something wrong (i.e. maybe a broken window, maybe a leaking flowerpot, etc.). Interpreting the meaning of a feature occurrence like the patch of water thus requires consideration of context. Due to the variety of possible situations, reasoning is required in order to state a hypothesis and reach a conclusion based on a manageable knowledge base which can differentiate normal from abnormal situations. In order to realize a system like the one sketched above, integrating perception with abstract reasoning in a way which matches human interpretation is necessary.

Ambient intelligent (AmI) systems and companion robots interacting with human beings are the scenarios proposed in this paper. These systems usually integrate digital cameras, from which they obtain information about the environment. As such systems need to align their interpretation with the human interpretation, one can go a step deeper and assume that the ideal systems for interacting with people would be those capable of interpreting their environment (captured by a digital image) cognitively (that is similarly to how people do it). In this way, those systems may directly align their concepts with human concepts and thus provide common ground for establishing sophisticated communication.

Psychological studies on image description point to the fact that people generally find the most relevant content and use words (qualitative tags) to describe it [1], [2], [3], [4]. Usually different colours/textures in an image indicate different objects/regions of interest to people [5]. Moreover, cognitive studies [6] explain that qualitative representations of images are in many ways similar to the mental images that people report when they describe what they have seen or when they attempt to answer questions based on visual memories [7].

Because digital images represent visual data numerically, most image processing has been successfully carried out by applying mathematical techniques to obtain and describe image content. Some examples are object feature invariant descriptors and detectors, such as SIFT and SURF (see the work by Mikolajczyk et al. [8] for an overview). All these approaches succeeded in extracting and using features from digital images for describing complex real world objects and then detecting them within other images. However, these approaches need to produce and store in memory huge numerical descriptions that cannot be interpreted or given a meaning. To establish semantics, features need to be grouped and linked to cognitive concepts first. A disadvantage of these feature detectors is their requirement of a repository of all possible images of objects existing in a scenario for identification, because they lack the ability to describe any feature of an object that they have not seen before. Qualitative approaches for object description in digital images are successful at identifying simple objects, but they may be ambiguous when detecting complex objects in the real world, since these approaches use abstractions of features which sometimes may produce too general categorizations [9], [10], but which can be complemented with spatial features (i.e. topology, distance, direction, location, etc.) for disambiguation. Advantages of qualitative descriptors are that they can be applied to: (i) describe objects which are ‘unknown’ by the system (i.e. not stored in memory, not seen before in a scenario) and (ii) identify them by matching their features without any previous training. Qualitative representations have also been successfully employed in querying spatial databases [11]. The main aim here is to analyse if they can enhance cognitive image/scene descriptions with reasoning, since qualitative relations have already been acknowledged for their contribution in object recognition [12], [13] and qualitative image description (QID) [14].

In the literature, QID has been applied to extract qualitative features from real-world scenarios such as (i) images captured by the webcam of a mobile robot [15], integrated also with qualitative distances for representing indoor scenes in robotics [16], and (ii) images of tiles captured by a camera located on a robotic arm used to detect tile pieces and assemble mosaics [9]. The QID description was designed for recognition by matching, then QID-Ontology [10], [17] was designed for formalizing the meaning of the representation and for categorizing objects by their qualitative shape and colour using description logics. In this paper, the QID approach is extended and redesigned for enhancing its reasoning capabilities by obtaining a qualitative image description in first order logics (QIDL) of shape, colour, topology and location of the objects in scenes captured by digital images, which are integrated with domain knowledge for scene understanding in an ambient intelligent system scenario, the Interact@Cartesium at the Universität Bremen.

The rest of the paper is organized as follows. In Section 2, the related work in the literature is presented and discussed. In Section 3, the approach for obtaining a qualitative image description in first order logics (QIDL) in Prolog [18] is outlined. Section 4 describes how the qualitative descriptors of shape, colour, topology and location are extracted. Section 5 presents the facts in first order logic which are obtained from the qualitative descriptors. Section 6 explains the domain knowledge provided to the system and Section 7 presents the logics defined. In Section 8, tests and results are presented. Section 9 discusses the results and finally, conclusions and future work are drawn in Section 10.