Elsevier

Applied Soft Computing

Volume 24, November 2014, Pages 948-961
Applied Soft Computing

Dynamic Fuzzy Rating Tracker (DYFRAT): a novel methodology for modeling real-time dynamic cognitive processes in rating scales

https://doi.org/10.1016/j.asoc.2014.08.049Get rights and content

Highlights

  • The article presents an original methodology for measuring fuzziness in human rating data.

  • It avoids traditional problems of standard rating scales and other types of fuzzy scales.

  • It models fuzziness by measuring some real-time biometric information during the cognitive process of rating.

  • Some real applications support the usefulness of the proposed novel methodology.

Abstract

Rating scales (such as, Likert scales, Guttman scales, Feelings thermometers, etc.) represent simple tools for measuring attitudes, judgements and subjective preferences in human rating contexts. Because rating scales show some useful properties (e.g., measurement uniformity, considerable flexibility, statistically appealing), they represent popular and reliable instruments in socio-behavioral sciences. However, standard rating scales suffer also from some relevant limitations. For example, they fail in measuring vague and imprecise information and, above all, they are only able to capture the final outcome of the cognitive process of rating (i.e., the rater's response). To overcome these limitations, some fuzzy versions of these scales (e.g., fuzzy conversion scales, fuzzy rating scales) have been proposed over the years. However, also these more sophisticated scales show some important shortcomings (e.g., difficulty in fuzzy variables construction and potential lack of ecological validity). In this paper, we propose a novel methodology (DYFRAT) for modeling human rating evaluations from a fuzzy-set perspective. In particular, DYFRAT captures the fuzziness of human ratings by modeling some real-time biometric events that occur during the cognitive process of rating in an ecological measurement setting. Moreover, in order to show some important characteristics of the proposed methodology, we apply DYFRAT to some empirical rating situations concerning decision making and risk assessment scenarios.

Introduction

In many empirical research areas such as, for instance, psychology, sociology, organizational and management sciences, marketing, and epidemiology, rating scales represent a widely used, simple and flexible tool for measuring attitudes, opinions, and subjective preferences [1], [2], [3], [4]. Let us assume that we are interested in measuring a person's degree of happiness. We could do this in a number of different ways, but one direct and efficient approach would be simply to ask the person, ‘How happy are you?’ and require them evaluate themselves on a Likert-type rating scale, ranging from ‘very unhappy’ to ‘very happy’. Rating scales typically consists of a variable to be measured (e.g., ‘happiness’) and a set of anchor points from which the rater selects the most appropriate description (e.g., very unhappy, moderately unhappy, neither, moderately happy, very happy). One widely used type of rating scale is the so-called numerical scale, where the anchor points either explicitly or implicitly are defined numerically (e.g., 1: low, 2: average, 3: high). Like checklists, rating scales are used for a wide variety of assessment purposes. For example, rating scales can be used to have one individual evaluate another, for example, when a physician might asses a patient as to degree of obesity, but the rating scales can also be applied as self-report measures. Unlike other types of ratings, self-report scales require the person provide a direct and explicit rating of their own behavior/opinion/preference, etc. Of course, the main assumption behind self-report measures is that individuals are in the best position to report their own opinion in a direct and transparent way. The great diffusion and success of rating scales are mainly due to the following major reasons that are all well documented in the literature: (1) rating scales can be administered to large groups of respondents at one single setting; (2) they can be administered under conditions that guarantee anonymity; (3) they allow the rater to proceed at their own pace; (4) they present uniformity of procedure; (5) they allow for great flexibility – for example, take-home questionnaires; and (6) the results are more amenable to statistical analyses (in particular for numerical scales) [5].

However, over the years several criticisms have been arisen against the empirical validity of rating scales. For example, because of the discrete and crisp nature of rating scales, some raters tend to avoid extreme categories in the anchor points (central tendency or restriction of range problem) [6]. In general, we may think of human opinions and subjective preferences as being characterized by more vague and imprecise information than the ones actually described by standard rating scales. Moreover, in some circumstances, the honesty assumption implicitly assumed in self-report rating administrations appears to be simply unrealistic. So, for example, in personnel selection surveys some candidates might not be so candid in admitting their imperfections and, therefore, the associated self-report evaluations could be definitively biased (e.g., [7]). Finally, another important limitation pertains the idea that what is being reported using rating evaluations is something that allows us to objectively explain and describe a person's behavior. However, several psychologists argue that such assumption is, in principle, ill-posed and that what we really need, instead, is to directly focus on the observable behavior involved in the process of rating (e.g., [8]).

In order to overcome the limitations of standard rating scales, some researchers have applied fuzzy set theory (FST) to directly modeling imprecise features of human rating evaluations [9], [10]. In particular, in the rating scale problem, FST has been mainly used in two different contexts: (i) for modeling data obtained by means of standard rating scales (fuzzy conversion scales) or (ii) for directly quantifying empirical evaluations using fuzzy rating scales. In the first case, FST is applied a-posteriori as a procedure for converting standard rating data into fuzzy data (i.e., raters express their judgements using a traditional rating scale which is subsequently converted into a fuzzy structure). By contrast, in the second case, FST is applied a-priori as a general interface for directly capturing fuzzy rating data (i.e. raters give their evaluations by means of computerized tools that allow to directly use fuzzy sets in place of crisp numbers).

In line with these approaches, in the present contribution we propose a novel methodology, called DYFRAT (Dynamic Fuzzy Rating Tracker), to measure some relevant behavioral dynamics involved in the rating process. Likewise fuzzy rating scales and fuzzy conversion scales, also DYFRAT represents human rating evaluations in terms of fuzzy sets. However, unlike fuzzy conversion scales and fuzzy rating scale, DYFRAT captures (in addition) some physical and biometric characteristics underlying observable behaviors involved in the process of rating and considers fuzziness as a natural property that spontaneously arises from the observed data. In this respect, DYFRAT is a formal procedure that explicitly focuses on the behavioral dynamics of rating and provides a continuous on-line measure of the cognitive aspects involved in this process.

The remainder of this article is organized as follows. In the second section we describe a comprehensive survey of the state-of-the-art of FST applications in human rating problems. In the third and fourth sections we present our new methodology. In the fifth section we describe a first computerized implementation of DYFRAT system. In the sixth section we show some empirical applications of DYFRAT to real data, whereas in the seventh section we conclude this article by providing some final comments.

Section snippets

Fuzzy conversion scales

Fuzzy conversion scales (FCS) are computational procedures based on a fuzzy system which convert standard crisp rating data into a set of fuzzy data. Fig. 1a shows a graphical representation of the rationale underlying the FCS approach. In general, two perspectives can be adopted to derive the conversion scale procedure from crisp rating data: (i) an expert-based approach and (ii) an empirical-based approach. In the first perspective, a researcher (a-priori) sets the main features of the fuzzy

Dynamic Fuzzy Rating Tracker: theory

The traditional rating scale paradigm often regards human rating as a discrete-stage process in which the final response represents its final stage. Unfortunately, the observed final response captures only the outcome of the rating process while the real-time cognitive dynamics that occur during this process are lost. To overcome this relevant limitation, we propose a new methodology (DYFRAT) which is designed to track real-time mental processes by using the so-called Mouse Tracking Methodology

Dynamic Fuzzy Rating Tracker: methodology

DYFRAT consists of a data-capturing procedure which implements a MTM based computerized interface for collecting the motor and temporal components in the process of rating and a data-modeling procedure which provides a fuzzy model for the recorded information.

Dynamic Fuzzy Rating Tracker: implementation

Data-capturing procedure: The first application consists in an executable stand-alone package available for Windows, OSX and Unix systems developed in Processing 2.0 (http://processing.org). All the main features of the DYFRAT graphical interface can be modified by the user (e.g., temporal delay between-items or within-items, labels positions, font type and text size, scale diameter, scale stroke, breakpoint width, mouse-movements sample-rate). In particular, label positions can be set to

Illustrative examples

By way of illustration we consider three simple applications using the DYFRAT methodology. The first example is about the evaluation of a well known cognitive problem in decision making. The second application considers data about rash driving behaviors among young people aged 18–26. Finally, the third application illustrates how one can perform an outlier detection analysis using the DYFRAT framework.

Final remarks

In this paper we proposed a novel methodology (DYFRAT) for measuring the fuzziness of human rating situations from an original perspective. By considering human rating as a temporal and dynamic changing course of information in which the final rater's response is only the outcome of peculiar latent cognitive processes, we modeled fuzziness as the result of the integration between two important physical/biometric measures. We adopted the mouse tracking methodology for capturing the motor and

Antonio Calcagní received his BSc (2010) and MSc (2012) in psychology from the University of Salento (Italy). Now, he is predoctoral research fellow in psychometrics at the Department of Psychology and Cognitive Science, University of Trento (Italy). He is student member of European Society for Fuzzy Logic and Applications (EUSFLAT) and Psychometric Society. His current research interests concern the application of fuzzy set theory to psychometrics and measurement problems in psychology, the

References (67)

  • A. Furnham et al.

    The good, the bad and the mad: response bias in self-report measures

    Personal. Individ. Diff.

    (1982)
  • H.-D. Cheng et al.

    Automatically determine the membership function based on the maximum entropy principle

    Inform. Sci.

    (1997)
  • L. Li et al.

    Fuzzy entropy image segmentation based on particle swarm optimization

    Prog. Nat. Sci.

    (2008)
  • A. De Luca et al.

    A definition of a nonprobabilistic entropy in the setting of fuzzy sets theory

    Inform. Control

    (1972)
  • R. Coppi et al.

    The fuzzy approach to statistical analysis

    Comput. Stat. Data Anal.

    (2006)
  • V.-N. Huynh et al.

    A parametric representation of linguistic hedges in zadehs fuzzy logic

    Int. J. Approx. Reason.

    (2002)
  • J. Greene et al.

    How (and where) does moral judgment work?

    Trends Cogn. Sci.

    (2002)
  • J. McGuire et al.

    A reanalysis of the personal/impersonal distinction in moral psychology research

    J. Exp. Soc. Psychol.

    (2009)
  • S. Nichols et al.

    Moral dilemmas and moral rules

    Cognition

    (2006)
  • J.J. Arnett et al.

    Reckless driving in adolescence: state and trait factors

    Accid. Anal. Prev.

    (1997)
  • B.A. Jonah

    Accident risk and risk-taking behaviour among young drivers

    Accid. Anal. Prev.

    (1986)
  • P. Finn et al.

    Perception of the risk of an accident by young and older drivers

    Accid. Anal. Prev.

    (1986)
  • D.M. DeJoy

    An examination of gender differences in traffic accident risk perception

    Accid. Anal. Prev.

    (1992)
  • H.A. Deery

    Hazard and risk perception among young novice drivers

    J. Saf. Res.

    (2000)
  • H.-T. Chung et al.

    A resolution-based system for symbolic approximate reasoning

    Int. J. Approx. Reason.

    (1995)
  • R. Göb et al.

    Ordinal methodology in the analysis of likert scales

    Quality Quantity

    (2007)
  • D.C. Miller et al.

    Handbook of Research Design and Social Measurement

    (2002)
  • L.R. Aiken

    Rating Scales and Checklists: Evaluating Behavior, Personality, and Attitudes

    (1996)
  • F.A. Pettit

    A comparison of world-wide web and paper-and-pencil personality questionnaires, Behavior Research Methods

    Instrum. Comp.

    (2002)
  • G.E. Domino et al.

    Psychological Testing: An Introduction

    (2006)
  • F.E. Saal et al.

    Rating the ratings: assessing the psychometrics quality of rating data

    Psychol. Bull.

    (1980)
  • M.R. Golfried et al.

    Traditional versus behavioral personality assessment: A comparison of methodological and theoretical assumptions.

    Psychol. Bull.

    (1972)
  • S.d.l.R. de Sáa et al.

    Fuzzy rating vs. fuzzy conversion scales: an empirical comparison through the MSE

    Synergies of Soft Computing and Statistics for Intelligent Data Analysis

    (2013)
  • Cited by (19)

    • A psychometric modeling approach to fuzzy rating data

      2022, Fuzzy Sets and Systems
      Citation Excerpt :

      There have been numerous attempts to make rating scales more sensitive to detecting components of response process such as decision uncertainty. Generally, there are three types of solutions to this problem, a first one involving the use of additional measures like response times or latencies along with standard multiple choice questions [38,74,112], a second one using extended item response theory (IRT) models on standard rating data [10,37,60], and a third one involving the use of alternative rating instruments such as those based on tracing methodology [95] and fuzzy rating scales [15,94,46]. Since the seminal work of [46], the latter has become popular only in recent years.

    • Descriptive analysis of responses to items in questionnaires. Why not using a fuzzy rating scale?

      2016, Information Sciences
      Citation Excerpt :

      ▲ labels for the given ‘values’ are expressed in terms that properly fit the intrinsic imprecision of the considered characteristics or attributes. Nevertheless, several weaknesses have been highlighted in the literature on this type of scale (see, for instance, Jamieson [31], Carifio and Perla [9], Calcagnì and Lombardi [8]), namely, ▼ the number of possible ‘values’ to choose among is small, so the variability, diversity and subjectivity associated with an accurate rating is usually lost, and the choice of the most appropriate ‘value’ is not necessarily a simple task (raters often prefer to have the opportunity of choosing in between two ‘values’);

    • Estimating Latent Linear Correlations from Fuzzy Frequency Tables

      2022, Communications in Mathematics and Statistics
    View all citing articles on Scopus

    Antonio Calcagní received his BSc (2010) and MSc (2012) in psychology from the University of Salento (Italy). Now, he is predoctoral research fellow in psychometrics at the Department of Psychology and Cognitive Science, University of Trento (Italy). He is student member of European Society for Fuzzy Logic and Applications (EUSFLAT) and Psychometric Society. His current research interests concern the application of fuzzy set theory to psychometrics and measurement problems in psychology, the application of fuzzy statistics in real problems, the development of novel measurement approaches for modeling uncertainty in cognitive contexts, the application of information theory to statistical estimation processes.

    Luigi Lombardi received his PhD in cognitive science at the University of Padua (Italy). He is currently associate professor of psychometrics at the Department of Psychology and Cognitive Science, University of Trento (Italy). His main research stream focuses on interrelated issues dealing with multivariate data analysis of discrete variables and formal models of higher level cognition, such as decision strategies, induction, similarity evaluation, and classification.

    View full text