Measuring design-level information quality in online reviews

Highlights

• A model to estimate the design contextual information in online reviews is presented.

• Links extracted nouns to ensure the relevancy of design features.

• The model evaluates three information components: content, complexity, and relevancy.

• The results confirmed that a lot of useful product design information exists in online reviews.

Abstract

Online product reviews are an important type of user-generated content. For product designers, they offer valuable information that identifies consumer likes, dislikes and desires. We investigate the volume and quality of product design information available in online reviews and introduce the design-level information quality (DLIQ) measure. DLIQ is indicative of the design contextual information stored in the online reviews for a given product. Three separate information components are evaluated: content, complexity, and relevancy. Key determinants of DLIQ are the number of reviews, sentences, words, noun words and feature matching noun words in a review database. DLIQ is formulated as an index and indicates information content relative to a sample of products. For a sample of ten products, RapidMiner was used to mine and illustrate the DLIQ. A validation test with a survey group confirms DLIQ is a reliable predictor of actionable design information in the review database.

Introduction

Online product reviews are an increasingly important type of user generated online content. For product designers they offer potentially valuable information that could identify consumer likes, dislikes and desires. Today, hundreds of consumer reviews for a wide range of products and services are recorded and publicly available on retailer, manufacturer and independent review websites. The reviews allow consumers to share their product and service experiences and are used by buyers to make purchase decisions (Anwer et al., 2010). Online reviews have rapidly evolved into an influential peer-to-peer opinion sharing platform and are now a key determinant of success in the online marketplace. It is therefore important for product designers to investigate these online reviews and integrate the embedded knowledge into the design process. This integration will allow designers to efficiently incorporate consumer trends, needs, and/or opinions into the design review process. Presently, this integration is very limited primarily because designers are unable to quantify the knowledge potential and thus justify the investigation. Further, the vast amount of review information and its widespread dissemination make it challenging for designers to collect the relevant information, interpret and summarize them, and organize them into a usable format. Traditionally, designers have used focus groups, interviews, surveys and customer service data to collect consumer feedback and identify exciting features and disappointments. Similar information is extractable from online reviews, and the feasibility has been reported in the literature (Yu et al., 2012, Yagci and Das, 2015, Yagci, 2014, Buyukozkana et al., 2007, Liu et al., 2013). Additionally, the reason for the review sentiment can be reliably identified, providing additional value to designers (Lee and Kim, 2017).

Yagci and Das (2015) showed how online review data is introducing large volumes of unstructured consumer experience information into the product knowledge space. In a survey of design engineers, they found significant consensus that design intelligence extracted from online reviews is complementary and different from that obtained via traditional methods. In many instances this knowledge may contradict or be unknown to designers. Frequently, though, designers are skeptical of the utility and validity of the online review knowledge. The free text nature of online reviews is not readily interpreted by designers, and the knowledge extraction process is often not obvious. Ghose and Ipeirotis (2007) observe that manufacturers are challenged to understand the true underlying quality information in online reviews. Further, Liu et al. (2013) caution that simply using the scaled evaluations of online reviews could mislead designers in identifying truly valuable and insightful opinions from a designers’ perspective. There is therefore a need to quantify the available design information in online reviews on a standard scale. Design teams can then more reliably estimate the likelihood that design contextual information will be successfully retrieved. Designers would be motivated to overcome their skepticism, and appropriately allocate tactical resources to the online review knowledge analysis effort.

Presently, online reviews are used by two groups, first by product buyers to investigate the opinions of other buyers, and second by marketing departments to evaluate consumer sentiment (positive to negative). Online reviews are a form of consumer intelligence, often including product use knowledge that is unknown to product designers and manufacturers. There is therefore a need to extend the utility of online reviews into the product designer group, and this research is motivated by the need for tools and models which facilitate this extension. Traditional design processes are focused on technology driven innovation, but addition of a consumer driven innovation component can lead to better and more successful products. Online reviews provide a conduit for this innovation by providing immediate and free access to feedback from hundreds, and possibly thousands, of customers. Liu et al. (2013) found that an efficient and timely design review process, with a focus on customer needs, is critical to market success. Our approach to access the consumer intelligence embedded in online reviews, is to mine the reviews for specific product intelligence that can lead to better designs. For example one may learn from the reviews that the image stabilization feature in a digital camera malfunctions in humid conditions. Frequently, such issues are not easily identified by consumer focus groups due to small group sizes and short study periods (Smithson, 2000, Button et al., 2013). Wang et al. (2018) have emphasized that online product reviews provide a good and reliable channel for not only understanding customer product needs but also analyzing a products’ competition in the market.

Yagci and Das (2015) introduced a method to efficiently extract or mine design intelligence from unstructured web reviews. The extraction process is complex and time consuming, and there is no guarantee their method will generate meaningful intelligence. Ideally then, an evaluation of the intelligence likely to be mined can justify the effort and motivate designers to integrate the method in the design process. The primary contribution of this research is the development of a quantitative measure which evaluates the design level intelligence likely to be mined from a given set of online reviews. The evaluation investigates the quantity and quality of the mined intelligence and proposes the design-level information quality (DLIQ) measure. DLIQ is indicative of the design contextual information that is stored in the online reviews for a given product, and likely to be efficiently extracted by the application of a mining method. The measure identifies product features mentioned in the online reviews and then evaluates the information attributes to assess quantity and quality (i.e., number of words, sentences, product features, etc.). The DLIQ measure contributes to the extension of web review intelligence in to the product design domain, by confirming the information value of the reviews in the context of product features. The measure supports the adoption and further development of web review intelligence mining methods.

Design information is referenced by features and feature-based opinion mining is widely reported in the literature. Feature based opinion mining can be performed at both the sentence or document level, and typically follows three main subtasks: identifying product features, identifying opinions (e.g., positive or negative sentiment) regarding the product features, and summarizing the results across features and opinions. Consider the following review of an automobile model: “High quality brakes, stops quickly even when road is wet. Since the vehicle has a low front bumper, in time they all get easily damaged, especially in driveways and bumps.” In this review, brake and front-bumper are two product features mentioned; and high quality and damaged are the associated opinion words. In the first case the sentiment is positive, while in the second it is negative.

In general, opinion mining research can be divided into two groups. the first focuses on opinion information while the second focuses on feature information (Binali et al., 2009, Li and Yang, 2005). Recently, an increasing number of researchers have proposed different methods to solve opinion mining problems and proposed different opinion-oriented information seeking systems (Hatzivassiloglou and Wiebe, 2000, Li, 2010, Hu and Liu, 2004a, Hu and Liu, 2004b, Anwer et al., 2010, Koen et al., 2002, Ding et al., 2008, Chen et al., 2015, Lee et al., 2017).

Turney (2002) introduced a semantic orientation approach, applying Web-based point-wise, mutual information statistics to determine review-polarity. Information content was calculated using hit counts from the reviews of multiple domains. Lee and Kim (2017) offered a semi-supervised sentiment labeling approach that adds predicted data to the training corpus in order to improve the initial classifier in self-training. Many researchers have presented research concerning sentiment orientation of opinion words at different granularity levels such as words, sentences, and entities in online reviews (Hu and Liu, 2004a, Hu and Liu, 2004b, Park and Lee, 2011, Mudambi and Schuff, 2010, Chechik et al., 2005, Lee and Kim, 2017, Lee et al., 2017). However, this information is not always useful to product designers who are looking to initiate design changes motivated by the opinions.

In addition, there has been limited research on measuring the design level value of online reviews. Feature based opinion mining is the starting point for such a measure, and recently there has been much reported activity. Zhen et al. (2014) identified opinion features by exploiting the difference in feature statistics across domain-specific corpus and domain-independent corpus. Jin and Liu (2010) introduced the helpfulness prediction technique, which focused on how to connect customer reviews to product designer ratings. They applied a regression algorithm to predict the online review’s helpfulness. The system has two phases. First, it creates the connection between the customer review and the designer rating with the help of the training set. Second, it extracts features from the reviews to aid in predicting helpfulness. This includes linguistic features, product features, and information quality (accuracy, timeliness, comparability, coverage, and relevance).

Park and Lee (2011) focused on how to design and utilize an online customer center in an effort to support new product concept generation. They introduced the decision support system that identifies customer needs and materializes them to develop R&D targets in the new product development process. Their system consists of four stages: extracting consumer reviews from the website, extracting a keywords list of entire documents and their frequencies, classifying the keywords into several groups based on customers’ expressed needs, applying the k-means clustering algorithm, and mapping customer needs with product specifications. A good review of feature-based opinion mining method is by Liu (2012). Further, they defined a quintuple model to describe aspects in a document. A method to extract the required aspects is then presented by Hu and Liu, 2004a, Hu and Liu, 2004b.

Other related work includes the design-feature-opinion-cause (DFOC) method by Yagci and Das (2015). DFOC integrates the evaluation of unstructured online reviews into the structured product design process. The key data element in DFOC is an online review and its associated opinion polarity. The DFOC method: (1) identifies a set of design features that are of interest to the product design community; (2) mines the online review database to identify, which features are of significance to customer evaluations; (3) extracts and estimates the sentiment or opinion of the set of significant features; and (4) identifies the likely design cause of the opinion.

The literature indicates the dominant theme in information measurement is entropy. Entropy is a measure of the uncertainty associated with a random variable. In this context, the term usually refers to Shannon entropy (Shannon, 1948), which quantifies the expected value of the information contained in a message, usually in units such as bits. It defines information as a purely quantitative measure of communication exchanges. Entropy assigns a score of uncertainty to a stochastic variable (Banerjee, 2007). Shannon defines entropy with a discrete random variable X, with possible states x₁,x₂, …x_n as:

$$H\left(X\right)=\sum _{i=1}^{n}p\left(x_i\right){log}_2\left(\frac{1}{p(x_i)}\right)=-\sum _{i=1}^{n}p\left(x_i\right){log}_{2}p(x_i)$$

where $p\left(x_i\right)=\text{Pr}(X-x_i)$ is the probability of the ith outcome of X. This formula implies that the more entropy a system has, the more information we can potentially gain once we know the outcome of the experiment. Maji and Pal (2010) used Shannon's entropy approach to measure the relevance and redundancy of features. Lee et al. (2017) adopted the entropy theory for their research to measure the ratio of the review text.

We investigate what amount of design-level-information is available in online reviews from the perspective of product designers. To develop the DLIQ measure, a Shannon entropy approach and the principle of maximum entropy were adopted. From the above equation, we know that entropy H(X) is a function of p(x_i). Maximum entropy occurs when all states are equally probable: p(x_i) = 1/x_i for all values of x_i in the range 0–1, therefore $\sum _{i=1}^{n}p\left(x_i\right)=1$. Substituting from H(X) = 1 · log₂ p(x_i) = log₂ p(x_i). Similar to other approaches by Burch and Gull, 1983, Wu, 2003, in the DLIQ measure p(x_i) represents the frequency of key parameters, specifically nouns, in the extracted review database.