Our contributions

We use penalized logistic regression to uncover factors associated with users’ decisions to vote for or accept answers on all Stack Exchange communities. To partly control for heterogeneity, we split data by community type and the number of answers each voter sees, and leave out the largest community in our training data to check the robustness of results. By community type we mean technical, programming-related communities (e.g., Ubuntu), non-technical (e.g., cooking), and meta communities that discuss aspects of a particular board itself rather than a particular topic. Parsing data in this way reveals significant differences in the importance of particular attributes, and the predictability of user behavior.

We find that only a few answer attributes are strongly associated with users’ choices, such as the order in which the answer appears and whether it has been accepted by the asker. Users appear to rely on simple heuristics to choose an answer based on its rank, how much screen space it occupies, or whether it was approved by others. These heuristics may be useful proxies for answer quality, but our work suggests otherwise. For example, voters are more likely to choose an answer after it has been accepted than before. Although answer acceptance is often viewed as a standard of answer quality [13–15], the only discernable difference in an answer after acceptance is a signal that the asker chose this answer, suggesting users view acceptance as a useful signal of quality, but are less able to discern that quality on their own.

Moreover, as the number of answers to a question grows, the importance of these attributes in describing behavior increases. Two different explanations are feasible as to why the number of answers a user sees affects their behavior. First, as the number of available answers grow, users may become less willing to thoroughly evaluate all answers, instead increasingly relying on heuristics when choosing an answer to vote for or accept. A similar effect exists in other domains. For instance, information overload impacts consumer’s choice of products [16] and the spread of information in online social networks [17, 18]. An alternative explanation is that later voters are different and happen to rely more on heuristics compared to people who vote early. This view is supported by the observation that users who answer early in a question’s life cycle on Stack Overflow, a programming-related community on Stack Exchange, have a higher reputation than users who answer later [11]; therefore, time acts as a potential source of heterogeneity. In either case, the finding that voters appear to rely on heuristics to a greater extent as the number of answers grows suggests limits of the “wisdom of crowds” on Stack Exchange: a crowd’s judgments become less reliable as proxies of quality as answers to a question accumulate.

The rest of the paper is as follows. In the related work section, we review work related to our current analysis, while, in the materials and methods section, we discuss our data and ways in which we analyze it. Next, in the results section, we discuss our main findings. Finally, in the conclusion section, we review our findings, discuss future work, and discuss ways to improve upon question answering sites.

Related work

Collective decisions of a group have been claimed to be superior to those of individuals, even better-informed individuals, when the group is large [3, 7], i.e., a crowd. The advantage arises from the group’s ability to aggregate diverse information and thereby, reduce the error [19]. This effect, coined “the wisdom of crowds” [4] is the basis of many online crowdsourcing applications, including question answering [4–6].

Prior research on Q&A sites has shown that a variety of attributes can provide useful insights into content quality [15, 20–22]. For example, Kim and Oh [14] examined how users evaluate information in Yahoo! Answers forums, by examining the comments askers leave on answers. They found socioemotional-, content-, and utility-related criteria are dominant in the choice of the best answer, and found users evaluate information based not only upon the content, but also on cognitive and collaborative aspects. Adamic et al. [23] similarly conducted a large-scale network analysis of Yahoo! Answers forums and found that, for both technical and non-technical sites, answer length and the number of other answers the asker has to choose from are the most significant features to predict the future best answer. One limitation in these previous studies, however, is in assuming that the answer an asker chose was the “best” answer, and did not correct for asker biases, such as user preference towards choosing items at the top of a list, when choosing an answer, which we find can explain user behavior increasingly well.

Several authors [13–15] use logistic regression to determine which attributes best describe high quality answers, and often assume that a “high quality” answer is one an asker accepts, which our work casts doubt on. Other works have examined the impact of speedy answerers on answer quality. Anderson et al. [11] found that early answers in Stack Overflow (the Stack Exchange community that deals with programming questions) tend to be posted by expert users with higher reputation, and subsequent answers come from lower reputation users. While the first answer tends to be more appreciated by the asker, the longer a question goes unanswered, the less likely that an answer will eventually be accepted. Similarly, Rechavi and Rafaeli [24] concluded that askers use response time as a parameter at evaluation time. However, this hypothesis was refuted in other works. Shah [25] analyzed the responsiveness in Yahoo! Answers forums, finding that more than 90% of the questions receive an answer within an hour. However, satisfactory answers may take longer, depending on the difficulty of the question. Interestingly, our work suggests that answer age and chronological order are not particularly important attributes for askers or voters, possibly because high reputation answerers do not strongly affect whether an answer gets voted on or accepted.

In contrast to many previous studies, our paper examines how user-voting patterns may be affected by various answer attributes. This is an important area of study, because people often use votes as a signal of the best answer to a particular problem. A previous paper deduced a set of possible factors that indicate bias in user voting behavior [26]. They provided a method to calibrate the votes inside Q&A sites, based in part on the average value of the answer and the average vote received in the answerer history. This type of calibration is useful to restrict the effects of users who are trying to game the system, or to signal the reputation of answerers. Our work, however, answers a different set of questions: we want to find the role heuristics play in answer evaluation, how voter and asker behaviors differ, and what drives heterogeneity within voter and asker populations. The role of heuristics in human decisions has been studied by behavioral economics [27, 28], but, to the best of our knowledge, our work is the first that investigates the potential impact of heuristics on the performance of crowdsourcing systems.

2.1 Taming heterogeneity

Automatically uncovering homogeneous populations within heterogeneous observational data remains an open research challenge. In our study of Stack Exchange, we use exploratory data analysis to identify potential sources of heterogeneity. For example, users who are interested in technical topics (e.g., programming) may be driven by different factors to contribute to Stack Exchange than those who are interested in non-technical subjects (e.g., cooking), or governance (meta boards). To account for this source of heterogeneity, we split the data by the type of board—technical, non-technical and meta—and run regression analysis separately on each dataset. We further split data by whether the asker eventually accepted an answer in our observation window, how an answer is chosen (vote versus accept), and the number of answers, but find that the greatest source of heterogeneity is the number of answers a question has at the time the user votes for or accepts it.

2.2 Logistic regression

Logistic regression is a common model from choice theory for binary decisions [29], because it can be written in the following way: (1) where (2) U can be interpreted as the utility of a choice, ϵ is a noise parameter to reflect choices being non-deterministic, β are the regression coefficients, and X are the answer attributes. Logistic regression can be interpreted as users tending to choose an option if its utility is greater than some threshold, where the probability to choose the option is moderated by noise. We are motivated to use logistic regression to understand the factors driving users to vote for or accept an answer on Stack Exchange, because of its simple theoretical interpretation, although other models, such as random forests [30], could be used as well. Whether other models match data better than the current one is left for future study. The logistic model may still seem an unusual choice, because it assumes users make binary choices, while, in reality, users choose one of several answers available, and therefore could be modeled using multinomial logistic regression. However, the effect the number of answers has on how askers and voters vote in our data seems to directly contradict the independence of irrelevant alternatives (IIA) assumption built into multinomial regression. Although other multi-class models do not depend on the IIA assumption (e.g., [31]), they can be very slow, and it can be difficult to avoid over-fitting without a penalized regression model that we describe below. For all these reasons, logistic regression is the most appropriate model.

Our data is highly multi-dimensional, therefore we use LASSO penalized regression, where parameters are determined by maximizing the likelihood function with the addition of a penalty, to avoid overfitting [32]. The value of this penalty is adjusted to minimize the mean error from ten-fold cross-validation (CV). As a check, we repeated the fits with a different type of penalty—ridge regression—and found the behavior to be qualitatively the same (S1 Fig). The fitting was performed with the R package “glmnet” [33], which allows for fast and accurate determination of regression coefficients, β.

We check the robustness of results by omitting data from the largest community for each board type (technical, non-technical, and meta), and then re-evaluate the regression parameters. The qualitative results were unaffected, and quantitatively, the results were very similar (S2 Fig). For the rest of the paper, we focus on LASSO penalized regressions with all boards included.

2.3 Error

The uncertainty in β (shaded regions in subsequent figures) is found by changing the LASSO penalty term, λ, such that (3) where D, the error term in our regressions, is the deviance: (4) where is the log-likelihood of the saturated model, with one degree of freedom per observation, while is the log-likelihood for the fitted model. is the minimum mean deviance, based on ten-fold cross-validation, and σ_min(D) is the standard deviation of this minimum. LASSO regression, like all penalized regression methods, does not have a standard method to calculate uncertainties with high dimensional data [34], therefore we use this method as a reasonable method to extract uncertainties in regression coefficients.

In contrast, the uncertainty in the area under the receiver operating characteristic (ROC) curves are calculated by bootstrapping the pairs of predicted probabilities and outcomes, calculating seperate ROC curve, and the associated area under the curves (AUC). We then use error propogation to determine the AUC error bars when averaged over the number of answers users see. The ROC curve represents the best true positive rate our model can achieve for a given false positive rate, as we vary the discrimination threshold. The AUC, on the other hand, can be interpreted as the average true positive rate across all discrimination thresholds, and is equivalent to the Wilcoxon rank test [35].

2.4 Attributes and normalization

We use the following answer attributes in the regression:

1. answerer’s reputation at the time the answer was created,
2. mean rate of reputation increase over time,
3. answer’s Flesch Reading Ease [36], or readability, score,
4. answerer’s tenure (i.e., time since joining the site) at the time of the answer,
5. number of hyperlinks per answer,
6. binary value denoting whether the answer was eventually accepted (for voting only),
7. answer score before each vote,
8. default web page order for an answer (i.e., its relative position),
9. chronological order of an answer (whether it was first, second, third, etc.),
10. time since an answer was created, or its age
11. number of words per answer,
12. answer’s word share, that is the fraction of total words in all answers to the question.

Answerer reputation [13], Flesch readability, and word count [37] were used in previous works as measures of answer quality. We also consider an answer’s rank in the list of answers (what we refer to as web page order) and score, because these variables affect how much attention the answer receives [9, 38, 39], and we empirically find answers are voted on more once accepted (therefore any questions without accepted answers were removed from the study). The other attributes listed above were also examined as additional factors that could affect how answers are voted or accepted. These were, however, not found to significantly affect our results.

There is large variability in attribute values within and across datasets. To account for the variability, we normalize all attributes except the web page order and chronological order by mapping them to their associated cumulative distribution function (CDF). CDF normalization is a non-parametric way to reduce the effect of outliers because values are evenly spread between 0 and 1. To test this, we created simulated data with varying distributions and found that CDF normalization creates less variability of regression estimates than normalizing by the standard deviation (, where is the estimated mean value of the attribute, and is the estimated variance), see S3 Fig. Normalization also allows us to compare the relative importance of different attributes via their logistic regression coefficients. We interpret each regression coefficient as the relative effect an increase of an attribute, X, by 1% has on the probability to choose that answer. For the web page order and chronological order attributes, we divided by the number of answers available, which is equivalent to a CDF when controlling for the number of answers visible.

When selecting attributes, we made sure correlations with other attributes were reasonably low, and, if they were greater than 0.7, we checked whether removal of the attribute increased the CV error, or reduced the AUC on the tested dataset significantly (see next section). This correlation condition seems very liberal, but we wanted to include as many attributes used in previous literature as possible. To check if this affected our results, we separately removed wordshare, score, whether the answer was eventially accepted, and webpage order, and found results were qualitatively the same.

3.1 Answer attributes and behavior

We take logistic regressions for votes cast before any answer was accepted, votes after an answer was accepted, as well as choices to accept answers. The average and variance of the regression parameters across 2 to 20 answers are shown in Fig 2. Because all attributes were normalized, the larger the value, the more a change in the respective parameter correlates to a change in user behavior, relative to all others in the regression.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Relative size of regression parameters for answer acceptance and voting.

Regression coefficients for answerers to accept (green circles) and voters to vote for an answer both before (red triangles) and after (blue squares) an answer is accepted on (a) technical, (b) non-technical, and (c) meta boards, averaged over the number of available answers from two to twenty. Higher values indicate a stronger relationship between attributes and user behavior (voting or accepting an answer). Error bars indicate the variance of the best-fit values values across two to twenty answers.

https://doi.org/10.1371/journal.pone.0173610.g002

We find that web page order and word share have the highest regression coefficients (Fig 2). These findings alone are not necessarily surprising. We know from previous research that people’s choices are biased by the rank order of items [38–40]. Word share is potentially correlated with higher answer quality, because relatively long answers may be more informative, or they may just be easier to see (take up a large portion of screen space). We notice that both of these regression coefficients are even higher for askers than voters, across different board types, already suggesting a surprising degree of heterogeneity. Other factors, however, such as an answerer’s reputation or tenure, how thoroughly an answer is documented with hyperlinks, or how easy it is to read (readability score) do not seem to play a big role in users’ choices of which answers to vote or accept.

3.2 Behavior vs number of answers

What is more surprising than the overall size of the regression coefficients, however, is that the largest coefficients, e.g., for web page order and word share, change substantially as the number of available answers to a question increases for most boards (Figs 3 and 4), and, as we will discuss shortly, the models describe the data increasingly well. Importantly, however, we see strong differences between board types. For example, the word share does not change significantly for meta boards and appears to be much smaller than for other board types.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Web page order regression coefficients increase with the number of answers users see.

Web page order regression coefficients for voting before (red triangles) and after (blue squares) an answer is accepted, as well as accepting an answer (green circles) for (a) technical, (b) non-technical, and (c) meta boards, with 2 to 20 answers. The shaded region represents the uncertainty in our values (see Section 2). Users increasingly depend on the web page order of an answer as the number of answers increases.

https://doi.org/10.1371/journal.pone.0173610.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Word share regression coefficients increase with the number of answers users see.

Word share regression coefficients for voting before (red triangles) and after (blue squares) an answer is accepted, as well as accepting an answer (green circles) for (a) technical, (b) non-technical, and (c) meta boards, with 2 to 20 answers. The shaded region represents the uncertainty in our values (see Section 2). Across all boards, voters appear increasingly likely to choose answers that take up a relatively large amount of web page space as the number of answers grows.

https://doi.org/10.1371/journal.pone.0173610.g004

A number of plausible explanations exist for why these coefficients depend on the number of answers:

1. Trivially, the dependence may be due to how we normalize our coefficients.
2. The subsequent answers improve upon the previous answer, or
3. Some unknown confounding variable affects both the number of answers as well as user behavior, or finally
4. User behavior changes as a function of the number of available answers.

To better understand the first hypothesis, one might object that the regression coefficient for word share, answer order, and chronological order will trivially depend on the number of answers because their normalization scheme itself depends on the number of answers. For example, if users choose longer answers over shorter ones, then, if all subsequent answers are long, the word share regression coefficients may change, even when the underlying mechanism, that users prefer longer answers, does not. To check for this effect, we incorporate both the word share and the number of words in each answer into our model. The word share attribute is found to have increasingly large regression coefficients as the number of answers grows, which suggests that the regression coefficient dependence is not trivially due to the normalization scheme. The webpage order regression coefficient, however, really does increase with the number of answers because of how the attribute is normalized. If we do not normalize the webpage order attribute, the regression coefficients would decrease with the number of answers. This is not an issue either, because our focus is on the relative attention that answers receive (the top answers recieve more attention than the bottom) rather than the absolute attention (that the third answer may recieve more attention when 10 answers are visible compared to when 3 answers are visible). This interpretation is similar for the chronological order attribute.

According to the second hypothesis, the last answer may be such an improvement on the previous ones that users will “flock” to it. Therefore, it should be no surprise that as the number of answers increases, changes in votes are seen. In theory, this should be captured by a significant dependence on answer’s chronological order: voters should prefer newer answers to older ones. In practice, this does not seem to be the case. The dependence on chronological order is relatively small (Fig 2), and furthermore does not change significantly with the number of answers, which is exactly the opposite of what should be expected if this hypothesis were true (S2 Fig).

The third hypothesis says that the number of answers and the behavior of the user both correlate to something else entirely; the results presented so far could be strongly affected by some confounding variable. For example, [11] finds that the reputation of later answerers on Stack Overflow, a technical board within Stack Exchange devoted to programming questions, is lower than the reputation of earlier answerers. If later voters similarly differ in reputation or some other attribute, this could potentially explain our results. We call this the “lazy voter” hypothesis, because later voters may simply be “lazier” and rely on heuristics to a greater extent. It is curious, however, that voter behavior does not seem to be significantly affected by the age of the answer, based on our regressions, and instead on the shear number of answers, as time progresses.

The last hypothesis is that users behave differently as the number of answers grows. Economics and psychologists believe that people usually do not have the time, nor inclination or cognitive resources, to process all available information, but instead, employ heuristics to quickly decide what information is important. This phenomenon, known as “bounded rationality” [27, 28], profoundly affects what information people pay attention to and the decisions they make [41]. Our results suggest that rather than thoroughly evaluating all available answers to a question on Stack Exchange, users employ cognitive heuristics to choose the “best” answer. These heuristics include choosing a top-ranked answer (Fig 3) or one that occupies more screen space (Fig 4). These heuristics become more pronounced when the volume of information (number of available answers, or cognitive load) grows.

Instead of being a cognitive heuristic, word share could plausibly reflect answer quality: high quality answers may be wordy. Interestingly, however, the regression coefficient of the number of words for each answer (rather than its share of words) is slightly negative (Fig 2), suggesting users overall prefer somewhat shorter answers if they prefer anything at all. It is intuitive that longer answers are more salient and catch a user’s eye, especially when there are many answers.

Whether the third hypothesis, or fourth, is true, our observation of a strong dependence of votes and accepts on the number of available answers suggests a strong limitation of crowdsourcing answer quality: collective judgment of quality may change with the number of answers, which is especially noticable with popular, and presumably important, questions which have many answers (Fig 1).

We see further evidence for the final two arguments in Fig 5, where we plot the regression coefficients for accepting an answer as a function of the number of answers for voters before, and after, an answer is accepted.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Answer acceptance increases the probability an answer will be voted on.

Regression coefficients for voting on an (eventually) accepted answer before (red triangles) and after (blue squares) that answer is accepted for (a) technical, (b) non-technical, and (c) meta boards, with 2 to 20 answers. The shaded region represents the uncertainty in our values (see Section 2). There is a large and increasing vote dependence on the accepted answer once the asker accepts it, compared to before the answer is accepted, meaning the signal that this answer is accepted appears to have a statistically significant effect on voter behavior.

https://doi.org/10.1371/journal.pone.0173610.g005

Voters are more likely to choose an answer that is eventually accepted (the regression coefficients are positive), but, curiously, voters are even more likely to choose the answer after it is accepted as the number of answers increase (the regression coefficient is usually even higher, and increases with the number of answers). In other words, although answer quality does not change before or after acceptance, users are more likely to vote on whatever the asker chooses, especially as the number of answers increase.

3.3 Predicting behavior

Finally, we use our trained model to predict users’ future choices of answers. After training our model on 2009-2013 data, we compared the predicted probabilities that an answer is chosen to the test set using ROC curves, in which we continuously varied the binary classifier threshold and determined from our data the true and false positive rate. Finally we take the area under the ROC curve (AUC) to determine the overall accuracy of predictions. If the AUC is near 0.5, then our model would be no better than chance. If the AUC is near 1, however, then our model predicts future behavior with a high accuracy.

We find that our model predicts which answer will be chosen with increasing accuracy as the number of answers increases (Fig 6), and approaches values of 0.9 or greater (greatly beating chance, in which the AUC is 0.5). We compare our full model to two null models: a position-based null model, in which the answer order is the sole determinant of which answer will be accepted or voted on, and a social influence null model, in which answer attributes might influence users due to social cues, namely answer score, whether the answer was accepted, and the answerer’s reputation (which users can see before they upvote an answer). Interestingly, our model initially outperforms the position-based null model but with enough answers, they are virtually indistinguishable, which suggests that the answer order is the dominant attribute in our model to predict user behavior. In comparison, the social signal-based null model tends to perform poorer than both models in this range, which suggests that social signals cannot describe user behavior as well as position-based heuristics.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. AUC versus number of answers for the full model, position null model, and social influence null model for technical boards.

AUC for (a) voting before an answer is accepted, (b) accepting an answer, and (c) voting after an answer is accepted versus the number of answers in technical boards (See S4 Fig for similar plots with non-technical and meta boards). Solid lines correspond to the full models, while the lighter dashed lines correspond to the position null model, in which the probability of picking answers decreases monotonically with the web page order. Finally, the dark dashed lines correspond to the “social influence” null model, in which social signals are the only attributes used in the model. Shaded regions correspond to standard deviations in values based on bootstrapping the test data.

https://doi.org/10.1371/journal.pone.0173610.g006

To determine the overall performance of the model, we found the mean AUC weighed by the number of votes, when two to twenty answers are visible (Fig 7). We notice that the full model has the overall highest AUC, and the greatest decrease in AUC is when we remove the web page order (“No Ranking”), or whether the answer will eventually be accepted (“No Accepted”), in agreement with expectations, due to their especially high regression coefficients, although, interestingly, removing the word share had little effect for tehnical and meta boards (“No Word Share”). We find that most drops in the AUC values are statistically significant (p<0.001). Furthermore, taking only the highest coefficient attributes, web page order, whether the answer will be accepted, and word share, we find AUC values close to the full model (“Only Top 3”), which agrees with our intuition that these are the major contributing factors to our model’s behavior. As we expected, the null models performed the worst (“Position Only Null” and “Social Influence Null”), although at different points either model performed worse than the other, making conclusions about overall performance difficult. We also notice, however, differences across board types, especially in technical boards, where votes after acceptance are significantly more predictable, regardless of the model used. This is seen in meta boards, but to a far smaller degree, suggesting that something unique about technical boards may contribute to this effect. We surmise that accepting an answer significantly changes voter behavior: users default to the accepted answer, simply because it is already known to answer the question.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Mean AUC versus models with various attributes removed.

The AUC for voting before (red lines) and after (blue lines) an answer is accepted, as well as accepting an answer (green lines) averaged over two through twenty answers, weighted by the amount of test data, in (a) technical, (b) non-technical, and (c) meta boards. See main text for descriptions of models. Error bars are standard deviations of the mean values. We remove attributes and measure the drop in mean AUC to determine the relative importance of various attributes. The statistical significance of the drop in AUC values compared to the full model is as follows: “***”, p<0.001; “**”, p<0.0028 (Bonferroni correction for 18 variables with α = 0.05 [42]); “–”, not significant.

https://doi.org/10.1371/journal.pone.0173610.g007

We also notice that askers are significantly more predictable than voters before acceptance (and usually after acceptance). We believe there are two possible reasons for this observation. First, question askers and voters look for very different answer properties. Voters may look for informative and general answers (which may be considered of higher quality), while askers look for answers that can simply solve their problem. They are, therefore, less concerned about the underlying quality so long as it is right. In addition, voters need a reputation above 15 in order to vote, which becomes a barrier to entry: typically, users must have provided answers or questions in the past that others upvoted in order to be able to vote. Askers on the other hand have no such reputation requirement. Askers might, therefore, be less able to recognize the highest quality answer, and instead rely on their own heuristics. In the past, accepted answers have been used as a gold standard of answer quality [13–15], but, if askers strongly rely on heuristics like an answer’s rank order, this puts into question whether accepted answers are the best standard. Instead, we find that the most upvoted answers may be a better standard, because voters appear to depend less on heuristics.