Item wording effects in self-report measures and reading achievement: Does removing careless respondents help?
Section snippets
Item wording effects
In psychological instruments, negatively and positively worded items are often used together for various reasons. In surveys and questionnaires, the combination of positively and negatively worded items can prevent respondents from selecting only positive (or only negative) responses systematically (e.g., Baumbartner & Steenkams, 2001; Weijters, Geuens, & Schillewaert, 2009). Furthermore, negatively worded items may distract respondents from showing acquiescence response style or force them to
Sample
The data of this study came from the 2016 administration of PIRLS, which is an international large-scale assessment organized by the International Association for the Evaluation of Educational Achievement. PIRLS is administered every five years to assess the reading literacy of fourth graders around the world. Results of PIRLS allow researchers and educational agencies to analyze trends in reading achievement and to make international comparisons. The PIRLS assessment consists of
Comparisons of the hypothesized factor models
Table 3 shows the results of the four CFA models (i.e., one-factor, two-factor, bi-factor, and second-order models) for each country. The one-factor (i.e., unidimensional) model did not seem to fit the SCR scale sufficiently in any of the countries, suggesting that item wording effects might be present in the scale. Although the CFI values were at or above the suggested cut-off value of 0.95 for some countries (e.g., Australia, Canada, and Ireland), both TLI and RMSEA did not indicate a good
Discussion
Previous research showed that if the phrasing of items (i.e., positive or negative) has a significant impact on how respondents answer the items, item wording effects may occur as a major threat to the validity of the instrument (Kam & Meyer, 2015; McKay et al., 2014; Zeng et al., 2020). To shed further light on the causes and treatment of item wording effects in self-report measures, this empirical study investigated the relationship between item wording effects and respondents’ reading
Data availability
The data used in this study is publicly available at https://timssandpirls.bc.edu/pirls2016/international-database/.
References (61)
- et al.
A study of polytomous IRT methods and item wording directionality effects on perceived stress items
Personality and Individual Differences
(2019) - et al.
Personality correlates of method effects due to negatively worded items on the Rosenberg Self-Esteem scale
Personality and Individual Differences
(2009) - et al.
The impact of wording effect on reliability and validity of the Core Self-Evaluation Scale (CSES): A bi-factor perspective
Personality and Individual Differences
(2015) Ascertaining the validity of individual protocols from web-based personality inventories
Journal of Research in Personality
(2005)- et al.
Processing negated sentences with contradictory predicates: Is a door that is not open mentally closed?
Journal of Pragmatics
(2006) - et al.
Ask me, I (Dis) agree! Acquiescence in student ratings of teaching quality in German vocational schools
Studies in Educational Evaluation
(2021) - et al.
Personality correlates and gender invariance of wording effects in the German version of the Rosenberg Self-Esteem Scale
Personality and Individual Differences
(2016) - et al.
The role of context in young children’s comprehension of negation
Journal of Memory and Language
(2014) - et al.
The proximity effect: The role of inter-item distance on reverse-item bias
International Journal of Research in Marketing
(2009) Effects of stem and Likert response option reversals on survey internal consistency: If you feel the need, there is a better alternative to using those negatively- worded stems
Educational and Psychological Measurement
(2000)
Response styles in marketing research: A cross-national investigation
Journal of Marketing Research
An IRT mixture model for rating scale confusion associated with negatively worded items in measures of social-emotional learning
Applied Measurement in Education
Item wording effects in psychological measures: Do early literacy skills matter?
Journal of Measurement and Evaluation in Education and Psychology
Further evidence on response sets and test design
Educational and Psychological Measurement
Further investigating method effects associated with negatively worded items on self-report surveys
Structural Equation Modeling
The effects of positively and negatively worded items on the factor structure of the UCLA Loneliness Scale
Journal of Psychoeducational Assessment
An examination of the short form of the Need for Cognition Scale applied in an Australian sample
Educational and Psychological Measurement
Cognitive abilities explain wording effects in the Rosenberg Self-Esteem Scale
Assessment
MplusAutomation: An R package for facilitating large-scale latent variable analyses in Mplus
Structural Equation Modeling
Acquiescent and socially desirable response styles in cross-cultural value surveys
The acquiescence effect in responding to a questionnaire
GMS Psycho-Social Medicine
Methods of detecting insufficient effort responding: Comparisons and practical recommendations
Educational and Psychological Measurement
Cut-off criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives
Structural Equation Modeling
Difficulties in comprehending affirmative and negative sentences: Evidence from Chinese children with reading difficulties
Journal of Learning Disabilities
Detecting and deterring insufficient effort responding to surveys
Journal of Business and Psychology
Insufficient effort responding: Examining an insidious confound in survey data
Journal of Applied Psychology
Assessing acquiescence in surveys using positively and negatively worded questions
How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction
Organizational Research Methods
Why are negative questions difficult to answer? On the processing of linguistic contrasts in surveys
Public Opinion Quarterly
Sample design in PIRLS 2016
Cited by (1)
Who responds inconsistently to mixed-worded scales? Differences by achievement, age group, and gender
2024, Assessment in Education: Principles, Policy and Practice