Elsevier

Ecological Economics

Volume 182, April 2021, 106904
Ecological Economics

Effects of the number of alternatives in public good discrete choice experiments

https://doi.org/10.1016/j.ecolecon.2020.106904Get rights and content

Highlights

  • Convergent validity does not hold for CE with SQ plus one, two or three alternatives.

  • A net matching effect is observed in the SQ plus one alternative treatment.

  • The net complexity effect increases with the number of alternatives.

  • Theory and empirical results support the use of the SQ plus one alternative design.

  • Our analysis suggests caution using more than one non-status-quo alternative in CE.

Abstract

Choice experiments (CEs) are commonly used to estimate monetary values for characteristics of public goods, but there are unresolved design issues. The number of alternatives is one of them. Increasing the number of alternatives increases the potential information learned from a sample of a limited size, which may assist subjects in selecting a preferred alternative (referred as matching) or may make choices more difficult (referred as complexity). A convergent-validity study is conducted to compare CE designs with status quo (SQ) plus one, two or three alternatives. To enhance convergent-validity insights, we use the SQ plus one treatment, which is a theoretically supported treatment, as a counterfactual treatment. We fail to find convergent validity between the one-alternative treatment and the two- and three-alternative treatments. Yet there is little difference in welfare estimates between the two- and three-alternative treatments. We find a net matching effect in the one-alternative treatment as the number of attribute-level changes increases, which reduces the likelihood of choosing the SQ alternative. We find net complexity effects in the two- and three-alternative treatments, which increases the likelihood of subjects choosing the SQ alternative as the number of choice questions increases. Our results support the use of SQ plus one-alternative design, suggest caution when using a SQ plus two- and three-alternative designs.

Introduction

Choice experiments (CEs) are commonly used to estimate monetary values for characteristics of public and private goods, but there are unresolved design considerations (Holmes et al., 2017). The number of alternatives in choice questions is a core design element. Increasing the number of alternatives can increase the information learned from a sample of limited size as subjects make choices over a larger set of attribute combinations. Increasing the number of alternatives may also assist subjects in selecting a preferred alternative, which has been referred to as matching (DeShazo and Fermo, 2002) or may make choices more difficult, which has been referred to as complexity (Hensher, 2006). Adding additional alternatives may affect coefficient estimates in two ways. On the one hand, additional alternatives would help subjects identify a desirable alternative, thus subjects possibly being less likely to choose the Status Quo (SQ) alternative. On the other hand, if additional alternatives increase complexity, subjects perhaps having a greater tendency to choose the SQ alternative to avoid making complex choices.

There have been a small number of studies that investigated different numbers of CE alternatives; yet, more research is needed. Collective insights are challenged because existing studies investigate different designs, use different analytical procedures, and have differing empirical outcomes that make it difficult to draw strong insights that have broad applicability for future CE designs. In addition, the comparisons of CE questions with different numbers of alternatives are investigations of convergent validity where the truth is unknown (Bishop and Boyle, 2019). Thus, multiple convergent-validity studies are required to develop insights on validity.

This paper adds to this line of research by exploring the effects of the number of alternatives in a CE on coefficient estimates, implicit prices, welfare estimates, and subjects' choice of the SQ alternative within the context of a public good. No other study to our knowledge has conducted all of these comparisons, which provide comprehensive insights regarding convergent validity. Tests of coefficient vectors, when rejected, imply there are differences in some estimates but not necessarily all or even the policy-relevant estimates. Moving to implicit prices removes the confound of the scale parameter involved in the coefficient comparisons, but unless a single attribute is of policy relevance, this may not tell the whole story. The welfare estimates are weighted combinations of implicit prices and can provide different insights than looking solely at implicit prices. The effects of matching and complexity (inferred from CE design features and subjects' self-reports) are also investigated in the context of subjects' choices of the SQ alternative. Finally, since convergent-validity comparisons require external information to enhance validity insights, we rely on theoretical incentive compatibility conditions of binary choices (single binary choices - Carson and Groves, 2007; repeated binary choices - Vossler et al., 2012; voting – Satterthwaite, 1975) as a counterfactual in the comparisons.

We investigate three choice-question designs: SQ plus one alternative (SQ + 1), two alternatives (SQ + 2), and three alternatives (SQ + 3). A split-sample design was implemented where subjects answered only one of the question formats within a sequence of eight choice questions. The application is a restoration project for the Macquarie Marshes in New South Wales, Australia.

We find mixed results regarding convergent validity. There are significant differences between the one-alternative treatment and two- and three- alternative treatments but not between the two- and three-alternative treatments. Comparisons of welfare estimates, the policy-relevant outcomes, reveal WTP estimates for the two- and three-alternative treatments are about half of those for the one-alternative treatment, which suggests that increasing the number of alternatives leads to more conservative welfare estimates. We find a net matching effect in the one-alternative treatment as the number of attribute-level changes increases, which reduces the likelihood of choosing the SQ alternative. We find net complexity effects in the two- and three-alternative treatments, which increases the likelihood of subjects choosing the SQ alternative as the number of choice questions increases. Combining that with the theoretical incentive compatibility conditions, our results support the use of status quo plus one alternative design and caution when using a status-quo plus two- or three-alternative designs.

Section snippets

Previous research

Thirteen studies have investigated the effects of the number of alternatives presented in CE questions (Table 1). A central theme of these studies is convergent validity, i.e., if two or more estimation procedures designed to estimate the same theoretical concept provide statistically similar value estimates (see Bishop and Boyle, 2019; Carmines and Zeller, 1979). Convergent validity occurs when the different numbers of alternatives in the CE questions (e.g., SQ + a vs. SQ + b, a ≠ b) provide

Study area and survey design

Our application is the valuation of restoration for the Macquarie Marshes, an ephemeral wetland along the Macquarie River in northwest New South Wales, Australia. A natural reserve contained in the Marshes is listed as a wetland of international importance under the Ramsar Convention. The Marshes have significant environmental benefits by providing important habitat for waterbirds, acting as a filter that improves downstream water quality, and providing high-quality feed for livestock and other

Conceptual model and econometric estimation

When answering the choice questions, subjects will choose alternative j in choice question k (Ajk) if U(Aj=l, k) > U(Ajl. k). Each treatment was analyzed using the same econometric specification. The utility for respondent n and alternative j in treatment s is decomposed into a systematic component (Vnjs) and a random component (εnjs):Unjs=Vnjs+εnjs=βnsxnjs+εnjswhere xnjs is a vector of observed attributes and βns is a vector of preference coefficients that vary over people and may vary over

Results

Subjects were randomly recruited from an online panel of New South Wales, AU residents provided by Research Now (https://www.researchnow.com). Subjects were directed to a website hosted by the Institute for Transport Studies at Sydney University. Each subject was randomly assigned to the SQ + 1, SQ + 2 or SQ + 3 treatment.

A total of 1827 subjects answered all eight choice questions. The final sample sizes for the treatments are: NSQ+1 = 609, NSQ+2 = 622 and NSQ+3 = 596. Using the estimation

Discussion and conclusions

Collectively, our findings indicate that the number of alternatives in CE questions affect statistical outcomes, but the pattern of differences varies across statistical comparisons. First, tests of coefficient estimates indicate significant differences across all three treatments: status quo plus one, two and three alternatives. This is not a strong insight because the tests do not indicate whether differences are due to differences in preferences or scale. Further, rejection of the null

Declaration of Competing Interest

None.

Acknowledgements

We are grateful for helpful comments and suggestions from anonymous referees and conference participants at the AAEA meeting, the Camp Resources workshop, and the 6th World Congress of Environmental Economics.

References (35)

  • J. Rolfe et al.

    The impact of offering two versus three alternatives in choice modelling experiments

    Ecol. Econ.

    (2009)
  • J.M. Rose et al.

    Identifying differences in willingness to pay due to dimensionality in stated choice experiments: a cross country analysis

    J. Transp. Geogr.

    (2009)
  • M.A. Satterthwaite

    Strategy-proofness and Arrow’s conditions: existence and correspondence theorems for voting procedures and social welfare functions

    J. Econ. Theory

    (1975)
  • R.C. Bishop et al.

    Reliability and validity in nonmarket valuation

    Environ. Resour. Econ.

    (2019)
  • P. Boxall et al.

    Complexity in choice experiments: choice of the status quo alternative and implications for welfare measurement

    Aust. J. Agric. Resour. Econ.

    (2009)
  • K.J. Boyle et al.

    Convergent validity of attribute-based, choice questions in stated-preference studies

    Environ. Resour. Econ.

    (2009)
  • E.G. Carmines et al.

    Reliability and Validity Assessment

    (1979)
  • Cited by (9)

    • Testing the sensitivity of stated environmental preferences to variations in choice architecture

      2023, Ecological Economics
      Citation Excerpt :

      However, choice architecture also includes decisions related to attribute representation and metrics, option labels, attribute/option order, visual design features, and various information, instruction, and decision aid elements (Johnson et al., 2012; Camilleri and Larrick, 2014; Münscher et al., 2016). For these reasons, a common approach to investigating design impacts in DCE research is to focus on a limited set of context-relevant design variations with a few split samples (e.g., Jacobsen et al., 2008, Pedersen et al., 2011; Zhang and Adamowicz, 2011; Kragt and Bennett, 2012; Zhao et al., 2013; Rolfe and Windle, 2015; Contu et al., 2016; Torres et al., 2017; Rid et al., 2018; Subroy et al., 2018; Glenk et al., 2019; Jensen et al., 2019; Ahi and Kipperberg, 2020; Dugstad et al., 2021; Weng et al., 2021). For example, Jacobsen et al. (2008) studied the effect of adding a verbal (“iconized”) descriptor to an attribute, while Kragt and Bennett (2012) examined three different quantitative attribute representations (relative levels versus absolute levels versus losses).

    View all citing articles on Scopus
    View full text