Elsevier

Journal of Health Economics

Volume 27, Issue 5, September 2008, Pages 1275-1284
Journal of Health Economics

Optimal recall length in survey design

https://doi.org/10.1016/j.jhealeco.2008.05.012Get rights and content

Abstract

Self-reported data collected via surveys are a key input into a wide range of research conducted by economists. It is well known that such data are subject to measurement error that arises when respondents are asked to recall past utilisation. Survey designers must determine the length of the recall period and face a trade-off as increasing the recall period provides more information, but increases the likelihood of recall error. A statistical framework is used to explore this trade-off. Finally we illustrate how optimal recall periods can be estimated using hospital use data from Sweden's Survey of Living Conditions.

Introduction

Self-reported data collected via surveys are a key input into a wide range of research conducted by economists and other social scientists. For example, in health economics, economic evaluations of health care interventions are often based on self-reported health care data that is collected during the course of a study. It is also routinely collected in national health surveys such as the Health Survey for England, Australia's National Health Survey or the Medical Expenditure Survey in the US. While these surveys involve interviews conducted over the course of a year, there is considerable variation in the period over which subjects are asked to recall their previous health care use. This has been highlighted by a recent OCED study comparing inequity in access to health care across 21 developed countries based on health surveys in different countries. The period of recall for primary care and aspects of hospital use ranged from 2 weeks in the Australian National Health Survey to one year in the European Community Household Panel (Health Equity Research Group, 2004).

It has been widely recognized that there is an inverse relationship between the length of time over which subjects are asked to recall prior use and the accuracy of the reported estimates. The longer the period of recall the greater the likelihood of error. An early study by Sudman and Bradburn (1973) on the impact of the length of the recall period on response suggested two types of memory error may arise: (i) a respondent may forget an episode of use entirely; and (ii) they may remember the episode but incorrectly recall when it occurred. In regard to the latter, a particular form of error is telescoping which arises when the respondent recalls events that occurred before the period in question. For example, a person my report they have been in hospital in the last three months, even though the actual event was five months prior to the date of interview. While these errors can work in opposite directions over the last three decades validation studies of self-reported health care use have consistently shown that under-reporting is more common than over-reporting especially of primary care visits (e.g. see Cannell et al., 1977 and for a more recent review Bhandari and Wagner, 2006). Underreporting over longer recall periods may also characterise other types of data that are a focus of health economic research such as reported use of illicit drugs (see Bachman and O’Malley, 1981).

While recall error is undesirable, a survey with a short length of recall provides very little information about an individual's normal health care use. As Deaton (1997, p. 24) has noted in the context of measuring general household consumption expenditure:

“…if the object of the exercise is to estimate average consumption over a year, one extreme is to approach a sample of households on January 1 and ask each to recall expenditures for the last year. The other extreme is to divide the sample over the days of the year, and to ask each to report consumption for the previous day. The first method would yield a good picture of each household's consumption, but runs the risk of measurement error… [t]he second method… will give a good estimate of the mean consumption over all households… [but] will yield estimates of individual expenditure that… are only weakly related to normal expenditures…”

Suppose self-reported consumption or utilisation data are to be collected by a survey. At the survey design stage there is a choice of the length of the recall window. Even if the variable of interest is consumption over a particular period, say a year, errors induced by having long recall windows equal to the period of interest suggest that one could opt for a shorter window or sub-period as the basis of the survey question. While this provides a less error ridden measure it comes at the cost of providing less information. Hence, the basic research question is: “What is the optimal recall window over which to ask the utilisation question when recall error increases with window length but a longer window is likely to provide more (albeit imperfect) information directly relevant to the variable of interest?”

We investigate these tradeoffs in order to determine what factors impact on the optimal recall window over which to ask consumption and utilisation questions. Our analysis distinguishes between two types of recall error. Firstly, error due to forgetting (or telescoping) that may impact on the average reported health care use in one direction leading to under (or over) reporting. This can be regarded as bias in the reported use. Another potential source of bias is rounding which is explored by Huttenlocher et al. (1990) who note respondents tend to round off large values. For example, a physician when responding to a survey on particular patients use over a year: “may use raw values up to roughly 5, then round by 5s to roughly 20 and round by 10s past 20” (p. 211). They argue that rounding is likely to impart a downward bias on the reported value. Secondly, reporting errors that are more random in nature. This may not result in a biased measure, but will increase the degree of variation in the reported measures.

The rest of this study is structured as follows. The next section outlines a simple framework that is used to capture the key dimensions of the research question. The third section derives statistical criteria for examining the consequences of choosing different recall periods. The fourth section applies these to illustrate how optimal recall periods can be estimated for a question on hospital usage using data from Statistics Sweden's Survey of Living Conditions. Section five discusses the implications of these results and avenues for future research. The final section draws some conclusions.

Section snippets

A framework for analysis

Denote the variable of interest for the ith individual by Yi (i = 1, …, N). If this could be recalled without error then there would be no problem. Such a situation represents a benchmark for comparisons as specific forms of recall errors are introduced. Suppose the period over which the utilisation variable is required, call this the target period, can be divided into S sub-periods where the division is determined so that individuals can accurately recall their utilisation over the most recent

Some results for optimal recall windows

Further investigation requires more structure for the data generating process assumed to be underlying this problem. In what follows both discrete and continuous utilisation variables are considered. The issues that arise and some of the conclusions are different enough to justify this distinction.

Some empirical results from Swedish data

To illustrate how the previously derived results can be used to inform survey design we provide an example that is based on information concerning recall error associated with the number of nights spent in hospital over the previous three months obtained from Statistics Sweden's Survey of Living Conditions (the ULF survey) which has been linked to the national Patient Register (the National Board of Health and Welfare). Every year, Statistics Sweden conducts systematic surveys of living

Discussion

While stylised, the modelling serves to highlight the key aspects of the generic problem. There are tradeoffs. While longer recall windows may invariably introduce recall errors, using shorter recall windows also come with costs associated with less information. The analysis so far has served to illustrate these tradeoffs and to stress that the case for eliminating recall errors is not necessarily compelling.

It is worth highlighting some important dimensions of the problem that impact the

Conclusions

A common criticism of statistical analyses using self-reported data is the problem caused by recall error. A possible response by survey designers is to choose extremely short recall windows in order to eliminate this source of error. For example, some national health surveys only ask health service use over the previous two weeks. One possible interpretation of our results is that such a reaction may not be entirely justified. Recall error may be eliminated but at a potentially huge cost in

Acknowledgements

This research was partially supported by: National Health and Medical Research Council through a Program Grant No. 254202 (Prof. Fiebig). Other funding was from the Swedish Council for Working Life and Social Research (2002-0376) and Swedish National Institute of Public Health for Prof Gerdtham. Dr. Clarke also received funds through a Swedish Council for Working Life and Social Research Visiting Fellowship. Comments of three referees and participants at the 2005 Australian Health Economics

References (20)

  • J.A. Bowman et al.

    The accuracy of self-reprted Pap smear utilization

    Social Science and Medicine

    (1997)
  • T. Philipson et al.

    Measurement errors: a principal investigator-agent approach

    Journal of Econometrics

    (1999)
  • J.G. Bachman et al.

    When four months equal a year: inconsistencies in student reports of drug use

    The Public Opinion Quarterly

    (1981)
  • Belkar, R., Chanthapun, W.P., Fiebig, D.G., 2007. A discrete choice model with misclassification and multiple recall...
  • A. Bhandari et al.

    Self-reported utilization of health care services: improving measurement and accuracy

    Medical Care Research and Review: MCRR

    (2006)
  • J. Bound et al.

    Measurement error in survey data

  • H. Benitez-Silva et al.

    How large is the bias in self-reported disability?

    Journal of Applied Econometrics

    (2004)
  • A.H. Briggs et al.

    Missing … presumed at random: cost-analysis of incomplete data

    Health Economics

    (2003)
  • C.F. Cannell et al.

    A summary of studies of interviewing methodology

    Vital and Health Statistics. Series 2, Data Evaluation and Methods Research

    (1977)
  • Carson, R., Groves, T., Machina, M., 1999. Incentive and Informational Properties of Preferences Questions, Plenary...
There are more references available in the full text version of this article.

Cited by (0)

View full text