Skip to main content
SpringerLink
Log in

Methodological aspects in the assessment of treatment effects in observational health outcomes studies

  • Review Article
  • Published:
Applied Health Economics and Health Policy Aims and scope Submit manuscript

Abstract

Prospective observational studies, which provide information on the effectiveness of interventions in natural settings, may complement results from randomised clinical trials in the evaluation of health technologies. However, observational studies are subject to a number of potential methodological weaknesses, mainly selection and observer bias. This paper reviews and applies various methods to control for selection bias in the estimation of treatment effects and proposes novel ways to assess the presence of observer bias. We also address the issues of estimation and inference in a multilevel setting. We describe and compare the use of regression methods, propensity score matching, fixed-effects models incorporating investigator characteristics, and a multilevel, hierarchical model using Bayesian estimation techniques in the control of selection bias. We also propose to assess the existence of observer bias in observational studies by comparing patient- and investigator-reported outcomes. To illustrate these methods, we have used data from the SOHO (Schizophrenia Outpatient Health Outcomes) study, a large, prospective, observational study of health outcomes associated with the treatment of schizophrenia.

The methods used to adjust for differences between treatment groups that could cause selection bias yielded comparable results, reinforcing the validity of the findings. Also, the assessment of observer bias did not show that it existed in the SOHO study. Observational studies, when properly conducted and when using adequate statistical methods, can provide valid information on the evaluation of health technologies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Table I
Table II
Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Hofer A, Hummer M, Huber R, et al. Selection bias in clinical trials with antipsychotics. J Clin Psychopharmacol 2000; 20: 699–702

    Article  PubMed  CAS  Google Scholar 

  2. Wells KB. Treatment research at the crossroads: the scientific interface of clinical trials and effectiveness research. Am J Psychiatry 1999; 156: 5–10

    PubMed  CAS  Google Scholar 

  3. Rubin DB. Estimating causal effects from large data sets using propensity scores. Ann Intern Med 1997; 127(8 Pt 2): 757–63

    PubMed  CAS  Google Scholar 

  4. Concato J, Shah N, Horwitz RI. Randomised, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000 Jun 22; 342(25): 1887–92

    Article  PubMed  CAS  Google Scholar 

  5. Benson K, Hartz AJ. A comparison of observational studies and randomised, controlled trials. N Engl J Med 2000 Jun 22; 342(25): 1878–86

    Article  PubMed  CAS  Google Scholar 

  6. Black N. Why we need observational studies to evaluate the effectiveness of healthcare. BMJ 1996 May 11; 312(7040): 1215–8

    Article  PubMed  CAS  Google Scholar 

  7. Black N. What observational studies can offer decision makers. Horm Res. 1999; 51 Suppl. 1: 44–9

    Article  PubMed  CAS  Google Scholar 

  8. Britton A, McPherson K, McKee M, et al. Choosing between randomised and non-randomised studies: a systematic review. Health Technol Assess 1998; 2(13): I–iv, 1-124

    PubMed  CAS  Google Scholar 

  9. Pocock SJ, Elbourne DR. Randomised trials or observational tribulations? N Engl J Med 2000 Jun 22; 342(25): 1907–9

    Article  PubMed  CAS  Google Scholar 

  10. Friedman HS. Observational studies and randomised trials [letter]. N Engl J Med 2000 Oct 19; 343(16): 1195–6

    PubMed  CAS  Google Scholar 

  11. Kunz R, Khan KS, Neumayer HH. Observational studies and randomised trials [letter]. N Engl J Med 2000 Oct 19; 343(16): 1194–5

    Article  PubMed  CAS  Google Scholar 

  12. Sacks HS. Observational studies and randomised trials [letter]. N Engl J Med 2000 Oct 19; 343(16): 1195

    PubMed  CAS  Google Scholar 

  13. Liu PY, Anderson G, Crowley JJ. Observational studies and randomised trials [letter]. N Engl J Med 2000 Oct 19; 343(16): 1195

    PubMed  CAS  Google Scholar 

  14. Smith RP, Meier P. Observational studies and randomised trials [letter]. N Engl J Med 2000 Oct 19; 343(16): 1196

    PubMed  CAS  Google Scholar 

  15. Hlatky MA, Califf RM, Harrell Jr FE, et al. Comparison of predictions based on observational data with the results of randomised controlled clinical trials of coronary artery bypass surgery. J Am Coll Cardiol 1988; 11: 237–45

    Article  PubMed  CAS  Google Scholar 

  16. Kunz R, Oxman AD. The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998; 317: 1185–90

    Article  PubMed  CAS  Google Scholar 

  17. Wolfe RA. Observational studies are just as effective as randomised clinical trials. Blood Purif 2000; 18(4): 323–6

    Article  PubMed  CAS  Google Scholar 

  18. Greene T. Are observational studies ‘just as effective’ as randomised clinical trials? Blood Purif 2000; 18(4): 317–22

    Article  PubMed  CAS  Google Scholar 

  19. Vineis P. Proof in observational medicine. J Epidemiol Commun Health 1997; 51: 9–13

    Article  CAS  Google Scholar 

  20. Holmberg L, Baum M, Adami HO. On the scientific inference from clinical trials. J Eval Clin Pract 1999 May; 5(2): 157–62

    Article  PubMed  CAS  Google Scholar 

  21. Wilson GT. The clinical utility of randomised controlled trials. Int J Eat Disord 1998 Jul; 24(1): 13–29

    Article  PubMed  CAS  Google Scholar 

  22. Ellenberg JH. Selection bias in observational and experimental studies. Stat Med 1994 Mar 15-Apr 15; 13(5–7): 557–67

    Article  PubMed  CAS  Google Scholar 

  23. Kemler MA, de Vet HC. Does randomisation introduce bias in unblinded trials? Epidemiology 2000 Mar; 11(2): 228

    Article  PubMed  CAS  Google Scholar 

  24. Rabeneck L, Viscoli CM, Horwitz RI. Problems in the conduct and analysis of randomised clinical trials. Are we getting the right answers to the wrong questions? Arch Intern Med 1992 Mar; 152(3): 507–12

    Article  PubMed  CAS  Google Scholar 

  25. Caplan LR. Is the promise of randomised control trials (“evidence-based medicine”) overstated? Curr Neurol Neurosci Rep 2002; 2: 1–8

    Article  PubMed  Google Scholar 

  26. Haro JM, Edgell ET, Jones PB, et al. The European Schizophrenia Outpatient Health Outcome (SOHO) study: rationale, methods and recruitment. Acta Psychiatr Scand 2003; 107: 222–32

    Article  PubMed  CAS  Google Scholar 

  27. Haro JM, Edgell ET, Frewer P, et al. The European Schizophrenia Outpatient Health Outcomes (SOHO) study: baseline findings across country and treatment. Acta Psychiatr Scand Suppl 2003; 416: 7–15

    Article  PubMed  Google Scholar 

  28. Haro JM, Edgell ET, Novick D, et al. Effectiveness of antip-sychotic treatment for schizophrenia: 6-month results of the pan-European Schizophrenia Outpatient Health Outcomes (SOHO) study. Acta Psychiatr Scand 2005; 111: 220–31

    Article  PubMed  CAS  Google Scholar 

  29. Haro JM, Kamath SA, Ochoa S, et al. The clinical global impression-schizophrenia (CGI-SCH) scale: a simple instrument to measure the diversity of symptoms present in schizophrenia. Acta Psychiatr Scand Suppl 2003; 416: 16–23

    Article  PubMed  Google Scholar 

  30. Guy W. Clinical global impression. In: Guy W, editor. ECDEU assessment manual for psychopharmacology, revised. Rock-ville (MD): National Institute of Mental Health, 1976: 217–22

    Google Scholar 

  31. Williams A. EuroQol: a new facility for the measurement of health-related quality of life. Health Policy 1990; 16: 199–208

    Article  Google Scholar 

  32. Kind P, Hardman G, Macran S. UK population norms for EQ-5D. York Centre for Health Economics Discussion Paper, 172. York: Centre for Health Economics, University of York, 1999

    Google Scholar 

  33. Lambert M, Haro JM, Novick D, et al. Olanzapine vs. other antipsychotics in actual out-patient settings: six months tolera-bility results from the European Schizophrenia Out-patient Health Outcomes study. Acta Psychiatr Scand 2005; 111: 232–43

    Article  PubMed  CAS  Google Scholar 

  34. Rosenbaum PR. Observational studies. 2nd ed. New York: Springer-Verlag, 2002

    Google Scholar 

  35. Starr TB, Dalcorso RD, Levine RJ. Fertility of workers: a comparison of logistic regression and indirect standardization. Am J Epidemiol 1986; 123: 490–8

    PubMed  CAS  Google Scholar 

  36. Altman DG. Comparability of randomised groups. Statistician 1985; 34: 125–36

    Article  Google Scholar 

  37. Rubin DB, Thomas N. Matching using estimated propensity scores: relating theory to practice. Biometrics 1996; 52: 249–64

    Article  PubMed  CAS  Google Scholar 

  38. Cochrane WG. The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics 1968; 24: 295–313

    Article  Google Scholar 

  39. Drake C. Effects of misspecifications of the propensity score on estimators of treatment effects. Biometrics 1993; 49: 1231–6

    Article  Google Scholar 

  40. Hylan TR, Crown WH, Meneades L, et al. Tricyclic antidepres-sant and selective serotonin reuptake inhibitors antidepressant selection and health care costs in the naturalistic setting: a multivariate analysis. J Affect Disord 1998; 47: 71–9

    Article  PubMed  CAS  Google Scholar 

  41. Lu M. The productivity of mental health care: an instrumental variable approach. J Ment Health Policy Econ 1999; 2: 59–71

    Article  PubMed  Google Scholar 

  42. Salkever DS, Slade EP, Karakus M, et al. Estimation of antip-sychotic effects of hospitalisation risk in a naturalistic study with selection of unobservables. J Nerv Ment Dis 2004; 192: 119–28

    Article  PubMed  Google Scholar 

  43. Hadley J, Polsky D, Mandelblatt JS, et al. An exploratory instrumental variable analysis of the outcomes of localized breast cancer treatments in a Medicare population. Health Econ 2003; 12: 171–86

    Article  PubMed  Google Scholar 

  44. Bound J, Jaeger DA, Baker RM. Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak. J Am Stat Assoc 1995; 90: 443–50

    Google Scholar 

  45. Heckman JJ. Sample specification bias as a specification error. Econometrica 1979; 47: 153–61

    Article  Google Scholar 

  46. Wooldridge JM. Econometric analysis of cross section and panel data. Cambridge (MA): MIT Press, 2002

    Google Scholar 

  47. Lee M-J. Panel data econometrics: method-of-moments and limited dependent variables. San Diego (CA): Academic Press, 2002

    Google Scholar 

  48. Spiegelhalter DJ, Feedman LS, Parmar MKB. Bayesian approaches to randomized trials (with discussion). J R Stat Soc Ser A Stat Soc 1994; 157: 357–416

    Article  Google Scholar 

  49. Jones DA. A Bayesian approach to the economic evaluation of health care technologies. In: Spiker B, editor. Quality of life and pharmaeconomics in clinical trials. 2nd ed. Philadelphia (PA): Lippincott-Raven, 1996: 1189–96

    Google Scholar 

  50. Sculpher MJ, Pang FS, Manca A, et al. Generalisability in economic evaluation studies in health care: a review and case-studies. Health Technol Assess 2004; 8: 1–206

    Google Scholar 

  51. Rice N, Leyland A. Multilevel models: applications to health data. J Health Serv Res Policy 1996; 1: 154–64

    PubMed  CAS  Google Scholar 

  52. Leyland AH, Goldstein H. Multilevel modelling of health statistics. Chichester: Willey, 2001

    Google Scholar 

  53. Goldstein H, Browne W, Rasbash J. Multilevel modelling of medical data. Stat Med 2002; 21: 3291–315

    Article  PubMed  Google Scholar 

  54. Manca A, Rice N, Sculpher MJ, et al. Assessing generalisability by location in trial-based cost-effectiveness analysis: the use of multilevel models [published erratum appears in Health Econ 2005; 14 (5): 486]. Health Econ 2005; 14: 47–85

    Google Scholar 

  55. Grieve R, Nixon R, Thompson SG, et al. Using multilevel models for assessing the variability of multinational resource use and cost data. Health Econ 2005; 14: 185–96

    Article  PubMed  Google Scholar 

  56. Chien CF, Steinwachs DM, Lehman A, et al. Provider continuity and outcomes of care for persons with schizophrenia. Ment Health Serv Res 2000; 2: 201–11

    Article  Google Scholar 

  57. Haro JM, Salvador-Carulla L, Cabases J, et al. Utilisation of mental health services and costs of patients with schizophrenia in three areas of Spain. Br J Psychiatry 1998; 173: 334–40

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgements

Frank Windmeijer received monetary compensation from Eli Lily and Company for econometric advice. Josep Maria Haro received monetary compensation from Eli Lily and Company for his participation in the SOHO Advisory Board. David Suarez is providing statistical consultancy work for Lilly. Stathis Kontodimas and Mark Ratcliffe are Eli Lilly and Company employees.

Author information

Authors and Affiliations

  1. Research and Development Unit, RIRAG Network (FIS G03/061), Sant Joan de Deu-SSM, Fundació Sant Joan de Déu, Sant Boi, Barcelona, Spain

    Josep Maria Haro & David Suarez

  2. European Health Outcomes Research, Eli Lilly and Company Limited, Windlesham, UK

    Stathis Kontodimas & Mark Ratcliffe

  3. Department of Quantitative Methods, RIRAG Network (FIS G03/061), University of Las Palmas de Gran Canaria, Las Palmas de Gran Canaria, Spain

    Miguel Angel Negrin

  4. Department of Economics, University of Bristol, Bristol, UK

    Frank Windmeijer

  5. Sant Joan de Déu-SSM, Fundació Sant Joan de Déu, Dr. Antoni Pujades, 42, 08830, Sant Boi de Llobregat, Barcelona, Spain

    Josep Maria Haro

Authors
  1. Josep Maria Haro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stathis Kontodimas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Miguel Angel Negrin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mark Ratcliffe
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Suarez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Frank Windmeijer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Josep Maria Haro.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Haro, J.M., Kontodimas, S., Negrin, M.A. et al. Methodological aspects in the assessment of treatment effects in observational health outcomes studies. Appl Health Econ Health Policy 5, 11–25 (2006). https://doi.org/10.2165/00148365-200605010-00003

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.2165/00148365-200605010-00003

Keywords

Search

Navigation