Skip to main content

Advertisement

SpringerLink
Log in

PISA Data: Raising concerns with its use in policy settings

  • Published:
The Australian Educational Researcher Aims and scope Submit manuscript

Abstract

This article considers the role played by policy makers, government organisations, and research institutes (sometimes labelled “think tanks”) in the analysis, use and reporting of PISA data for the purposes of policy advice and advocacy. It draws on the ideas of Rizvi and Lingard (Globalizing Education Policy, 2010), Bogdandy and Goldmann (Governance by Indicators/ Global Power through Quantification and Rankings, 2012) and others to explore the ways in which such “agents of change” can interpret, manipulate and disseminate the results of data arising from large scale assessment survey programs such as PISA to influence and determine political and/or educational research agendas. This article illustrates this issue by highlighting the uncertainty surrounding the PISA data that have been used by a number of prominent, high profile agents of change to defend policy directions and advice. The final section of this paper highlights the need for policy makers and their advisors to become better informed of the technical limitations of using international achievement data if such data are to be used to inform policy development and educational reforms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Alexander, R. J. (2010). World class schools—noble aspirations or globalised hokum? Compare: A Journal of Comparative and International Education, 40(6), 801–817.

    Google Scholar 

  • Ashenden, D. (2013). Class sizes and the dead hand of history, inside story: Current affairs and culture from Australia and beyond, viewed 20th February 2015 http://insidestory.org.au/class-sizes-and-the-dead-hand-of-history/.

  • Ball, S. (1990). Self-doubt and soft data: social and technical trajectories in ethnographic fieldwork. International Journal of Qualitative Studies in Education, 3(2), 157–171.

    Article  Google Scholar 

  • Ball, S., & Exley, S. (2010). Making policy with ‘good ideas’: Policy networks and the ‘intellectuals’ of New Labour. Journal of Education Policy, 25(2), 151–169.

    Article  Google Scholar 

  • Bogdandy, A. V., & Goldmann, M. (2012). Taming and framing indicators: A legal reconstruction of the OECD’s program for international student assessment (PISA). In K. E. Davis, A. Fisher, B. Kingsbury, & S. E. Merry (Eds.), Governance by Indicators/Global Power through Quantification and Rankings. Oxford: Oxford University Press.

    Google Scholar 

  • Buckingham, J. (2012). Keeping PISA in perspective: Why australian education policy should not be driven by international test results, Issue Analysis, No. 136. Centre for Independent Studies, Sydney. Retrieved from 21st February http://www.cis.org.au/images/stories/issue-analysis/ia136.pdf.

  • Byrne, B., Coventry, W. L., Olson, R. K., Wadsworth, S. J., Samuelsson, S., Petrill, S. A., et al. (2009). Teacher effects in early literacy development: Evidence from a study of twins. Journal of Educational Psychology, 102(1), 32–42. doi:10.1037/a0017288.

    Article  Google Scholar 

  • Chen, C., & Stevenson, H. W. (1995). Motivation and mathematics achievement: A comparative study of Asian-American. Caucasian-American and East Asian high school students, Child Development, 66(4), 1214–1234.

    Google Scholar 

  • Cosgrove, J., & Cartwright, F. (2014). Changes in achievement on PISA: The case of Ireland and implications for international assessment practice. Large-scale Assessments in Education, 2014(2), 2. doi:10.1186/2196-0739-2-2.

    Google Scholar 

  • Feniger, Y., & Lefstein, A. (2014). How not to reason with PISA data: An ironic investigation. Journal of Education Policy, 29(6), 845–855.

    Article  Google Scholar 

  • Figazzolo, L. (2009). Impact of PISA 2006 on the Education Policy Debate. Brusells: Education International, Retrieved from 21st February 2015 http://download.ei-ie.org/docs/IRISDocuments/Research%20Website%20Documents/2009-00036-01-E.pdf

  • Gorur, R., & Wu, M. (2014). Leaning too far? PISA, policy and Australia’s ‘top five’ ambitions. Discourse: Studies in the Cultural Politics of Education,. doi:10.1080/01596306.2014.930020.

    Google Scholar 

  • Goyette, K., & Xie, Y. (1999). Educational expectations of Asian American youths: Determinants and ethics differences. Sociology of Education, 72(1), 22–36.

    Article  Google Scholar 

  • Grek, S. (2009). Governing by numbers: The PISA ‘effect’ in Europe. Journal of Education Policy, 24(1), 23–37.

    Article  Google Scholar 

  • Grisay, A., Gonzalez, E., & Monseur, C. (2009). Equivalence of item difficulties across national versions of the PIRLS and PISA reading assessments. In M. von Davier & D. Hastedt (Eds.), IERI monograph series: Issues and methodologies in large scale assessments (Vol. 2). Hamburg: IER Institute.

    Google Scholar 

  • Hao, L., & Bonstead-Bruns, M. (1998). Parent-child differences in educational expectations and the academic achievement of immigrant and native students. Sociology of Education, 71(3), 175–198.

    Article  Google Scholar 

  • Hsin, A., & Xie, Y. (2014). Explaining Asian Americans’ academic advantage over whites. The Proceedings of the National Academy of Sciences of the USA, June 10, 2014. 111(23), 8416–8421. http://www.pnas.org/content/111/23/8416.full.pdf+html

  • Hwang, T. (2015). The studying and striving of secondary students, chapter 7. In S. Hsu & Y.-Y. Wu (Eds.), Education as cultivation in Chinese culture, Education in the Asia-Pacific region: Issues, concerns and prospects. Singapore: Springer Science + Business Media.

    Google Scholar 

  • Jensen, B. (2010). Investigating our teachers, investing in our money. Melbourne: Grattan Institute. Retrieved from 21st February, 2015. http://grattan.edu.au/wp-content/uploads/2014/04/057_report_education_investing_teachers.pdf

  • Jensen, B. (2012). Catching up: learning from the best school systems in East Asia: Summary Report. Melbourne: Grattan Institute.

    Google Scholar 

  • Jensen, B., Hunter, A., Sonnemann, J., & Burns, T. (2012). Catching up: learning from the best school systems in East Asia. Melbourne: Grattan Institute.

    Google Scholar 

  • Lee, J. (2014). Minister apologies for 12 year compulsory education confusion, The China Post. Retrieved from 23rd July, 2015. http://www.chinapost.com.tw/print/409965.htm.

  • Leigh, A. (2010). Estimating teacher effectiveness from two-year changes in students’ test scores. Economics of Education Review, 29(3), 480–488.

    Article  Google Scholar 

  • Liu, J. (2012). Does cram schooling matter? Who goes to cram schools? Evidence from Taiwan. International Journal of Educational Development, 32, 46–52.

    Article  Google Scholar 

  • Liu, J. (2013). An overview of student achievement and the related factors in Taiwan, Chapter 9.13. In J. Hattie & E. Anderman (Eds.), International guide to student achievement. New York: Routledge.

    Google Scholar 

  • Ministry of Education. (2013). Education in Taiwan. Taipei: Ministry of Education, http://www.edu.tw

  • Monseur, C., & Berezner, A. (2007). The computation of equating errors in international surveys in education. Journal of Applied Measurement, 8(3), 323–335.

    Google Scholar 

  • OECD (2010a). PISA 2009 results: Learning trends: Changes in student performance since 2000 (Volume V). Retrieved May 15, 2012, from http://dx.doi.org/10.1787/9789264091580-en

  • OECD. (2010b). PISA 2009 results: What makes a school successful?—Resources, policies and practices (Volume IV). Retrieved May 15, 2012, from http://dx.doi.org/10.1787/9789264091559-en

  • Pyne, C. (2012). Aust obsessed with small class sizes: The Australian, 2nd August, Retrieved from 21st February 2015, http://www.theaustralian.com.au/news/latest-news/aust-obsessed-with-small-class-sizes-pyne/story-fn3dxiwe-1226428178883

  • Rizvi, F., & Lingard, B. (2010). Globalizing education policy. Abingdon: Routledge.

    Google Scholar 

  • Robertson, R. (1995). Glocalization: Time-space and homogeneity-heterogeneity. In M. Featherstone & R. Robertson (Eds.), Global modernities (pp. 25–44). London: Sage.

    Chapter  Google Scholar 

  • The Economist. (2012). Lessons from East Asia: The classroom crush, Why class sizes in England may be sent to expand, Retrieved from 21st February 2015, http://www.economist.com/node/21547854

  • Thrupp, M. (2014). When PISA meets politics—a lesson from New Zealand. The Conversation, 20 May 2014. Viewed 21st February 2015, http://theconversation.com/when-pisa-meets-politics-a-lesson-from-new-zealand-26539

  • Tran, T. T. (2012). Is the learning approach of students from Confucian heritage culture problematic? Education Research for Policy and Practice,. doi:10.1007/s10671-012-9131-3.

    Google Scholar 

  • Tuchman, B. W. (1984). The march of folly: from Troy to Vietnam. New York: Ballantine Books.

    Google Scholar 

  • Urbach, D. (2013). An investigation of Australian OECD Pisa Trend results. In M. Prenzel, M. Kobarg, K. Schöps & S. Rönnebeck (Eds.), Research on PISA: Research Outcomes of the PISA conference 2009. Springer, doi: 10.1007/978-94-007-4458-5.

  • Watkins, T., Kirk, S., Small, V., & Levy, D. (2012). Backlash forces government class size U-turn. Report by Stuff.co.nz at http://www.stuff.co.nz/national/education/7059177/Backlash-forces-Government-class-size-U-turn

  • Wei, M., & Eisenhart, C. (2011). Why do Taiwanese children excel at math? Kappan, 93(1), 74–76.

    Article  Google Scholar 

  • Wu, M. (2009). A comparison of PISA and TIMSS 2003 achievement results in mathematics. Prospect, 39, 33–46. doi:10.1007/s11125-009-9109-y.

    Article  Google Scholar 

  • Wu, M. (2014). Evidence-based policy making in education. International Journal of Contemporary Educational Research, 1(1), 1–8.

    Google Scholar 

  • Zyngier, D. (2014). Class size and academic results, with a focus on children from culturally, linguistically and economically disenfranchised communities. Evidence Base, 1, 1–23.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The University of Melbourne, Parkville, Australia

    Shelley Gillis & John Polesel

  2. National Taiwan Normal University, Taipei, Taiwan

    Margaret Wu

Authors
  1. Shelley Gillis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John Polesel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Margaret Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shelley Gillis.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gillis, S., Polesel, J. & Wu, M. PISA Data: Raising concerns with its use in policy settings. Aust. Educ. Res. 43, 131–146 (2016). https://doi.org/10.1007/s13384-015-0183-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13384-015-0183-2

Keywords

Search

Navigation