Abstract
In this paper, we expand the traditional perspective on standard setting to include the necessary antecedents to a genuinely valid setting of standards, and use that conceptual framework to propose a new foundation for standard setting. These necessary antecedents include (a) the definition of an underlying variable on which the “standard” will be set in a way that is designed to be suitable for that standard setting, (b) the selection of a qualitatively definable point on that variable that corresponds to “enough” for the standard to be met, (c) the development of a suitable procedure (“test”) and expression of its results in a suitable way to readily afford use in a standard setting procedure, and (d) the application of a suitable method for deciding the observable cut score that reflects attainment of the standard. From this new perspective, we critique two examples of the traditional approach, the “Modified Angoff” and the “Matrix method.” We then describe an approach consistent with the more broadly-based foundation, centered on the Construct-Mapping line of thinking. We give an example of this in a unidimensional context. This approach is then generalized to address multidimensional constructs. We also illustrate a software application that has been developed to facilitate this process. We conclude by discussing some consequences of adopting the new approach, and survey needed next steps in research and development.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This label is confusing when it refers to just one aspect of “standard setting,” but that is the typical usage.
- 2.
Note that, if one had a polytomous item, say with 3 ordered response categories, one might expect that this would match, say two levels of a construct map, though they might not be consecutive.
- 3.
Note that for polytomous items, the labels actually show the difficulties of the respective scores—hence “Range-2” is the second threshold difficulty for the item “Range.”
References
Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508–600). Washington, DC: American Council of Education.
Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80(2), 139–148.
Cizek, G. (2011). Setting performance standards: Foundations, methods, and innovations. New York: Routledge.
Draney, K., & Wilson, M. (2011). Selecting cut scores with a composite of item types: The construct mapping procedure. Journal of Applied Measurement, 12(3), 298–309.
Hoskens, M., & Wilson, M. (1999). ConstructMap [Computer program]. Berkeley: Berkeley Evaluation and Assessment Research Center.
Kane, M. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.
Lehrer, R., Kim, M.-J., Ayers, E., & Wilson, M. (2014). Toward establishing a learning progression to support the development of statistical reasoning. In A. Maloney, J. Confrey, & K. Nguyen (Eds.), Learning over time: Learning trajectories in mathematics education (pp. 31–60). Charlotte: Information Age Publishers.
Livingston, S. A., & Zieky, M. J. (1982). Passing scores: A manual for setting standards of performance on educational and occupational tests. Princeton: Educational Testing Service.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessment. Measurement: Interdisciplinary Research and Perspectives, 1(1), 3–67.
National Research Council. (2001). Knowing what students know: The science and design of educational assessment (Committee on the Foundations of Assessment. J. Pellegrino, N. Chudowsky, R. Glaser (Eds.), Division on Behavioral and Social Sciences and Education). Washington, DC: National Academy Press.
Reckase, M. D. (1998). Analysis of methods for collecting test-based judgments. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.
Wang, N. (2003). Use of the Rasch IRT model in standard setting: An item-mapping approach. Journal of Educational Measurement, 40, 231–253.
Wilson, M. (2004). Constructing measures: An item response modeling approach. Mahwah/New York: Erlbaum/Taylor and Francis.
Wilson, M., & Draney, K. (2002). A technique for setting standards and maintaining them over time. In S. Nishisato, Y. Baba, H. Bozdogan, K. Kanefugi (Eds.), Measurement and multivariate analysis (Proceedings of the International Conference on Measurement and Multivariate Analysis, Banff, Canada, May 12–14, 2000) (pp. 325–332). Tokyo: Springer.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix: The BEAR Assessment System
Appendix: The BEAR Assessment System
The BEAR Assessment System (Wilson 2004) consists of interrelated components (see Fig. 3.8), called building blocks, that are used to design measuring instruments and which are congruent with recent efforts to reform measurement in the domain of educational assessments (National Research Council 2001). The first building block is the construct map, which seeks to describe the variable being measured, from one extreme (say, low) to the other (say, high), and which is delineated by qualitatively-distinct levels. This is then used to develop an items design, which is the generic term for methods to generate responses from the respondents. These responses are then coded and valued using an outcome space. The resulting codes are analyzed using a measurement model, which is chosen to allow the analysis results to be related back to the construct map. In its development phase, these building blocks form a cycle of improvement for the measuring instrument. The building blocks enable and enhance the interpretation of the measures.
The Bear Assessment System (BAS)
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Wilson, M., Santelices, M.V. (2017). Weaknesses of the Traditional View of Standard Setting and a Suggested Alternative. In: Blömeke, S., Gustafsson, JE. (eds) Standard Setting in Education. Methodology of Educational Measurement and Assessment. Springer, Cham. https://doi.org/10.1007/978-3-319-50856-6_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-50856-6_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-50855-9
Online ISBN: 978-3-319-50856-6
eBook Packages: EducationEducation (R0)