Princeton ACM / IEEE Computer Society Meeting

PRINCETON ACM / IEEE COMPUTER SOCIETY CHAPTERS

MAY 2003 ANNUAL DINNER MEETING

INTUITIVE TEST THEORY

Robert Mislevy
University of Maryland

In this talk I will discuss some about “phenomenological primitives” (p-prims) about testing and assessment. (“P-prims” is a term coined by the psychologist Andrea diSessa to explain why people have incorrect notions about science, such as “heavy objects fall faster than light objects”.) This talk will show some of the testing p-prims we have recognized in our experiences, and think about the implications of intuitive test theory for educational policy. Here are some examples of test theory p-prims:

A test measures what it says at the top of the page.
Any two tests that measure the same thing can be made interchangeable, with a little equating magic.
A score is a score is a score – don’t give me any of this “measurement error” nonsense. (How could there be a “truer” score than the score a student actually gets?)
You score a test by adding up scores for items.
70% is passing – on any and every assessment.
It’s easy to write test items, as long as you’re careful with your grammar.
You can tell if an item is good by looking at it.
Multiple-choice questions only measure recall.
Technology will solve testing problems by making it possible to get voluminous amounts of data.

Henry Braun of Educational Testing Service and I are starting to examine p-prims that underlie nonexperts' reasoning about assessment. They form the basis of what one might call folk test theory or intuitive test theory. They underlie discussions of test theory in the classroom, in the news, and in educational policy-making. Intuitive test theory works reasonably well for everyday uses like Friday's math quiz. It fails when you want to design an adaptive test or measure the change in the proportion of students reading Above Basic from a complex sampled assessment like the National Assessment of Educational Progress.

Robert Mislevy is a Professor in the Department of Measurement, Statistics, and Evaluation at the University of Maryland. Previously, Dr. Mislevy was a Distinguished Research Scientist at Educational Testing Service, where he had worked since 1984. Dr. Mislevy’s research interests center on applying developments in statistics, cognitive psychology, and technology to practical problems in educational and psychological measurement. His current projects include simulation-based assessment of troubleshooting in computer networks with the Cisco Learning Institute, and a design framework for assessing inquiry in science, with SRI International. More information on his current work can be found on his web site at http://www.education.umd.edu/EDMS/mislevy.

Date: Thursday, May 22, 2003. 6:30 PM cash bar, 7:00 PM dinner, 8:00 PM presentation
Location: Good Time Charley's Restaurant, Route 27, Kingston, NJ. Ask for the "ACM Group".
Cost: $20
Additional Information: Dennis Mancl (908) 582-7086, Jim Matey (609) 734-2868
URL: http://www.acm.org/chapters/princetonacm/
E-mail: princetonacm@acm.org

RESERVATIONS REQUIRED – please fill out the form below

Make my reservation for the annual ACM Dinner Meeting on May 22:

Name: __________________________________________  Phone number: ________________________

Menu choice:

___ Charley’s Special Steak     ___ Broiled Flounder    ___ Chicken Cordon Bleu

Dinner includes salad, baked potato, vegetable, house dessert, and coffee/tea.
Send your check for $20 per person and this form to:

	PRINCETON CHAPTER OF ACM
	Treasurer, P. O. Box 1324, Princeton NJ 08542