Sniffing out test irregularities in Chicago

by Benjamin Herold

The model for detecting cheating developed by Brian A. Jacob, a professor at Harvard University, and Steven Levitt, a professor at the University of Chicago, uses complex statistical analysis to answer two basic questions. Do any classrooms have unexpected fluctuations in their test scores? Are there suspicious patterns to the test responses of any classrooms?

Jacob and Levitt's model identifies whether a group of students from a particular classroom have identical answers on consecutive questions in the same portion of the test, and whether a group of students consistently miss easy questions while consistently answering difficult ones.

If a classroom shows high gains and no unusual patterns, the gains are most likely evidence of good teaching. If a classroom shows high gains and highly unusual patterns of student responses, the hard questions really begin.

Does the same classroom, with the same teacher, show similarly high gains and highly unusual patterns over multiple subjects? Over multiple years? If the answer to these questions is "Yes," an investigation into the classroom is launched.

Chicago's investigation process included retesting of three groups of classrooms: those considered most likely to have cheated, those with large gains but no suspicious patterns of student responses, and a group of randomly selected classrooms. Classrooms in which there were gains due to good teaching sustained their gains, and randomly selected classrooms generally performed about the same on both tests. Classrooms in which high gains were lost, however, came under intense scrutiny for potential cheating.

Jacob commended the current CEO of the Chicago Public School system, Arne Duncan, for his willingness to take such a thorough look at the possibility of cheating on standardized tests. Jacob added, however, that many districts don't have either the capacity or the political will to take on this issue.

For many districts, said Jacob, "It's not clear why it's in their narrow self-interest" to risk possible exposure of cheating.