Did state officials bury 2009 cheating report?

by Benjamin Herold on Jul 12 2011 Posted in Latest news

New state Secretary of Education Ronald Tomalis is “concerned” that a 2009 report flagged dozens of Pennsylvania schools for possible cheating – then languished for two years.

“Why the report wasn’t brought up in 2009, I don’t know,” said Pennsylvania Department of Education (PDE) spokesperson Tim Eller. “There’s no indication of what was done with that report two years ago.”

The “data forensics technical report” in question used statistical analysis to look for highly improbable test score gains and suspicious erasure patterns on statewide 2009 test score results on the Pennsylvania System of School Assessment (PSSA) exam. It was commissioned by PDE while the department was under the leadership of former Secretary Gerald Zahorchak, an appointee of former Gov. Ed Rendell. It was brought to light by a recent Notebook inquiry to state officials.

“I don’t want to judge prior actions, but it is concerning that [the report’s findings] were not addressed,” said Eller. 

Zahorchak, the current superintendent of the Allentown School District, said he could not recall any specific findings from the report being brought to his attention, and he could not recall any specific investigations that were initiated based specifically on the report. He did say, however, that in the cases where a school was flagged for multiple suspicious results, he was “certain that the proper investigations ensued.”

School District of Philadelphia officials, however, reiterated on Monday their stance that they were never given a copy of the July 2009 analysis.

“The District is confused as to why PDE leadership had decided not to share the report results,” said District spokesperson Jamilah Fraser. “It is important to know [that], to this date, PDE has not notified the District of any serious breach in our testing process.”

Fraser added that the District is willing to investigate any problematic findings.

Philadelphia is featured prominently in the report’s narrative, with 22 District schools and seven Philadelphia charters flagged three or more times for suspicious results in a single grade. 

Statewide, 60 schools were similarly flagged for multiple statistical irregularities. Over 200 are flagged for suspicious erasure patterns. One expert said the scope of the problems resembled the recently exposed cheating scandal in Atlanta.

But Eller said that while Tomalis is upset about the report’s findings, he does not believe that either Pennsylvania or Philadelphia has a widespread problem with cheating.

“I don’t think [Tomalis] is in that frame of mind at this point,” said Eller. “Right now, this is a report that shows some inconsistencies. Further investigations will be conducted.”

On Friday, the Notebook reported on the report’s main findings, which sought to identify schools that had some combination of improbable test score jumps, unlikely improvements in student performance levels, and unusually high number of wrong answers that were erased and changed to the correct answer.

A further Notebook review of the detailed erasure analysis done for the report found that 225 schools statewide – 44 percent of them in Philadelphia – had been flagged for highly suspicious numbers of “wrong-to-right” erasures. Although the “erasure analysis” flagged 88 District schools and 11 Philadelphia charters, numerous suburban schools, including the Elkins Park School in Cheltenham Township and Lower Merion High School, had suspicious erasure patterns in at least one grade.

The threshold for suspicion on the erasure analysis was so high as to make it almost impossible that the problematic wrong-to-right erasure patterns occurred purely by chance, said Andrew Porter, the dean of the Graduate School of Education at the University of Pennsylvania and an expert on testing.

“You have to ask, 'How could this happen except for cheating?’" he said.

At least 12 District and three Philadelphia charter schools were flagged for suspicious erasure patterns in both reading and math in every tested grade in their school.  Franklin S. Edmonds, Emlen, and Forrest elementary schools were flagged eight times apiece in the erasure analysis – in both reading and math for 3rd, 4th, 5th, and 6th grades.

“When [Secretary Tomalis] reviewed the report and saw some of the schools repeating themselves, he was really upset that nothing was done in the past,” said PDE spokesperson Eller.

The report also flagged individual students’ response sheets for improbable wrong-to-right erasure patterns. At Nebinger Elementary in South Philadelphia, 59 percent of 4th graders had their reading response sheets flagged for highly suspicious wrong-to-right erasure patterns. Half of Nebinger’s 4th graders also had their math response sheets flagged.

Porter expressed shock at the number of Philadelphia schools implicated in the erasure analysis – nearly one-third of the schools in the District. 

The results sound “awfully comparable to Atlanta,” said Porter, referring to the Georgia school system that has been rocked by last week’s results of an investigation that revealed widespread cheating. Porter conducted an earlier analysis of the Atlanta test score results, but officials buried his report, according to Georgia state investigators.

It appears that the 2009 Pennsylvania report, prepared by the Data Recognition Corporation (DRC), originally met a similar fate. Despite the scope of the report’s findings, it appears that PDE may not have shared the results or initiated any further investigations into possible cheating.

Former education secretary Zahorchak said he could not recall if the report was shared with individual districts across the state. 

Current PDE spokesperson Eller said that to his knowledge, the report was never shared, but that he could not confirm that definitively.

“I can’t talk about the past. I can only talk about what’s going on now with [the report],” said Eller. “We’re going to ask those districts [named in the report] to address those questions and concerns.”

School District of Philadelphia spokesperson Fraser said the District would be willing to reopen old investigations or start new investigations into test-score cheating from 2009.

“If PDE has questions about testing irregularities in the schools noted in the report, this District stands ready to assist them in any way possible,” said Fraser.

Fraser added, however, that doing so would depend on cooperation from PDE and would be challenging because of staff and student turnover at the involved schools in the two years since the analysis was conducted.

Monday night, Eller acknowledged those challenges.

“Obviously, we understand the concerns that are going to arise from a report that surfaced two years after the fact,” said Eller.

Penn’s Porter said that even now, an investigation is warranted.

“You can’t ignore it,” said Penn’s Porter. “We still don’t know who cheated, and we still don’t know for sure whether it was cheating. But at that level, it’s probably unlikely that it’s a school problem.”

No forensic data analysis was commissioned by PDE in 2010. Eller said Monday that Tomalis has ordered that the analysis be done for the recently administered 2011 PSSA exams. 

The results of that report should be publicly available by the first week of August.