Donate today!
view counter

Philly district suspends school rating system, seeks fix

By the Notebook on Nov 1, 2012 06:59 AM

by Benjamin Herold, for NewsWorks, a Notebook news partner

The Philadelphia School District revealed Wednesday that its system for rating schools is faulty.

The District has suspended use of its "School Performance Index," or SPI.

District leaders are now seeking outside help to fix the complicated formula that converts more than a dozen factors into a single score given to every public school in the city, including charters.

For the last two years, SPI scores have been used to help guide a wide range of major decisions, including which schools should be closed down or converted into charters. It has also been used to evaluate charters' bids for renewal or expansion.

Leaders of several of the city's charter schools have long taken issue with the index.

"We are, at this point, confident that there were some mistakes made," said new District Deputy Superintendent Paul Kihn. "We honestly don't know how extensive the problem is."

Kihn said the issues with SPI stem from human error in how the accountability measure was calculated – not from faulty data resulting from cheating on state standardized tests.

"They are two different matters," Kihn said.

A state-commissioned analysis of results from 2009-11 found evidence of widespread cheating at dozens of schools across the state, including 53 traditional public schools and three charters in Philadelphia. For the spring 2012 testing, after tight new test security measures were put in place, scores at many of those schools dropped dramatically – in some cases by 50 percentage points or more.

"I am not prepared to comment about the cheating matter in relation to this," Kihn said.

A far-reaching tool

The District developed its School Performance Index in 2009, when Arlene Ackerman was superintendent. The formula boils down either 13 indicators (for elementary and middle schools) or 17 indicators (for high schools) into a single score of 1-10, with 1 being the best.

For all schools, indicators relating to student scores on the Pennsylvania System of School Assessment (PSSA) exams are weighted most heavily, and measures of parent, student and teacher satisfaction also taken into consideration. High schools' success in preparing students for college and careers also factors in.

In an interview Wednesday, Kihn acknowledged the central role that SPI has played in the District's Renaissance Schools initiative, facilities master planning process, and decisions regarding charter schools. But Kihn stressed that the rankings were never "used as the sole determinant for any decision that we have made."

The measure will be disregarded altogether as District officials target several dozen more schools for possible closure by next fall, he said.

"Going forward, we've made the decision that we will not use SPI at all in the facilities master planning process."

Transition a factor

The District first became aware of a potential problem with its performance index in May, Kihn said. In response, officials conducted an internal three-month investigation that concluded in late August. Action was not taken earlier because of the District's recent change in leadership, he said.

Kihn was not specific about the nature of the errors and how they may have impacted the reliability of the school’s rating. But it is clear from the District’s detailed online explanation of SPI that the calculations rely on complex statistical analyses.

In addition to looking at the overall percentage of students in each school that achieved proficiency on standardized tests, it used student-level data to calculate how well the school helped individual students make significant progress from year to year – the so-called “growth model.”

SPI also relies on the individual student test-score data to determine whether schools were able to narrow the racial and ethnic achievement gap between White and Asian and Black and Latino students.

"Because [incoming Superintendent William Hite] and I were just transitioning in, a decision was made to hold on taking any action until we could be briefed and make the decision about how to proceed," Kihn said.

The District's Office of accountability remains responsible for developing and maintaining the School Performance Index.

In August, the School Reform Commission approved the termination of Daniel Piotrowski, then the District's executive director of accountability and assessment, effective July 14.

Neither District officials nor Piotrowski would comment on whether his termination was connected to the problems with SPI.

Earlier this month, NewsWorks and the Philadelphia Public School Notebook reported that Piotrowski, who also served as the head of the District's testing security program, had called in March for a full investigation of the testing procedures at Wagner Middle School, where he had reported more than a dozen testing violations. Piotrowksi was removed from Wagner, and other officials overturned his judgment that an investigation was warranted.

Charter concerns

Some charter operators have criticized SPI since its inception, claiming the District's numbers were not trustworthy and criticizing the formula.

"I'm glad that they recognized [the problem]," said David Hardy, the CEO of Boys' Latin charter in West Philadelphia.

But Hardy, who has long advocated for an accountability measure that puts more emphasis on students' postsecondary outcomes and less on their test scores, said his issues with the performance index go beyond a "math problem" resulting from faulty calculations.

"What they need to do is change the types of things they're measuring and the weight of their measurements and take a broader view of success," Hardy said. "In an environment of school choice, parents have to have [good] information so they know what they're buying."

Kihn acknowledged such concerns and said he personally notified leaders in the charter community of the problems with SPI on Wednesday morning.

Kihn also said he recently received a proposal for how to improve the performance index from a working group of the Great Schools Compact committee, which has convened District officials and charter leaders to examine the issue for months.

"We are certainly taking those very seriously as a set of thoughtful recommendations," Kihn said.

Most immediately, though, the District is looking for an outside vendor who can recalculate SPI scores from 2009-10 and 2010-11, and calculate 2011-12 scores for the first time.

Kihn said that he hopes to receive bids by mid-November and that the work will take no more than a few months. All results and analysis of SPI in the future will be publicly posted to encourage transparency, he stressed.

Making recommendations about how to improve the measure will be optional for whichever vendor is selected, Kihn said.


Additional reporting by Notebook contributing editor Dale Mezzacappa

This story was reported as part of a partnership in education coverage between WHYY/NewsWorks and the Notebook.

Click Here
view counter

Comments (13)

Submitted by Annonymous (not verified) on November 1, 2012 8:35 am
So, Mr. Kihn, how then will you determine which schools to close if performance is not being considered? How can you turn any District school over to a charter? Since the so called "Great School Compact" is leading this effort, we know the outcome. It is to promote private and charter schools while draining the life out of District schools. And who will pay for this Mr. Kihn? Is it another Nowak/William Penn Foundation/etc. behind the scene move? Might as well call is the Universal / Mastery School District.
Submitted by Joe (not verified) on November 1, 2012 6:35 pm
Of course, you are exactly right, just another card being played from the bottom of the deck and all clear thinking people know it. If it weren't so immoral and anti democratic, it would be funny but since it is so cold blooded and corrupt, it's a lot of things but funny isn't one of them.
Submitted by Anonymous (not verified) on November 1, 2012 7:42 pm
Even us lowly classroom teachers knew that the SPI was faulty when David Weiner introduced it. I find it interesting that they want to go back 3 years and recalculate the index. Hmmm, you can make the numbers say anything you want if you rearrange them correctly. It will make it much more convenient to justify whatever decisions have already been made
Submitted by Anonymous (not verified) on November 1, 2012 7:42 pm
Lies, damned lies, and statistics...
Submitted by Rob (not verified) on November 1, 2012 8:45 pm
What about the schools that have become promise academies or have been turned over to private operators because of SPI. Hope Moffett redux
Submitted by Anonymous (not verified) on November 1, 2012 8:12 pm
What about resources and teachers assigned to schools because of the SPI.
Submitted by Anonymous (not verified) on November 1, 2012 10:13 pm
Where are they getting money for another contract that doesn't directly impact child's learning at all? I thought we were in a budget crisis.
Submitted by Anonymous (not verified) on November 2, 2012 9:23 am
Will King High School be re-evaluated? What about Germantown? "440" is not only financially bankrupt, they are ETHICALLY and MORALLY bankrupt.
Submitted by Anonymous (not verified) on November 2, 2012 12:17 pm
Yes they are and they have been ethically and morally bankrupt for many years now.
Submitted by Education Grad Student (not verified) on November 2, 2012 9:35 pm
The SPI was far too simplistic of a measure of a school's quality. How it even came to be used is beyond me. I wonder if an elected school board would have adopted the SPI.
Submitted by Sherry Gearheart (not verified) on January 22, 2013 1:41 pm
School Performance Index (SPI) score is not just enough. Paul Kihn said, there is problem with accountability with SPI, so it needs to be sorted out as soon as possible.
  • You'll find on line pokies, roulette, video poker in addition to large jackpots in a online pokies that is specially focused intended for on the internet gambling gamblers with big cash signup bonuses as well as jack pots.
Submitted by Erica Harrell (not verified) on January 25, 2013 1:36 pm
School Performance Index (SPI) score is not just enough. Paul Kihn said, there is problem with accountability with SPI, so it needs to be sorted out as soon as possible.
  • Registering to receive a shady wager site on-line might be tempting when they're presenting an incredibly tough to refuse reward which enables you to enjoy by making use of a ton of no value pounds. Numerous are fast to think about your money, whilst not so joyful about forking out a payment of $30,000 any time you strike a winner.
Submitted by Mark Jacobs (not verified) on October 23, 2015 5:03 am
School Performance Index (SPI) score is not just enough. Paul Kihn said, there is problem with accountability with SPI, so it needs to be sorted out as soon as possible.
  • You'll find on line pokies, roulette, video poker in addition to large jackpots in a online pokies that is specially focused intended for on the internet gambling gamblers with big cash signup bonuses as well as jack pots.

Post new comment

The content of this field is kept private and will not be shown publicly.

By using this service you agree not to post material that is obscene, harassing, defamatory, or otherwise objectionable. We reserve the right to delete or remove any material deemed to be in violation of this rule, and to ban anyone who violates this rule. Please see our "Terms of Usage" for more detail concerning your obligations as a user of this service. Reader comments are limited to 500 words. You are fully responsible for the content that you post.

Follow Us On

Read the latest print issue

Philly Ed Feed

Recent Comments


Public School Notebook

699 Ranstead St.
Third Floor
Philadelphia, PA 19106
Phone: (215) 839-0082
Fax: (215) 238-2300

© Copyright 2013 The Philadelphia Public School Notebook. All Rights Reserved.
Terms of Usage and Privacy Policy