Donate today!
view counter

KIPP's success significant but hard to explain, says study co-author

By Dale Mezzacappa on Feb 27, 2013 04:05 PM

A new study by Mathematica Policy Research found that students at KIPP charter schools showed greater gains in test scores in core subjects during their middle school years than their peers at traditional public schools. Brian Gill, one of the report’s authors, answered several questions about the implications and particulars of the KIPP study, shedding light on the significance of its conclusions.

The main takeaway from the study, according to Gill, is this: “KIPP’s impacts on student achievement are statistically significant, substantial in size, consistent across four academic subjects, consistent across the great majority of schools included in the study, and consistent even on a nationally normed assessment we administered that had no stakes for students. It’s pretty robust positive findings on achievement impacts. … In 15 years of studying educational interventions, I personally haven’t studied anything that had impacts this consistently positive.”

Impact estimates were done separately for each school, and although most schools had positive effects, Gill said, there was considerable variation in the size of the effects. But the researchers had an agreement that schools wouldn’t be identified in the report. That means how the two Philadelphia KIPP schools did compared to others or to the national sample won't be made known.

Gill noted that, because the methodology was largely a value-added one -- tracking individual student improvement, not overall proficiency rates -- it is possible for a school not to meet federal learning goals (Adequate Yearly Progress) and still show significant student growth. 

“Given the population," he said, "it wouldn’t surprise me if some schools were producing substantial positive effects and not making AYP.”

So what about the KIPP schools led to more rapid student improvement?

“We didn’t learn as much as we hoped to learn there,” said Gill. The researchers surveyed KIPP principals in an effort to distinguish the best performers from those that didn’t do as well. Gill said he didn’t find anything statistically significant, with a couple of exceptions.

“One was related to having a comprehensive system of high expectations for student behavior. Schools that had that tended to do better,” he said.

The other exception, related to school time, he said, was a little more complicated. Although all KIPP schools have days that are notably longer, the ones that had the longest school days actually did worse, according to Gill.

"This possibly says something about how far you can push the school day," he said. "At the same time, we found that schools that put more time into core academic subjects got better results.” Understanding this interplay completely, he said, “was not possible to tease out in our research design.” 

Many in Philadelphia and elsewhere believe that one reason KIPP performs well is that it doesn’t replace students who wash out, for one reason or another. The study found a similar attrition rate for KIPP and comparison schools -- about 37 percent. But Gill said that the study included all students who started at KIPP, regardless of where they ended up.

“That is not a problem in our analysis because we keep the kids and continue to count them as KIPP kids even if they left KIPP,” he said. “There is no way the results can be artificially inflated by losing those kids.”

On the other hand, he said there could be a more subtle "peer effect" due to more motivated students staying together. “That’s where the attrition and replacement is potentially relevant -- not to the validity of findings, but to their interpretation," he said. “But in that regard, it’s clear, we don’t see evidence that peer effect could explain any substantial part of KIPP’s effect.”

Finally, as part of the study, the researchers interviewed both those students who won the KIPP lottery and those who didn’t to try to identify any effects of KIPP on student attitudes and behaviors. They found, not surprisingly, that KIPP students do more homework. But the other finding here was surprising and the only negative note in the report.

“According to the students' self-reports, the KIPP kids are more likely to engage in kinds of undesirable behavior that lead to conflicts with teachers or parents.” The researchers are not sure why that might be, but Gill has a couple of guesses: KIPP students possess more self-awareness, or they simply have more time to get into arguments, given the longer days and additional homework.

The study did not follow KIPP students beyond their middle school years to see how well they do in high school or college.

“We don’t have data to follow the kids long enough for that,” Gill said. "We would like to do that at some point in the future.”

Click Here
view counter

Comments (12)

Submitted by anonymous (not verified) on February 27, 2013 9:18 pm
Hmmm... all those student gains and no water fountain guarantee in their contracts?! How it it possible!?!?
Submitted by Anonymous (not verified) on February 28, 2013 6:26 am
Submitted by Helen Gym on February 28, 2013 9:05 am

I have concerns about studies that arrive at broad conclusions without detail. "KIPP does great but . . .  we can't really tell you why or how or where specifically" is a little bit of an odd conclusion for a research study. KIPP is the largest or one of the largest school operators in the country. They operate a diverse system in vastly different parts of the country under a variety of conditions. It's a tremendous disservice and is somewhat intellectually dishonest to not parse out in what scenarios a school like KIPP flourishes and under what conditions it struggles. KIPP West has an SPI of 8, ranking it among the worst in the district, for example. 

Submitted by Anonymous (not verified) on February 28, 2013 7:27 pm
Thanks for speaking the truth!!!
Submitted by Education Grad Student (not verified) on February 28, 2013 8:37 pm
Helen, I disagree with your point that the authors arrive at broad conclusions without detail. I would be curious to hear you elaborate on your statement that there is a lack of detail supporting the conclusions. I disagree with your point about the "broad conclusions without detail" because I see a great deal of statistical analysis in this study. (I took a statistics course and a research methods course in college, so I can understand some of the stats in the study.) Someone with more of a background in statistics might be able to make a more educated comment on the merits of the research design and the statistical methods. Based on skimming through it, the authors used the control and treatment conditions (students who attended KIPP vs. students who were not selected in the lottery to attend KIPP). They go into great detail about the variables or factors they are studying (reading scores, math scores, special education status, length of school day, school-wide behavior management, attrition, etc.). Their sample size is impressive, with over 30,000 students among the different conditions. They include levels of significance, one and two tailed tests, and regression. Again, someone with a master's or PhD in statistics, educational measurement, or psychological measurement could make more definitive and reputable statements about the research design's strengths and weaknesses, as well as whether or not the data support the conclusions. One issue I do have with the presentation of the data is that p values (levels of significance) are often omitted from the narrative. It is standard practice in peer-reviewed publications to include the p values (e.g. p = 0.02) in the narrative as well as in tables or matrices. As I stated in another comment on this article, I would put more confidence in the findings if the study is rigorous enough that a reputable educational research journal will accept it for publication. There is a great deal of education "research" available that has poor research design. A prime example is the Measures of Effective Teaching research report (see a critique at With regard to test score, these scores are a common component feature in much of today's educational research. One final point I have is regarding informed consent? Did the researchers obtain consent from the students/parents to use the data in the study? I only skimmed through the study, but I didn't see any mention of informed consent. Education Grad Student
Submitted by Anonymous (not verified) on February 28, 2013 11:16 am
Actually, there are some fine and dedicated teachers at KIPP. And, in fact, they are treated with more respect than teachers in the district are about to be treated given the new contract proposals. However, we should always be a little careful when using test score gains to measure achievement. Not knowing how these students do in high school, and then in college, raises a variety of questions. We already do know from some research (that KIPP has been open about) that doing well on standardized tests does not necessarily translate into success in college. In fact, as a long time teacher, I would argue that what's necessary to get through some of these tests has nothing to do with the kind of thinking and learning required in higher education. Let's not ever forget, as well, that charters like KIPP can spend more money per student than the district; the private donations and funding are not irrelevant.
Submitted by Education Grad Student (not verified) on February 28, 2013 7:48 pm
I will be able to put more faith in Mathematica's findings if this research is published in a peer-reviewed journal. There are so many reports that come from even reputable organizations like Mathematica and the RAND Foundation. Who knows whose giving them money. Also, the gold standard of research is to have it in a peer-reviewed publication, especially one of the American Educational Research Association's journals.
Submitted by Ken Derstine on March 2, 2013 8:07 am
There is an in depth analysis of the KIPP Study at School Finance 101. "The Non-reformy Lessons of KIPP " is at:
Submitted by Annonym (not verified) on March 2, 2013 9:50 am
Thank you for sharing the article - "But the reality is that what underlies the KIPP model, and that of many other “high flying” no excuses charter organizations, are a mix of substantial resources, leveraged in higher salaries, additional time – lots of additional time (and time is money) and reasonable class sizes, coupled with a dose of old-fashioned sit-down-and-shut up classroom/behavior management and a truckload of standardized testing. Nothin’ too sexy there. Nothin’ that reformy. Nothin’ particularly creative."
Submitted by Ms Pat (not verified) on March 2, 2013 9:23 am
I appreciate that they looked at individual student gains, not system-wide proficiency rates. Most teachers would agree that when everyone works hard, the student wins. AYP is not relevant to an individual student's progress. All school should use this type of data, rather than AYP; then we would see how much our public school students improve and how terrific they truly are!
Submitted by Joe (not verified) on March 2, 2013 10:50 am
Listen, I hate to throw a monkey wrench into your soup but how can anybody trust this data ??? There is NO objective analysis of something over which you have no control. They tell you what they want you to hear. Is that objective, I think not. Everything is done intramurally so everything is up for debate. Hitchens once said, "Assertions made without proof can be dismissed immediately." Actually, I butchered his quote but I was close enough to make the point.
Submitted by Joe (not verified) on March 2, 2013 10:37 am
"What can be asserted without proof, can be dismissed without proof." Chris Hitchens. Listen, all charters handle everything intramurally so everything they say is up for debate. There is no transparency so no objective conclusions can be drawn by objective people. Sorry, I ain't buyin none of it.

Post new comment

The content of this field is kept private and will not be shown publicly.

By using this service you agree not to post material that is obscene, harassing, defamatory, or otherwise objectionable. We reserve the right to delete or remove any material deemed to be in violation of this rule, and to ban anyone who violates this rule. Please see our "Terms of Usage" for more detail concerning your obligations as a user of this service. Reader comments are limited to 500 words. You are fully responsible for the content that you post.

Follow Us On

Read the latest print issue

Philly Ed Feed

Recent Comments


Public School Notebook

699 Ranstead St.
Third Floor
Philadelphia, PA 19106
Phone: (215) 839-0082
Fax: (215) 238-2300

© Copyright 2013 The Philadelphia Public School Notebook. All Rights Reserved.
Terms of Usage and Privacy Policy