Dissonance: The Broad Prize .vs. Massachusetts AYP

( And the answer is - the Broad Prize got it right, and Massachusetts AYP formula requires arbitrarily high performance gains for a school to be labeled "adequate")

A couple other blogs have posted about the Broad Prize.

Boston won the Broad Prize this year, but 65% of schools in Boston were identified by the state as failing to make Adequate Yearly Progress.

I've demonstrated that only a handful of schools in Massachusetts matched the scores that will eventually be mandated by the state to avoid being labeled as inadequate schools and subjected to No Child Left Behind sanctions.

Today, I decided to ferret out why 65% of Boston's schools have failed AYP, while only about a quarter of schools across the state have failed.

To do this, I looked closer at Massachusett's improvement_targets.xls spreadsheet. This spreadsheet shows CPI scores for each subgroup in each school across the state, as well as "Improvement Targets". The improvement targets for each subgroup are (I am told - the actual formulas are not included in the spreadsheet - only current year data is included. Wonder why?) require that subgroup to improve from the baseline scores to 100% proficient by 2014.

That means, for example, if baseline score in 2002 was 60, then the school is mandated to improve by 40 points over 12 years, or 3.3 points each year.

OK, so we are told about 25% of schools across the state failed to make AYP. But if you calculate whether a school actually made it's improvement target, I find that 1216 subgroups out of 5054 made it for English Language Arts; and 1287 out of 5054 made it for math. 522 out of 5054 made BOTH.

So about one out of ten subgroups made it across the state. That is, NINETY PERCENT OF SUBGROUPS FAILED TO MAKE THE PROGRESS MANDATED BY THE FORMULA.

So how come only 25% of schools are identified for improvement? After all, a school is supposed to be identified for improvement if any subgroup fails either math or ELA.

And the secret is that in addition to the individual school's improvement target, there is also a state threshold. If your CPI beats either, your school is called "adequate" for that year. The state thresholds started at 53.0 for math, and 70.7 for english, in 2002. Both hit 100 in 2014. So, as of spring 2006, the state cut scores were 68.7 for math, 80.5 for english.

What this effectively says is that any school that scored above the state average in 2002 has been given a "get out of AYP Free" card for a few years. These schools can be deemed as making adequate progress even if their test scores don't improve - for the time being. But meanwhile, schools that started at or below the state average have to improve by leaps and bounds every year.

So - there you have it. Why do 65% of Boston's schools fail AYP while only 25% fail statewide? The answer is that 90% fail statewide, if you use the calculation that actually requires the school to make progress.

Without any knowledge about the Broad Prize's selection criteria, we can say that it is more likely that the Broad Prize uses a more rational and more accurate means of evaluating excellence and improvement of urban public school systems than No Child Left Behind does.