Testing Our Credulity Again

In this morning's NY Post a former senior analyst at the Board of Education lays out the case that the fraudulent state test scores were even worse than we had first thought: "New York's "test mess" is worse than even the avowedly reformist state education leaders have acknowledged -- and it may not be over yet, either. A close look at the data (some of which became available only via the Freedom of Information Law) strongly suggests that the exams created each year by CTB/McGraw-Hill -- which purportedly measure the math and English proficiency of 1.2 million New York students -- are fundamentally flawed. That means that even Regents Chancellor Meryll Tisch and state Education Commissioner David Steiner's "recalibrating" of the scoring can't fix the problem."

It seems that there are two distinct parts to the math test, one multiple choice, and the other reasoning involving calculation, and the two parts have been way out of synch: "Like most such tests, CTB's exams contain both multiple-choice and constructed-response questions. The latter ask students to produce a response, for example showing how they solved a math problem or writing answers to express their understanding of reading passages. Constructed-response items take more time and money to administer and score -- but educators generally believe these questions measure a higher order of knowledge and thought than multiple-choice items, which kids typically find less challenging. Yet results from both types of questions should point in the same direction -- that is, if this year's 4th graders do markedly better on the math multiple-choice questions than they did the year before, then they ought to improve on the math constructed-response items, too."

But did they? Not so much: "In other words, on well-developed tests, the results on both types of questions are in harmony -- pointing in the same direction and nearly parallel from one year to the next. After all, each is supposed to tap a different level of knowledge of the same subject. Performance should move in a synchronized way. That's exactly the pattern shown on the National Assessment of Educational Progress -- nationally and in New York. The "nation's report card" uses both types of items to measure reading and math proficiency -- and the performance of New York kids on both is strikingly consistent over time. Not so, the results on the state exams."

This means that these flawed tests can't simply be recalibrated and a full scale investigation is called for: "Any testing professional should recognize this as an alarm bell: Something is seriously wrong with these exams. (And it is the tests, not the students or anything else: Again, the NAEP exams, covering the same areas, do not show these bizarre divergences over time.)...What's needed is an independent probe of the testing program, one with sweeping authority to investigate the role of Education Department officials, CTB measurement specialists and the state's technical advisers in all aspects of the program."

But what this also means-and New Yorkers are getting wise to the fraud-is that the Bloombergistas scammed us when they were huffing and puffing about miraculous test increases: "Larger gains were usually made on multiple-choice items than constructed response. This boosted the overall score -- leading to press releases and headlines that suggested everything was improving. Worse, data that contradict that storyline went undisclosed: The public didn't see separate analyses of constructed-response scores."

Now we know that memory fades, but our memory about the Bloomberg education pledge-and the hoo ha about graduation rates-remains strong; and DOE's constant trumpeting would be enough to cure us of any onset of senile dementia: "After years of near stagnation, our reforms have increased the graduation rate each year since we’ve been in office—and I’m so proud to see that graduation rates are up again this year,” said Mayor Bloomberg. “This year, English and math scores went way up, schools got much safer, and many more of our high school seniors have earned their diplomas. This is a great day for New York City schools.”

But in the end, we were simply taken to the cleaners; underscoring the age old maxim that, while figures don't lie, liars figure.