Borderland

Knowing what works in education doesn’t interest me as much as knowing how something works, or what purpose something has. In my experience, the right tool for the job is what matters most, and I’ve noticed that in the last several years we’ve been sold a lot of do-everything programs. They call them comprehensive, which to me means bloated and complicated. I end up using bits and pieces, always thinking about how to adapt program materials to the needs of the kids in the room.

Richard Allington in Ideology Is Still Trumping Evidence said that effective teachers aren’t:

…focused on doing things “right” but were dedicated to doing the right things….They didn’t necessarily reject commercial instructional packages or the directions that invariably accompany them, but they were rarely observed actually following such advice with any fidelity. Instead, they took their cues from the children they were teaching.

One of the reasons that I’m generally skeptical of educational research is that it invariably doesn’t work “as advertised,” which I assume is because either the kids haven’t read the teacher’s manual, or the publisher hasn’t met the kids. Allington explains this is because education research focuses on what works “generally,” but “no study has ever identified an educational treatment that has worked effectively for all participants.”

The other reason I’m skeptical of educational research is that I’ve participated in several studies, and (I confess) I don’t always follow the plan. I try, but things come up, or the “treatment” doesn’t fly, and we have to improvise. Of course I report those things. Mostly.

What Doesn’t Work
Dick Allington’s comment on the previous post sent me on a little journey into What Works land. He mentioned a report on Reading Recovery which found positive effects for the program, differing from what Louisa Moats and other whole language critics charge.

When I first saw the What Works (WWC) site, I wasn’t sure what level of bullshit filter I should read the reports with since it’s run by the US DOE Institute of Education Sciences (IES). The US government’s advocacy for NCLB, and Reading First, which has been having political problems lately, had me wondering about the objectivity of any recommendations on the What Works site.

Education Week reported the Reading Recovery story yesterday. The WWC found that Reading Recovery had “postitive effects.” Education Week commented, “That’s high praise from the clearinghouse, which critics have dubbed the “nothing works” clearinghouse because so few education studies have met its strict standards of evidence.” And they included a quote from the WWC:

“Our job is not to weigh in on whether Reading First had the right curricula or not in the programs that districts have chosen,”….“We’re simply giving people research facts so they can decide on their own how much weight they want to put on the findings and make their own judgments.”

To me, this quote is more interesting than the finding about Reading Recovery. I’ve still got my crap filter set to high. In looking around for information about the WWC itself, I found an article written by Alan Schoenfeld, a mathematics education researcher who has had some things to say about the Math Wars, which I wrote about a few months ago.

Schoenfeld wrote an article called What Doesn’t Work, about the WWC. And they responded. He described problems he recognized using meta-analytical methodologies to evaluate curricula with dissimilar student performance objectives (as in, skills vs. problem-solving approaches). “Tests that focus on only a subset of the desired range of performance can give misleading results,” he said. He observed anomalous results in some cases, indicating that tests may not be equally sensitive to the full range of purposes built into various programs, possibly producing either “false positives” or “false negatives” in research findings.

He wanted to publish his views on these limitations of the research in a WWC publication, and was opposed by the IES, which ultimately lead him to resign from his “senior content advisor” position, claiming “WWC is at minimum complicit in an act of censorship, having acceded from the beginning to IES requests to remove important material from its reports.”

I’m not pointing to this as any kind of scandal. But it does sound like the government is defending it’s evidence-based turf, not wanting to admit that there may be gray areas in their research methodology. Shoenfeld tells a good story.

Nobody in their right mind would conduct real science experiments on kids without also planning on going to prison. The government calls the studies they’re doing scientific, but they’re really just statistical analyses. Education research isn’t science. It’s social science. But, of course, social science doesn’t have quite enough of an authoritative ring to justify massively disruptive policy decisions. As a sales pitch, calling these government reports scientifically-based research does seem to be working. As a means of improving educational outcomes, though, it’s all snake oil.