When Hell Freezes Over
Several years ago when the Testing Monster grew claws and teeth, our school administration adopted a data analysis package called the Cox Model. Our staff examined complicated matrices of “disaggregated” data based on test scores and racial, ethnic, language, and other demographic categories. I was uncomfortable throughout this process, a required staff development exercise. It felt like racism, institutionalized, and I refused to say anything after my stomach became so knotted that I could only rant and groan in the small group discussions.
When we looked at the test scores of our students, I noticed that all of my below-proficient-scoring students had histories of domestic abuse. I raised my hand and asked, “Will the administration allow us to include Domestic Abuse as a demographic category?” because it seemed like a significant variable. The whole staff was silent. My principal waited a moment for the question to sink in and diplomatically replied, “No.” The meeting continued.
This little story was prompted by a recent post in the Change Agency blog called, Data Analysis and the Four C’s of Change. Miguel applied the graphic that Stephanie used to his post, Multiple Measures of Data, asking about it’s applicability to the read/write web. Stephen Downes commented with a challenge to this sort of data analysis:
Of course, once you admit these dimensions of measurement, what is to argue against a variety of other measurements - nutrition intake, for example, local crime rate, perhaps, or per-student computer budget - into the same sort of calculation.
My experience exemplifies the problem that Stephen noted. The permissibility (or not) of specific data in school reform initiatives is a bold imposition of power and politics on children and their teachers. Data-driven recommendations for change will be useful when we all agree on whose data counts, and when we reach consensus on appropriate interventions. And that will happen when hell freezes over, an event that appears increasingly improbable.
Stephanie wrote,
Doug,
Both you and Stephen make good points about the use of data in decision-making. In my post — and in my presentation — I am advocating for much more in-depth data analysis that goes far beyond the usual “surface” analysis that typically occurs. I too am sick and tired of the typically data analysis that goes on in our schools and our districts. Simply disaggregating test scores into demographic groups tells us nothing about what really needs to occur to help these students learn.
If we are going to improve learning for all students then we do have to pay attention to the details — including those off-campus distractors that do have an impact on a student’s ability to focus on what is happening in the classroom.
I do have my “when hell freezes over” moments when everything feels hopeless — but, with the blog posts on this topic and my work with educators face-to-face, I am continuously pushing for better data-driven dialogue that takes us to the heart of the issue — so we can “treat the cause” and not just the “symptoms”.
Stephanie
Link | August 11th, 2006 at 7:14 am
Brian Crosby wrote,
Obviously the demographic categories are set to support NCLB which assumes that schools are ultimately in control of all factors related to student learning - I’m not sure that schools are in control of even half of the factors related to student learning, especially when NCLB pushes us (intentionally or not) to teach curriculum and not children and their actual needs.
Link | August 11th, 2006 at 7:55 am
Doug wrote,
Stephanie, the analytical model you presented is preferable by far to the conventional approach. But a model shouldn’t be applied without an acknowledgement of its design limitations. If I left the impression of hopelessness, that wasn’t my intention. I am hopeful. I hope that the insanity of administrative myopia will one day be overruled by its failure to produce a meaningful difference. I hope that teachers assume moral responsibilty for making substantive decisions in their classrooms. I am hopeful that the data-driven dialog won’t have to include discussions about “off campus distractors” like child abuse, racism, or poverty. As Brian points out, treating the cause is a discussion that has to happen away from school. But it’s easier to farm that discussion out to school administrators who seem all too willing to make program adjustments that ignore the mountain of evidence kids bring with them when they come to school early for their free breakfasts.
Link | August 11th, 2006 at 9:30 am
Brian Crosby wrote,
Right on Doug - the people responsible for NCLB (or at least too many of them) needed to find someone or something to blame that they could seemingly do something about - schools and teachers just not doing the right things or not trying hard enough (because they are not accountable) is something we can work on by raising accountability - the general public can understand that and then those in power seem to be doing a good job. Dealing with the real culprits - poverty, child neglect and abuse, racism and the rest are EXPENSIVE and not easy to deal with and don’t fit with a cut taxes - less government approach so which approach was picked?
I don’t think ALL the discussion about treating the cause should happen away from school - I wouldn’t mind helping to deal with the underlying problems outside of school if it was agreed they were a MAJOR contributor to underachievement (curriculum-wise) and making changes in kids (and families) lives were seen as a step in their educational progress. 10 years ago at the very school I teach now we felt good about those students that we worked hard with just so they had the self respect and discipline to get through the day without being a negative influence on themselves and everyone around them and begin to engage and learn - they were behind in school, but now at least they could begin to catch-up - now they are able to do school without all the garbage in their lives that held them back - we used to celebrate that! Now … is that on the test!!!???
Link | August 11th, 2006 at 10:39 am
Stephanie wrote,
Doug —
I agree with you on all points. And I didn’t think you were expressing an opinion of hopelessness — I took that as an expression of the frustration that so many of us feel in this current climate where policy makers and administrators are looking for quick-fixes. I believe it’s a miracle that we can all maintain some level of optimism or hopefulness that we will see (and maybe help create) real improvement in our educational systems.
As for the analytical models — you are correct in pointing out the limitations. I’ve yet to see any model that doesn’t have some kind of limitation. My purpose for presenting it and working with it stems from my frustration of seeing (literally on a day-to-day basis) excessive data analysis done outside of any kind of framework other than just “how did all of the sub-groups score on the test and which objectives are they missing?” This is usually followed by the decision to load the curriculum calendar with drill and kill instruction over those objectives that were the most weak. And all of this is done without consideration for other factors that may have had an impact on those test scores, which leads to decision-making that doesn’t address the underlying issues (some of which are in educators hands).
I also agree with the points made by Brian. The process of educating children is a very complex process and there are so many variables that affect our ability to help our students learn — and many of those variables are out of teachers’ hands. If we are to achieve the stated goal of leaving no child behind, then the effort has to become a community-wide goal that involves everyone — and simply analyzing test scores to death is not the solution.
Stephanie
Link | August 11th, 2006 at 6:08 pm
Doug wrote,
We do need to work on moving the national discussion toward a more holistic vision for schools. The media may be starting to “get it.” Take a look at this article in the New York Times. The model you’re using might come in handy for me eventually. I bookmarked the post, and I plan to share it with my principal. My reservations with analytical tools like this is that they’re often presented without explicitly acknowledging their limits. We need to consider our aims as well as the means.
Link | August 11th, 2006 at 9:07 pm