The Trouble With Data

Several weeks ago, I listened to our high school principal extol the virtues of data that they will use to "guide" every one of their decisions on how to run the school. They were already collecting buckets (my word) of data that would help them tailor the curriculum and teach to the needs of students. Decisions would not be based on anecdotal evidence, nor clouded by penchants to emotional responses. From the facts, the proper course would be clear. The data would keep the school on a course for continual improvement.

My ocular muscles got quite a work out that evening. I won't even cover the fact that his school has no ability to alter the curriculum. Neither does the district school board or, I'm beginning to believe, even DESE.

Touting the virtues of data seemed just a little too Madison Avenue for promoting the longitudinal database that is working its way into the educational landscape.

This data is something we want. It's something we need. It will make us better. Cooler. Whenever they ask for it, we should be only too happy to supply it to them. WE LOVE DATA!

I wonder what the principal would do with the data I have on my son. He struggled all last year in science, barely passing the course by the skin of his teeth. In fact, by the numerous test scores (data) he accumulated, it seemed clear that he was either not putting in the effort, or was just not able to grab the basic science concepts. This was so obvious, according to the data, that he was bumped off the advanced science track for the following year.

Imagine our surprise then when we received his MAP scores for science last week and found that his scale score put him 12 points above the minimum to be categorized as being in the top Advanced achievement level. According to the MAP folks this means he has a "thorough understanding of the content at this grade level." His Terra Nova score put him better than 80% of the students nation wide.

The MAP test is designed to assess each student's comprehension of the Show Me Standards and has been the benchmark against which teacher performance is measured in this state so we must assume that it is accurate, or at least close.

So here are two sets of well documented, publicly recognized data that are in complete contrast to one another. Decisions with long range implications were made based on at least one set of these data; that my son would no longer be on the honors track in science. The other set of data not only says he could hack honors science, but he might actually have a knack for it. Unfortunately, the decision about which high school science course he would take was made way back in February when scheduling requests were due and the MAP data was not available until 10 weeks into the new school year.

What conclusions can be drawn from the data? We could probably conclude that the difference is not due to test taking anxiety, otherwise both scores would be low. We could conclude that my son chose not to work in this teacher's class, but decided it was worth the effort to do well on the MAP test. I love my son, but I don't give him enough credit to think up that effort scenario.

We could guess that the teacher was not teaching to the test which, to some, may be a plus. My son could have performed poorly on the unique information she was teaching, but gleaned enough to pass the MAP test. This would mean, however, that she was trying to teach almost twice as much material as the other grade level science teacher. I'm all for pushing kids and striving for more educational exposure, but my son's grades (and those of many other parents of advanced students who had this teacher) would indicate that this teacher is not very good at covering so much material. Given that academic performance in this particular year pegs a child's long term high school course placement, this approach to teaching the subject should have been rejected.

It would be difficult, if not impossible, to get at the truth of this data at this point. No one is doing the type of data comparison or analysis that would be needed. Once a child leaves one school for another, nobody bothers to look at the data. The old school will choose to accept the MAP score because that data has actual cash value. The parents , unless they have another child rising through the ranks, will not push for an inquiry and simply be glad to shake the dust from that school off their shoes.

Contrary to what the high school principal claimed, the data used to determine course placement did not tell the whole story so proper conclusions were not reached. Nor will anyone ever get that teacher to adjust her lesson plans because no one is looking at the comparison of her kids' classroom scores versus MAP scores. Those kids are gone. The damage is done.

The more I read about the education machine in this country, the less I worry about these kinds of decisions. I quizzed my son prior to the science final and was satisfied that he knew the information. The MAP scores confirm that. The loss is only a year until he can get his confidence back. Then I am sure he could take on the honors track again if he chooses. The kids whose parents aren't paying attention, however, may be permanently lost to science.

So the principal will have to excuse me for yawing and rolling my eyes when he touted hiss school's reliance on data for their decisions. That is an addiction that will prove as useless or destructive as any other addiction.