The Data Dagger: Why Policymakers Need Student Stories

Test scores are a means to an end. For many power players in education, test scores have become the end in themselves, with devastating results.

Part of our job as teachers is to remind policymakers that children are far more than the sum of their data.

 

 

 

A couple months after starting at a new school, my students took a math test and scored incredibly well: a class average of 87% in our school full of high-poverty English Learners, where the school average was below 50%.

People kept coming up to congratulate me, and I kept trying to dissuade them with two simple facts:

  1. Only four of my 25 students were tested. The other 21 were low enough in their level of English proficiency that they were exempt from the test.
  2. One of those four students, who scored 100%, came to my class three days before the test. Whatever he may have learned in the three math classes I had taught him had almost nothing to do with his high score.

People thought I was being humble. I wasn’t. Like most teachers, I know which data means something significant about my students’ growth, and by extension, my teaching. This data didn’t fall into that category.

I also knew that a month or year later, I could be holding the other end of that skewed data dagger. Another sample size of four or five students might have dramatically low scores, and that kid who walked in my door three days before the test might be far below proficient instead of far above.

 

Stories Matter

Policymakers, researchers, and administrators sometimes sneer at individual stories. Anecdotal evidence isn’t considered to be evidence at all.

I understand their skepticism. We did have an era when it was too easy for teachers to say of a student, “But she has made so much growth,” even if there was no data to back up that heartfelt claim. There’s some value to taking the emotion out of the data, to looking at cold hard evidence and admitting when our teaching isn’t working for some kids in the class.

But the pendulum has swung so far toward blind faith in data, any data, that even powerful people like Secretary Duncan and the Council of Chief State School Officers have begun to weigh in with warnings. They have realized the truth related by an Indian educator: when you relentlessly weigh the elephant, but don’t spent much time feeding it, it begins to starve.

Secretary Duncan gave a startling address early this school year. He has begun to listen what teachers have told him for the past six years:

“Testing should never be the main focus of our schools. There’s a whole world of skills that tests can never touch that are vital to students’ success. No test will ever measure what a student is, or can be. Yet in too many places, testing itself has become a distraction from the work it is meant to support.”

He also said this: “No school or teacher should look bad because they took on kids with greater challenges. Growth is what matters. No teacher or school should be judged on any one test, or tests alone.”

Part of the problem is that we’re expecting tests to do too much—measure the effectiveness of entire states or districts, assess individual teacher performance, help teachers improve our practice, provide meaningful information to parents, motivate students to work harder.

But another problem is that policymakers often take a “see no evil, hear no evil” approach to flaws in standardized tests. Teachers are quick to be branded as oppositional excuse-makers if we bring up problems with a given test or the way it is being used.

But case studies of individual students and teachers matter. School primarily exists for the well-being of students and their families, not for politicians, textbook companies, or think tanks. If parents and teachers bring forward a single case of an individual student being harmed by unintended consequences, chances are good that there are many more students in the same situation.

Examples abound, like this unethical but common post-NCLB strategy: To raise the number of proficient students, focus extra tutor time on “bubble kids” just below proficiency. The students desperately far below grade-level are unlikely to catch up in a single year, so investing time in them is not a good strategic bet.

 

To Live the Lives They Dream

The single greatest problem in education I see is that we have lost sight of the true purpose of school: Helping students to someday live the lives they dream of. We have reduced school’s purpose to helping students increase their test scores.

Some tests matter. Learning to pass those tests can be a means to the end of a student’s eventual goal, like getting into a good college or pursuing the career she wants. But the test scores are a means to an end. For many power players in education, ranging from Commissioners to legislators, test scores have become the end in themselves, with devastating results.

Abilities that most matter for success in college and life—higher-order thinking, ingenuity, collaboration—have been ignored or diminished, often for negligible bumps in test scores.

When you consider the toll NCLB has taken on students’ love of learning, it’s discouraging to remember how far short we have fallen of the goal set in 2002 that 100% of students would be proficient by 2014. According to NAEP, proficiency rates are still below 50% in Reading and Math for 4th and 8th graders in every racial group except Asian-Americans. (See NPR’s recent article for a damning look back at the law’s legacy.)

Parallels between teaching and the medical profession can be illuminating. Both professions are glutted with data, yet they come down to human outcomes for individual students/patients.

I was struck by an article in Sunday’s New York Times titled “Why Doctors Need Stories,” written by a psychiatry professor. The author could just as easily have been writing about the data-fixation in education.

Consider these lines:

 

“We have entered an era in which a narrow, demanding version of evidence-based medicine prevails. The inclusion of a single anecdote in a research overview can lead to a reprimand, for reliance on storytelling.”

“We need storytelling to set us in the clinical moment, remind us of the variety of human experience and enrich our judgment.”

“Data are important, of course, but numbers sometimes imply an order to what is happening that can be misleading. Stories are better at capturing a different type of ‘big picture.’”

 

In teaching, like medicine, data matters. We can’t become so fixated on the experience of our own students that we ignore the broader picture. Professional judgment demands that we see the forest, not just the trees.

But policymakers are often guilty of the opposite fixation: They can’t see the individual trees for the forest. If test scores in a state have slightly increased, but the experience of schools as reported by students, their families, and teachers has actually gotten worse, there’s a problem. Maybe it’s a problem with the test itself, or maybe it’s a problem with the way that test is being used.

Part of our job as teachers is to remind policymakers and administrators—our principal, our superintendent, our state legislature, our state department of ed—that children are far more than the sum of their data.

I can tell you that my six-year old daughter weighs 47 pounds and is 3 feet 9 inches tall, but those numbers don’t begin to encompass who she is. They won’t tell you anything about her strengths, interests, and needs, and they won’t tell you how good or bad a job I’m doing as her parent.

Most data do mean something, but they don’t mean everything. If we forget that, we’re lost, and we will have failed the students in our care.

 

Note from Justin: I just found out that a grant I wrote to scale up home libraries for about 1,800 students in our district is one of 15 finalists in the nation. Five projects will receive $100,000; for the home library project, this money would put about 30,000 books into the hands and homes of kids living in poverty. The five winners are chosen solely based on the number of votes they receive online. Please take a minute each day between now and November 30th to vote for our project; it’s in the South Central region, and this is the link to vote: Dream Big Challenge.  Many, many thanks.

*Note: Photo of Indian punch dagger from the collection of the Asian Art Museum in San Francisco was taken by Marshall Astor.