The Data Dagger: Why Policymakers Need Student Stories

Test scores are a means to an end. For many power players in education, test scores have become the end in themselves, with devastating results.

Part of our job as teachers is to remind policymakers that children are far more than the sum of their data.

 

 

 

A couple months after starting at a new school, my students took a math test and scored incredibly well: a class average of 87% in our school full of high-poverty English Learners, where the school average was below 50%.

People kept coming up to congratulate me, and I kept trying to dissuade them with two simple facts:

  1. Only four of my 25 students were tested. The other 21 were low enough in their level of English proficiency that they were exempt from the test.
  2. One of those four students, who scored 100%, came to my class three days before the test. Whatever he may have learned in the three math classes I had taught him had almost nothing to do with his high score.

People thought I was being humble. I wasn’t. Like most teachers, I know which data means something significant about my students’ growth, and by extension, my teaching. This data didn’t fall into that category.

I also knew that a month or year later, I could be holding the other end of that skewed data dagger. Another sample size of four or five students might have dramatically low scores, and that kid who walked in my door three days before the test might be far below proficient instead of far above.

 

Stories Matter

Policymakers, researchers, and administrators sometimes sneer at individual stories. Anecdotal evidence isn’t considered to be evidence at all.

I understand their skepticism. We did have an era when it was too easy for teachers to say of a student, “But she has made so much growth,” even if there was no data to back up that heartfelt claim. There’s some value to taking the emotion out of the data, to looking at cold hard evidence and admitting when our teaching isn’t working for some kids in the class.

But the pendulum has swung so far toward blind faith in data, any data, that even powerful people like Secretary Duncan and the Council of Chief State School Officers have begun to weigh in with warnings. They have realized the truth related by an Indian educator: when you relentlessly weigh the elephant, but don’t spent much time feeding it, it begins to starve.

Secretary Duncan gave a startling address early this school year. He has begun to listen what teachers have told him for the past six years:

“Testing should never be the main focus of our schools. There’s a whole world of skills that tests can never touch that are vital to students’ success. No test will ever measure what a student is, or can be. Yet in too many places, testing itself has become a distraction from the work it is meant to support.”

He also said this: “No school or teacher should look bad because they took on kids with greater challenges. Growth is what matters. No teacher or school should be judged on any one test, or tests alone.”

Part of the problem is that we’re expecting tests to do too much—measure the effectiveness of entire states or districts, assess individual teacher performance, help teachers improve our practice, provide meaningful information to parents, motivate students to work harder.

But another problem is that policymakers often take a “see no evil, hear no evil” approach to flaws in standardized tests. Teachers are quick to be branded as oppositional excuse-makers if we bring up problems with a given test or the way it is being used.

But case studies of individual students and teachers matter. School primarily exists for the well-being of students and their families, not for politicians, textbook companies, or think tanks. If parents and teachers bring forward a single case of an individual student being harmed by unintended consequences, chances are good that there are many more students in the same situation.

Examples abound, like this unethical but common post-NCLB strategy: To raise the number of proficient students, focus extra tutor time on “bubble kids” just below proficiency. The students desperately far below grade-level are unlikely to catch up in a single year, so investing time in them is not a good strategic bet.

 

To Live the Lives They Dream

The single greatest problem in education I see is that we have lost sight of the true purpose of school: Helping students to someday live the lives they dream of. We have reduced school’s purpose to helping students increase their test scores.

Some tests matter. Learning to pass those tests can be a means to the end of a student’s eventual goal, like getting into a good college or pursuing the career she wants. But the test scores are a means to an end. For many power players in education, ranging from Commissioners to legislators, test scores have become the end in themselves, with devastating results.

Abilities that most matter for success in college and life—higher-order thinking, ingenuity, collaboration—have been ignored or diminished, often for negligible bumps in test scores.

When you consider the toll NCLB has taken on students’ love of learning, it’s discouraging to remember how far short we have fallen of the goal set in 2002 that 100% of students would be proficient by 2014. According to NAEP, proficiency rates are still below 50% in Reading and Math for 4th and 8th graders in every racial group except Asian-Americans. (See NPR’s recent article for a damning look back at the law’s legacy.)

Parallels between teaching and the medical profession can be illuminating. Both professions are glutted with data, yet they come down to human outcomes for individual students/patients.

I was struck by an article in Sunday’s New York Times titled “Why Doctors Need Stories,” written by a psychiatry professor. The author could just as easily have been writing about the data-fixation in education.

Consider these lines:

 

“We have entered an era in which a narrow, demanding version of evidence-based medicine prevails. The inclusion of a single anecdote in a research overview can lead to a reprimand, for reliance on storytelling.”

“We need storytelling to set us in the clinical moment, remind us of the variety of human experience and enrich our judgment.”

“Data are important, of course, but numbers sometimes imply an order to what is happening that can be misleading. Stories are better at capturing a different type of ‘big picture.’”

 

In teaching, like medicine, data matters. We can’t become so fixated on the experience of our own students that we ignore the broader picture. Professional judgment demands that we see the forest, not just the trees.

But policymakers are often guilty of the opposite fixation: They can’t see the individual trees for the forest. If test scores in a state have slightly increased, but the experience of schools as reported by students, their families, and teachers has actually gotten worse, there’s a problem. Maybe it’s a problem with the test itself, or maybe it’s a problem with the way that test is being used.

Part of our job as teachers is to remind policymakers and administrators—our principal, our superintendent, our state legislature, our state department of ed—that children are far more than the sum of their data.

I can tell you that my six-year old daughter weighs 47 pounds and is 3 feet 9 inches tall, but those numbers don’t begin to encompass who she is. They won’t tell you anything about her strengths, interests, and needs, and they won’t tell you how good or bad a job I’m doing as her parent.

Most data do mean something, but they don’t mean everything. If we forget that, we’re lost, and we will have failed the students in our care.

 

Note from Justin: I just found out that a grant I wrote to scale up home libraries for about 1,800 students in our district is one of 15 finalists in the nation. Five projects will receive $100,000; for the home library project, this money would put about 30,000 books into the hands and homes of kids living in poverty. The five winners are chosen solely based on the number of votes they receive online. Please take a minute each day between now and November 30th to vote for our project; it’s in the South Central region, and this is the link to vote: Dream Big Challenge.  Many, many thanks.

*Note: Photo of Indian punch dagger from the collection of the Asian Art Museum in San Francisco was taken by Marshall Astor.

 

 

 

  • marsharatzel

    Teach a strong curriculum

    Justin,

    I am a kindred spirit in this quest.  Test scores, whether we like it or not are here.  So are lots of other things that are annoying.

    The antidote to this is to build curriculums within schools and districts that build student interest, confidence, intellgience and test scores.  At the risk of starting a math war, here’s what I’ve observed in my math classrooms over a 10 year period.  If you teach a strong, standards-based curriculum that balances conceptual understanding, problem sovling and computation…..students will do fine on any standardized test you put in front of them.

    Why?

    Because they will know how to think, reason, solve problems and have confidence.  Building in discourse and argumentation will only boost up engagement, which then translates in high achievement.  And the cycle can spiral up and up and up and up.

    I am lucky to work in a district where the math and science leadership has always built a curriculum based on national standards and best practices research.  They have the confidence to believe that if teachers teach well and stick to that curriculum, the test scores will come.  Hear me…..we don’t teach to the test.  We teach our curriculum.  And over the past 15 years, scores have increased every single year.

    It is a wonderful antidote to teacher test anxiety….because if you’re doing what you should be doing every day, bringing the wonders and curiosities of your curriculum to your students, test scores won’t be so much pressure.  You will have the confidence.  Your students will have the confidence.

    • JustinMinkel

      I mostly agree…

      Marsha, I agree with the heart of what you’ve said. I think one of the saddest things that happened during NCLB was schools continuing to drill test prep without actually teaching kids the skills they’d need to get the right answer–like testing someone over and over on how to play a scale on the trumpet when they haven’t first learned how to play the instrument. Kids ended up practicing tests wrong over and over, often guessing or relying on tricks.

      Where I’m not quite as comfortable with the state of testing is that so much of what has come to matter for success in college and life–collaboration, ingenuity, complex real-world problem-solving–just isn’t captured by most current tests. I think given our “Measurement” climate, we need to figure out how to measure those kinds of abilities, even if they’re not tied to a standardized test, for them to become/remain central to the curriculum.

      Thanks for writing.

  • ChiquitaToure

    Wow!

    Thank you so much for such a timely response to all the testing that we experience as school coummunities. Like you, I understand and concur that students’ stories are an integral part of the learning process and evaluation. Too often we rely on a test scores to give us a wholistic picture of our students when in reality there are layers and layers that must be peeled away. In graduate school I remember being taught that statistics don’t lie , statistician do. What does this mean? Those who claim to be proponents of higher standards in education can find ways to manipulate data to suppor their claims. However, what happens to education when we neglect to delve deeper into the lives of our students by providing qualitative data to present a clearer picture of progress and what it looks like.  It’s also interesting that whoever proposes the contribution of  qualitative research are referred as action researchers or “not legitimate research”. I disagree. Somehow we must find a way in integrate qualitative research methods into educational policy. Again thank you.

    • JustinMinkel

      Action Research

      Thanks for the warm words! At the International Summit on teaching in NYC a few years ago, several countries’ delegations talked about Action Research as their most effective model for PD. I think we need to raise the attention given to practitioners. In academica, squabbles between advocates of quantitative vs. qualitative methods are common, but when it comes to people actually trying to do something to improve a system, most of us know you need both.

      The stories put the data in context; the data keeps everything from being anecdotal and subjective.

      Thanks for writing.