Time for a cleansing breath and a serious look at teaching effectiveness data

Value-added methodologies are becoming the standard measurement for teacher effectiveness. Yet, how accurate are these measurements? Join one education expert as he questions the validity of value-added data by highlighting the numerous variables that are consistently affecting their results.

Reading the mass media accounts of the current teaching effectiveness debates leaves me wondering how much of the debate is over substantive policy and how much is plain hard-nosed politics.

There is no question that teachers should be assessed on how well their students learn. But a simple use of student achievement data (and value-added or growth models) may lead to capricious conclusions as to who is effective or not. There are three major studies, undertaken by economists over the last six months, concluding that it is VERY difficult, if not impossible, to simply claim an individual teacher is effective or not, even when using so-called sophisticated value-added tools. For example:

• Teachers have greater value-added gains when they are in schools with others who generate high value-added gains. This means there is a large peer effect and some teachers who are deemed ineffective would be effective if they teach in schools with other high value-added teachers. ^[1]
• Teachers have greater value-added gains when they teach the same subjects and grade levels consistently—especially in their first 5 years of teaching. Using 12 years of valued-added data, Ost found that a teacher who teaches the same grade for each of her first five years helps students perform 0.137 standard deviations better than students with a novice teacher.  This means that some novice teachers who are deemed ineffective would be effective if they taught the same stuff year after year.^[2]
• Teachers who are effective in one school will not necessarily be effective in another—pointing to the fact that varying school conditions may account for 25 percent of teacher effects on student learning. The researchers used a longitudinal database to estimate teachers’ value-added to student test scores before and after a move between schools and determined there was a “good” match in terms of improved value-added and the different school. Match quality explains about 7% of school effects in math, and 30% in reading. Match strength is also highly predictive of teacher retention, a finding that is also robust to inclusion of teacher and school fixed effects.^[3]

Even Rick Hess, a long-time fan of the concept of value-added methodologies, has raised cautions about assuming that VAM, in its current state, is a policy panacea. In a recent Education Week blog post, Hess writes that “advocates who are waging an admirable fight to end or dramatically scale back hyper-rigid, industrial-era state policies governing teacher tenure and compensation display a worrisome tendency to mandate that, henceforth, teachers will be evaluated in large part on (thus far) largely nonexistent, hyper-rigid, value-added metrics.” Current policy initiatives that rely heavily on these statistical metrics, he adds, represent “crudely drawn, sketchily considered, and potentially self-destructive efforts.”

So, as the disputes between labor and management as well as between thoughtful scholars and high-pitched policy wonks continue, it might be prudent for policymakers to take a cleansing breath and look carefully at the pros and cons of using standardized student achievement data to assess who is an effective teacher and who is not.

The need to explore these issues more deeply is why the Bill & Melinda Gates Foundation launched a $335 million investment in teacher effectiveness, with major grants to three large school districts and two public charter school networks to transform their systems of professional development, tenure, evaluation, and compensation. Working collaboratively with both administrators and unions—not just one or the other—the Gates-fueled approach to teacher effectiveness will rest not only on standardized test score data but other metrics derived from new tools to gauge student engagement and analyze teaching practices in depth.

It is also time, as our work supported by the Ford Foundation underscores, to pay more attention to the right working conditions that make a difference for student achievement. These include conditions such as teaching in-field (the subjects a teacher knows well); opportunities to share and critique lessons with colleagues, and resources to work closely with students and families outside of the traditional school day.

We have found that these conditions can determine who will be effective or not. I wonder when policymakers and the mass media that informs them will begin to grasp this critical point. Examining conditions is not an excuse, it is a reality of teaching. The optimum conditions for student learning must be addressed if effective teachers are to maximize their teaching effectiveness.

[1] Jackson, C. K. & Bruegmann, E. (2009). Teaching students and teaching each other: The importance of peer learning for teachers (NBER Working Paper 15202). Washington, DC: National Bureau of Economic Research.
[2] Ost, B. (2009). How Do Teachers Improve? The Relative Importance of Specific and General Human Capital. Ithaca, NY: Cornell University. Retrieved on February 1, 2010.

[3] Jackson, C. K. (2010). Match quality, worker productivity, and worker mobility: Direct evidence from teachers. (NBER Working Paper 15990). Washington, DC: National Bureau of Economic Research. Download PDF of full report