Measurement at what cost?

Jose- There was one key point that Bill Gates made in the clip you provided that I wanted to explore. The Measuring Effective Teaching (MET) study Bill Gates funded and cited has, according to some, “figured out what makes a good teacher”. We have a number of colleagues who participated in this study including my […]

Jose-

There was one key point that Bill Gates made in the clip you provided that I wanted to explore. The Measuring Effective Teaching (MET) study Bill Gates funded and cited has, according to some, “figured out what makes a good teacher”.

We have a number of colleagues who participated in this study including my friends and Center for Teaching Quality teacherpreneurs, Ryan Kinser and Megan Allen. What this study is purported to do is identify effective teaching. What the real goal seemed to be, was to identify an effective way of evaluating teachers. Or according to some, including researcher Jay Greene, find the best mix of measures to implement so that teachers would accept student assessment as a valid way to measure teacher effectiveness. Jay said,

Not surprisingly, a composite teacher evaluation measure that mixes classroom observations and student survey results with test score gains is generally no better and sometimes much worse at predicting out of sample test score gains. The Gates folks trumpet the finding that the combined measures are more “reliable” but that only means that they are less variable, not any more predictive.

The other point made by Greene is that this effective evaluation system would actually be very expensive, almost to the point of being prohibitive.

And to collect the classroom observations involves an enormous expense and hassle. To get the measure as “reliable” as they did without sacrificing too much predictive power, the Gates team had to observe each teacher at least four different times by at least two different coders, including one coder outside of the school. To observe 3.2 million public school teachers for four hours by staff compensated at $40 per hour would cost more than $500 million each year. The Gates people also had to train the observers at least 17 hours and even after that had to throw out almost a quarter of those observers as unreliable. To do all of this might cost about $1 billion each year.

However, Bill Gates didn’t get to be Bill Gates by thinking simplistically. At the same time that Gates began the MET study he also funded several grants to support and empower the “effective teachers” his forthcoming evaluation system would identify. He funded the New Millenium Initiatives (NMI) and the Empowering Effective Teachers Initiative in Hillsborough County, Florida. This project has been transformed into the Collaboratory Geo Labs where teachers in places like Hillsborough, Denver, and Seattle are working together to transform their evaluation systems for the better based on teachers’ perspectives and to positively effect their students education through collaborative approaches. As with most reforms implemented from outside of the classroom, the MET project fell a little flat while the money Gates put into the NMI and teacher leaders like Ryan and Megan is still yielding results.

What really concerns me is not the establishment and research of an effective measure but its application to student learning and its cost, both financially and professionally to education. Microsoft has used a reliable evaluation system called “stack ranking” with its employees and its effect on the company and employees seems to have been detrimental. However, it is still being used because it effectively supports productivity by identifying top programmers, average programmers, and less effective programmers. What happens in this system? Employees attack and dis-empower each other to make themselves look good. I would hate for this to happen in education, oh wait, it already does. The great potential of a measure of effective teaching is its ability to help teachers become stronger as the Hillsborough NMI learned from the MET study.

Video: Ryan Kinser on teacher evaluation.

Related categories: