By far, the most interesting discussion of the week for me started with a simple tweet:

Did I ever share this data literacy survey that I created with you? It’s from my book on building PLCs. about 17 hours ago from TweetDeck

My goal was pretty direct:  I wanted to point people to a tool to help learning teams measure their comfort levels with data as a tool to drive instruction.

I created the survey with my learning team in mind because we’ve always struggled to figure out just what to do with data, and I’ll bet that other teams are in the same boat.  The way I see it, if my experiences and resources can help other teams, life is grand.

Jon Becker—an assistant professor for Ed Leadership at VCU, a provocative, well respected scholar, and one of the people following my posts in Twitter—replied with a question that got me thinking:

@plugusin sorry if I sound “professorly,” but have you validated the instrument at all? about 15 hours ago from TweetDeck in reply to plugusin

The conversation that followed was pretty fascinating.  First, I tried to explain to Jon that teachers aren’t hung up on validated tools.  Instead, we’re looking for quick and easy resources that we can use immediately to start conversations with each other.

Here’s a piece of that conversation.  For those who aren’t Tweeting yet, My comments to Jon are colored red.  His comments to me are colored blue:

@jonbecker : Professorially validated? No. Validated based on years of experience actually doing the work? Yes. about 13 hours ago from TwitterBerry

@plugusin no, psychometrically validated. How do you know you’re measuring what you’re claiming to measure? about 13 hours ago from TweetDeck in reply to plugusin

@jonbecker The tool isn’t for research. Instead, its for gathering information at the team/school level for setting direction. about 13 hours ago from TwitterBerry

@plugusin information comes from data, so the information you end up with is only as good as the data collection process. about 13 hours ago from TweetDeck in reply to plugusin

@jonbecker Waiting for validated tools developed by “experts” has never helped our team move forward. about 13 hours ago from TwitterBerry

@plugusin there are existing instruments that have been validated for the same purposes, including ones developed by @mcleod about 13 hours ago from TweetDeck in reply to plugusin

@jonbecker : I think you’re over estimating exactly what teams do with these tools. They aren’t about research. They’re about conversations. about 13 hours ago from TwitterBerry

Over time, I started to wonder out loud whether the concept of validation is changing in the Web 2.0 world.  To traditionally trained scholars, school leaders and experts, validation means putting your work through a semi-scientific review process that—as you can imagine—is time consuming and intimidating, covering questions like:

  • Does this test accurately measure the surveyed domain?
  • Does this test measure the range of behaviors possible within the surveyed domain?
  • How close does this test correlate with other measures that it should theoretically correlate with?
  • Can the creator of this survey demonstrate that it is valid by relating it to another measurement known to be valid?
  • Has the survey creator controlled for any extraneous variables that may interfere with validity?
  • To what degree can causal inference be generalized from the sample studied to the target population of the survey?

As a result, teachers and other “lay people” have traditionally been locked out of the community of creators.

Because we didn’t have the sophisticated understanding of the steps involved in making our work “valid” as defined by authorities, we were left to sit on the sidelines taking direction from those “in the know.”  And because our society has always been driven by hierarchies, we had no chance of becoming recognized experts with worthwhile ideas to share.

Times are changing, though:

@jonbecker :  I also wonder if our definitions of validation are changing. Someone’s got to like the stuff I create. Otherwise they wouldn’t follow me. about 13 hours ago from TwitterBerry

@jonbecker :  That’s the beauty of digital tools. My ideas/materials stand on their own merits. I don’t need anyone else’s permission to publish them. about 13 hours ago from TwitterBerry

@jonbecker :  If you like what I have to offer, great! If you don’t, great. But at least I can be heard too—and at least you have the choice to listen. about 12 hours ago from TwitterBerry

@jonbecker :  For classroom teachers, that’s pretty empowering, actually. I don’t need a doctorate or a principal’s license to be considered credible. about 12 hours ago from TwitterBerry

@jonbecker :  I live in a check your title at the door, Jon. Credibility comes from producing ideas that make my work eaiser… about 11 hours ago from TwitterBerry

Interesting stuff, huh?  This is a conversation that still has me thinking.  Here’s what I’m wondering:

  • Are there any real risks associated with using surveys and tools that haven’t been psychometrically validated?  I’m pretty confident that teams and teachers are going to use any tool that they find to be valuable, whether it has been tested or not.  Does that carry consequences that I haven’t considered?
  • Has “empowering the masses” actually harmed education?  I can’t even begin to explain how much I learn every day from people who have no formal credentials.  Social media—blogs, Twitter, Skype, Facebook—give me instant access to minds that I wouldn’t have ever found before.  But in a sense, social media cheapens the formal knowledge earned through systematic study of classical practices and texts.  Will that carry any consequences that I haven’t considered?
  • Do teachers stand a chance of ever being seen as the intellectual equals of those who have “advanced” within the existing system?  Even as digital tools make it possible for teachers to raise their voices, our P-20 school systems still function as hierarchies—and in those hierarchies, teachers remain at the bottom with little organizational power.  Does that mean our contributions will always be questioned by those who have “moved up?”  If so, what consequences does that carry for our profession?

Looking forward to your responses….

Share this post: