The Caveat Emptor Index

Abstract

IN January 2006, I used this space to depart from my all-too-dystopic view of the world of education reform to welcome, albeit tepidly, the Canadian Council on Learning (CCL). I spent several paragraphs describing the political risks that historically accompany national education initiatives in Canada, but I wrote that the initial signals had been positive. While the CCL had fallen well short of pledging to speak truth to power, its leadership seemed determined to create and use research to inform its own practice and the advice it would offer to policy makers. It had accumulated significant amounts of good will with stakeholders and was making the right noises about collaboration, vision, and new paradigms. It was true that some of the CCL's key players seemed as recycled as the corporate-speak they used to promote their product, but, hey, my New Year's resolution for 2006 was to be less judgmental. Now that we're well into 2007, I can report that neither that resolution nor this year's sequel--to be less critical--survived its respective holiday season. Others who made similar resolutions were apparently more successful in keeping them, since I have been unable to find any stakeholder prepared to criticize the CCL's flagship initiative, the Composite Learning Index (CLI). Everyone with any interest in lifelong learning, however tangential, praised the release of the CLI: 2006. The CCL even excerpted on its website choice bits of fan mail it had received. (1) Shortly after it was established with $85 million in funding from the federal government, the CCL promised to create an annual index. Skeptics who had been battered by superficial, league-table approaches to comparing educational outcomes were heartened by CCL's avowed fealty to the vision expressed in Learning: The Treasure Within. (2) This 1996 report to UNESCO, often referred to as the Delors Report, remains the touchstone of many progressive educators. CCL's president and CEO, Paul Cappon, promised to choose indicators associated with the four pillars of the Delors Report: Learning to Know, Learning to Do, Learning to Live Together, and Learning to Be. The conceptual breadth of the project pleased those who believe that education is not well served by simplistic comparisons of test scores. The provinces were pleased that the index would eschew any awkward mention of inputs, such as funding for education, in favor of outputs. Colleges and universities believed that including items pertaining to postsecondary participation might advance their arguments for deregulated tuition, increased funding, or both. Employers' groups and corporate lobbies were assured that competitiveness and productivity would be front and center. The quant-o-philes would be able to play with interactive online charts of data and surf an impressive website. So much mileage from a mere 15 indicators and a single grand total! To be fair, relying on indicators is a risky business--made much riskier by counting on bundles of indicators to describe complex phenomena. An indicator is always a proxy for the real thing being measured: the average price of a restaurant's entree selections may, or may not, indicate the quality of the food being served. Everything flows from the nature of the relationship between the proxy and the real thing. Well, almost everything. Even if the proxy is a valid indicator of what is being measured, how accurately the proxy is assessed matters as well. Knowing what the restaurant charged for steak three years ago--under different management and presided over by a different chef--can mislead the most discerning client. I should acknowledge that educational researchers are often forced to settle for second-or even 10th-best indicators when the ones they want don't exist. The information void at the center of Canadian education, a problem I also addressed in my previous article on the CCL, has resulted in a dearth of national data in certain domains and an overreliance on data that were collected for very different purposes or too long ago to be helpful. …