U of T president blunt about Maclean's survey

"If one of your hands is plunged in boiling water, while the other is frozen in a block of ice, then the average temperature of your two hands is just fine. That's exactly what happens when a range of data about a university are averaged into a single ranking."

University of Toronto president David Naylor pulled no punches in an article published in Saturday's Ottawa Citizen, defending the decision by his university and others to boycott Maclean's magazine's fall graduate survey. He said university leaders respect and support the magazine's annual spring rankings, but he dismissed the the methodology and the way the fall survey results are presented as part of a disturbing trend.

"These academic leaders respect Maclean's magazine's spring review of campuses and the readable compilation of a wide variety of performance indicators by the magazine, "Naylor said. "Many of us would happily collaborate with Maclean's if the spring format could be strengthened in some way, perhaps by grading different dimensions of a university's performance.

"But what is rapidly losing credibility is the Maclean's fall ritual of lumping a wide range of very different measures into a single set of rankings and proclaiming each year's "winners" and "losers."

"Here's the problem: Rankings and "league tables" are a good measure of success in things like sports and sales, where winning generally comes down to a single number. But no single measure can accurately reflect even a mid-sized university, where hundreds of professors and lecturers teach hundreds of courses across disciplines as varied as engineering and religion.

"Such concerns go well beyond Maclean's. They raise an important question for an era that is, rightly, concerned with measurement, accountability and transparency. But when does a metric become so oversimplified for the sake of newsworthiness that it is no longer worth using?

"My institution has found Maclean's useful for one thing only: marketing. None of us really believe that the ranking has much intellectual rigour. As academics we devote our careers to ensuring that people make important decisions on the basis of good data, analyzed with discipline. But Canadian universities have been complicit, en masse, in supporting a ranking system that has little scientific merit because it reduces everything to a meaningless, average score."