In one of my classes, I taught students how to develop measurement systems that actually reflect what is important. At the beginning of class, I asked them to write down on a sheet of paper what they thought were the components in the most prominent ranking of universities. Then I asked them to give me their rankings. I put these on the board. Then I circled the components from their list that were actually used in the rankings. On a good semester, one or two of their components were circled. The students were astonished.
The reality is that rankings often reflect the bias of the people making up the rankings. In the rankings of universities, the components have a very definite elitist bias.
The point I made after this exercise was that measurements need to reflect what really matters. If you are going to rank universities, you need to do extensive work in surveying what parents and students want from a university.
Next, you need to be sure that your measurements are true indicators of what they are actually meant to represent. A popular ranking of the social side of universities often confuses party atmosphere with campus spirit. Universities where students are spirited and engaged are often thought to be party schools.
Rankings should also be sealed in the right direction. A ranking provided by a national accounting firm measures states in their attractiveness to business. One component in the ranking is the state’s environmental management. The more money the state spends on environmental management programs, the lower their ranking. For most responsible businesses, this ranking is clearly in the wrong direction.
The purpose of the ranking and the funding source behind it also needs to be considered. Rankings are often used for political purposes to shame legislative and executive branches of state governments. Both conservative and liberal groups do this.
If there is one generalization that could be made about rankings, it is this: “Don’t use them for anything other than amusement.” You need to know what’s important to you and do your own research in your evaluations of choices you need to make.
* * *
“If you get good ratings, they’ll cover you even if you have nothing to say.” – Donald Trump