Lecture (Bit.ly/DSIA02E) is the fifth Part of Second Lecture on Descriptive Statistics: An Islamic Approach. The 14m video is followed by a 1600 word writeup.
How were the seven factors that enter US News & World Report (USN&WR) College Rankings chosen? We could think of a lot of important factors related to college education which are not part of the ranking factors. To answer this question, we must realize that choice of factors represents values. According to dominant popular subjective valuations, Harvard, Yale, and other elite universities have high rank and status in public perception. This Intuitive assessment of quality comes first. When creating a ranking, we try to choose factors which which will MATCH our pre-existing intuition. That is, we already know in advance of creating rankings, which colleges should come out on top. We choose factors to support this intuition. When this method for choosing factors according to pre-existing prejudices is concealed, an illusion of objectivity is created. Concealment of Values involved in choice of weights and factors, and of role of intuition in statistical models, is one of the important aspects of deception.
To illustrate how choice of factors is based on values, Malcom Gladwell notes that the Price of Education is not included, even though, for most students, this is the most important aspect of education. The authors of the survey offer no reason for this. They say that it is “Just our subjective judgment”. But a deeper reason could be that putting in price of education could go against “intuition”. The “best” universities in popular public image are also the most expensive ones. MG asks “What happens if we include price?” He shows that some relatively unknown universities appear in the top TEN. What does this mean? It means that some universities can provide a great education at a very low price. However a ranking which put a lowly university at the top would create distrust in the ranking, which may be a reason for avoiding it.
To show how arbitrary the whole business of ranking is, MG refers to an online “Rankings Game” created by Professor Slater. He has collected Data on many different characteristics of Law Schools used in rankings. You can assign weights and watch the rankings change. Once you understand how the game works, you can create almost any ranking you like, and you can make any university come out on top.
Going outside the MG article, we can illustrate how public perception of power shapes the measures used for ranking. The dominant power gets to make the rules about what is measured. Before World War 1, “Brittania rules the waves”, and the measure of power was defined by Sea Power, Coal Mines, and others which favored Great Britain. After the war destroyed European economies, the USA emerged as the dominant economy on the Globe. In 1934, Simon Kuznets introduces the concept of the GDP, according to which US was the world leader. Later, when some tiny Oil Economies got higher GDP/Capita, the idea was adusted to include a reasonable Income Distribution. That is, if a few families in the nation are very rich, that does not make the nation the richest in the world. Even later, some European economies like Switzerland overtook the USA in GNP per capita. The measure of national wealth was re-adjusted to include Infrastructure and natural resources, where USA has a huge lead over Europe. The point is that the dominant power has the ability to dictate which factors should be used to rank nations. Currently, depending on which criteria are chosen, US or China could come out on top, reflecting the shifting balances of international power. If I choose criteria, I can make Pakistan come out on top. I would choose suicide, crime rates, Number of people who live in stable families with both parents, psychiatric patients, drugs, alcohol, percentage of population in jail, etc. Large numbers of factors which accurately reflect human lives could be used to make Pakistan come out ahead of the USA. But what does this, or any other ranking, MEAN? This question is of central importance, and not part of conventional statistics.
MG answers that Ranks are Implicit Ideological Judgments. The choice of Factors represents Value Judgments. However, positivism teaches us that values are not scientific facts. This is why conventional statistics conceals values contained in numbers. MG provides another illustration of values by discussing two factors which are opposed to each other. One of these is Selectivity: What percentage are admitted? High Selectivity automatically leads to low ranking on Graduation Rates – This requires some explanation regarding Graduation rate. The Graduation rate is not the percentage who graduate, because that would create a bias against colleges who admit poor students. Graduation Rate is the IMPROVEMENT achieved by the university over the graduation rate that its students would have in general. To explain this better, FIRST calculate EXPECTED rate for ADMITTED STUDENTS – Yale admits superstar students who would have 98% graduation rate. How much can Yale IMPROVE this rate? At most, it can achieve 100% which would give it 2 percentage points on this factor. As opposed to this, a college which admits poor students who have a 50% chance of graduation, and achieves 80% graduation rates, can get a score of 30%. It turns out that the low ranked Penn State does a great job on this factor.
Given that these two factors work against each other, how should we rank Selectivity vs Graduation Rate? There is no right answer. USN&WR makes Selectivity twice as important as the Graduation Rate. This just reflects a personal preference for Yale over Penn State. Students selecting colleges may prefer low selectivity, as it maximizes their chances of getting into the college. Governments wanting to fund education may prefer large public universities which admit everybody and do the best possible job on the worst students. The choice of weights, and the rankings, depend on the purpose for which it is done.
Rankings are not objective; rather they are ideologies masquerading as objective numbers Impact of Ideology. What is the IMPACT of these ideologies? The college rankings are no longer a harmless game that we play with numbers. These rankings have massive impact on public mindset, funding, choices made by students, professors, salaries. It is a matter of great importance that these rankings ignore the price. This means the colleges do not have an incentive to provide good education at lowest possible price, because this would not affect the ranking. The bias in favor of the wealthy is shown by the fact that the top 20 schools are always the private elite class schools. Most of the ranking factors relate strongly to WEALTH. Heavy endowments of private universities make them impossible to compete with. Even though Penn State is Most Popular university – 115,000 application – but there is no way they can get into top 20, without Billions of dollars
Malcom Gladwell concludes that Rankings reflect mindset of Ranker. He gives the example of a professor Huntington who took a survey of his colleague asking them to rank civilizations around the globe. There is no surprise that the survey ranked USA and UK at the top, given that all respondents were familiar with these civilizations, and had no knowledge of others.
Our own concluding remarks for this second lecture are as follows. There is a famous saying that “Statistics are the eyes of the State”. Factors which are measured get attention, while aspects of society which are not measured tend not to receive attention in public policy. In particular, the most important numbers are the GDP, where there is a concerted effort by Ministries around the globe to improve rankings in GDP. But the target of the efforts is the NUMBER and not the reality behind the number. The Bureau of Statistics can do many kinds of manipulation to increase GDP and increase growth rates, without having any effect on the lives of the people. Focusing on NUMBERS leads to harmful policies, while focusing on the REALITY that numbers are meant to measure would improve policies. This point is made forcefully in a book by Stiglitz with the title: “Mismeasuring our Lives: Why GNP does not add up.”. He shows how essential aspects of our lives are not measured in GNP, and this leads to very poor economic policies which cause a lot of harm in the dimensions which are not measured, while improving the measured dimensions.
This problem does not relate to GDP or College Rankings alone. Rather, we use numbers to measure performance in many dimensions, and these same problems arise in nearly all the rankings that we use. An Islamic Approach to statistics requires us to do two things in this context. One is to make the value judgments in choice of factors and weights explicit and aligned with Islamic values. The second is to focus on the Reality behind the numbers, and not the numbers by themselves.
Links to Related Materials for This Lecture: DSIA02E
This lecture: Values Embodied in Factors & Weights: bit.ly/dsia02e
First Four Parts of Lecture 2: bit.ly/dsia02a,dsia02b,dsia02c,dsia02d
- Comparing Numbers: We need to look at WHY we are comparing, and WHAT do the numbers MEAN?
- Arbitrary Rankings: Numbers used for ranking are creating by mixing subjective weights with objective characteristics.
- What Do College Rankings Measure? What Do the Numbers Mean? How are they computed?
- Goodhart’s Law: How USE of rankings to measure quality causes distortions when Universities follow policies to rise in ranking.