[…] to fairly compare an underrepresented to an overrepresented group, you should never use the top individuals. […]

A Thought Experiment

[…] It’s really quite simple. Let’s say I have two groups, A and B. Group A has 10 people, group B has 2. Each of the 12 people gets randomly assigned a number between 1 and 100 (with replacement). Then I use the highest number in Group A as the score for Group A and the highest number in Group B as the score for Group B. On average, Group A will score 91.4 and Group B 67.2. The only difference between Groups A and B is the number of people. The larger group has more shots at a high score, so will on average get a higher score. The fair way to compare these unequally sized groups is by comparing their means (averages), not their top values. Of course, in this example, that would be 50 for both groups – no difference!

Indian women play as well as men on average

At this point, you might think that this is just a theoretical argument – surely, when looking at chess ratings, it cannot be that simple? So let’s have a closer look at chess ratings. I downloaded the October 6, 2020 FIDE Standard rating list, selected all players of the Indian federation, and removed all junior players (born 2000 or later), since their ratings are often not reliable. I was left with 19,064 players, of whom 17,899 (93.9%) were male and 1,165 (6.1%) were female. The best male player was a certain Viswanathan Anand at 2753, and the best female player was Humpy Koneru at 2586 – a gap of 167 points. GM Koneru, ranked 15th among all Indian players, is the only female in the top 20. On the surface, these facts superficially seem to point to a gender gap in achievement.

The huge discrepancy between the blue and orange lines reflects the participation gap. To compare the distributions more easily, we change the vertical axis from number of players to proportion of players (within each gender):

The line for female players is more jagged because there are fewer of them, but other than that, these two distributions don’t look radically different from each other. Indeed, the average ratings of men (1434) and women (1466) are comparable. And averages are the fairer metric for comparing men and women.

Is 167 points an unexpectedly large gap [between the top male and top female Indian chess player]?

But this does not answer our questions. Is, for example, a gap of 167 points between the male and female top players unexpectedly large? To answer this question, we are now going to look at all ratings as a single pool, dropping the gender identifiers altogether. We then randomly draw 17,899 ratings from this pool. These form the “overrepresented” group, and the remaining 1,165 ratings form the “underrepresented” group. These numbers are exactly the numbers of male and female players in our data, but we have instead created completely arbitrary groups with these numbers of individuals. We record the top rating in both groups. We repeat this process 100,000 times. (For the aficionados: we are following the logic of permutation tests.)

Guess what? The difference between the top ratings in the Overrepresented and Underrepresented groups is a whopping 153 points on average (with a standard deviation of 93). Again, remember that these groups are completely identical to each other except in their number of individuals. The mere fact that the underrepresented group constitutes only 6.1% of the population causes a large difference in top ratings. In this light, the real gap of 167 points could easily be due to chance instead of due to a real difference between women and men, just like the gap in our thought experiment. It is that simple.

Other widely used metrics don’t show evidence for a gender gap either. For example, based on participation alone, one would expect only 1.2 female players in the top 20 overall. So Humpy Koneru being the only female in the Indian top 20 is completely in line with statistical expectations based on the participation gap.

Conclusion

We conclude that at least among non-junior FIDE-rated Indian players, there is no evidence that the “achievement gap” is anything but a participation gap. That is not to deny the first-person perspective of top female players, who might feel that they have reached a personal ceiling in their performance. But statistically, there is nothing to suggest that top female players are underperforming given the overall ratio of female to male players. In fact, taking into account the systemic injustices and biases that they had to overcome to get where they are, they are likely overperforming.

Take-aways:

1. If you want to compare chess achievements between men and women, given their vastly unequal numbers, it is a very bad idea to focus on the top male and female players.

2. If you insist to focus on the top players, you will have to account for the participation gap using an analysis similar to the one presented here. Just a casual remark won’t do it!

3. Even if, hypothetically, you were to find a gender difference based on average ratings (rather than top ratings), you cannot jump to the conclusion that that difference is due to innate or biological factors. The first place to look would be systemic disadvantages and stereotype threat experienced by female players.

The statistical arguments presented in this article are elementary enough for an introductory statistics class in university. In case you want to repeat the analysis for different countries, you can check my Matlab code for the details. If you prefer to read a published paper, look no further than this excellent paper by Merim Bilalić, Kieran Smallbone, Peter McLeod, and Fernand Gobet (2009). (The free PDF can be found through Google Scholar.) Its title is the first question everyone should be asking: Why are (the best) women so good at chess? And everyone’s second question should be: How can we reduce the insane participation gap?