Cricket analytics needs axiomatic characterizations
The search for sensible metrics
A new subfield has taken birth in cricket analytics. Its name is metrification. For the major part of its history, our neophobic sport was content to be described by variants of three simple metrics—average, strike-rate, and number of balls bowled—the holy trinity of cricket statistics. Even important aspects of cricket distributions, such as the spread of scores, could be ignored. But the advent of T20 has changed everything. With increasing realization dawning about the irrelevance of traditional metrics, analysts have been conjuring up a new metric every day to describe the complex calculus of T20 cricket, and have made bloody good progress at it.
Numerous examples abound. Perhaps the high-priest of this movement should be cited as Kartikeya Date, whose adaptation of the Elo Ranking method to cricket is the first public-facing work I know of which attempts to solve the problem of ranking Test teams which have played in different conditions, against different teams, in different eras. Anantha Narayanan of ESPNcricinfo has been publishing a routine column since 2013 in which he attempts to rank the best batters and bowlers in cricket using innovative ranking methods. Himanish Ganjoo has pioneered the use of the Duckworth-Lewis Pro method to rank various batters in a tournament by their impact scores. This is in addition to the hobbyist fan, who, especially in the last year, has shaken up the field with their can-do fervour. (For a longer list of their exploits, refer here.)
One interesting aspect of the above problems is that each of these is a social choice problem. For the uninitiated, social choice theory is the subfield of economics which uses mathematical tools to isolate good means for aggregating individual opinions to generate a “collective” ranking or allocation. More specifically, the theory studies desired properties of processes for aggregating individual opinions to form a collective opinion. Perhaps the most famous example of a social choice result is the so-called Arrow’s impossibility theorem, which states that there is no aggregation rule satisfying a set of highly minimal and undemanding requirements. An example closer to home is Ariel Rubinstein’s analysis of methods of ranking teams in a tournament, which shows that the scoring system used in the IPL is the only method satisfying some highly unimposing axioms. Given a set-up, a social choice theorist asks: what are good ways of ranking agents in this set-up?
How does a social choice theorist distinguish between good and bad metrics? Enter axiomatic characterizations. Given a particular rule R for the aggregation of individual rankings into a social ranking, can we find a set of axioms that are sound and complete for R? In other words, we seek to find a set of requirements that R satisfies; moreover, every aggregation rule that satisfies these requirements should simplify to R. A result showing such an axiomatization is termed an axiomatic characterization, or a representation theorem, and it captures the exact essence of—and assumptions behind—the use of that particular rule.
What does a characterization accomplish? It helps us understand the metric we are advocating for. Without knowing what essential properties a metric satisfies and doesn’t, there is no way of reliably telling whether it is a good fit or not for the set-up, and for the abstract quantity we want to measure. To put it more generally, for a choice of a measure for a specific application to be based on its performance in some other settings, its intuitive interpretation or its popularity, cannot be considered a reasonable scientific approach. What is needed is a rigorous analysis of the properties it satisfies and whether these properties are fit for the present purpose. Let me make the same point with an example.
In 1999, Larry Page, the founder of Google, published an academic article titled “PageRank: Bringing Order to the Web”. The purpose of this article was to introduce a new ranking function which treated the Internet as a network of pages, with hyperlinks from one page to another connecting those two pages, and was able to identify the most and least “important” pages in this network using some simple and highly elegant linear algebra techniques. The crux was this: pages which are linked by important pages are themselves deemed important. PageRank took off like no other citation ranker. To date, it has been used for all and sundry purposes, including ranking the most influential Twitter posters, evaluating the importance of academic journals, and identifying cancer genes in biological data.
In the early days of tennis data, many enthusiastic analysts used PageRank to rank the greatest tennis players of all time. There is a recent cricket application too. But the catch was that for two decades mathematicians had struggled to nail a characterization of the general PageRank method, so no one knew what were the constituent characteristics of the method they used to chronicle tennis. Recently this monumental task has been achieved, and it so happens that one of the properties PageRank satisfies is agnosticism about the number of times the same player has been beaten by another player. More generally, if a page u once links to page v, and then goes on to link to v a second time in a later part of u, PageRank outputs the same scores to pages u and v in both possible worlds, and this is a fundamental, unavoidable property. This is a desirable property for ranking Internet pages. But what it says in the sports context is that Roger Federer and Novak Djokovic must be ranked exactly similarly in two possible worlds—one in which the former beats the latter in five out of five match-ups, and another in which the Swiss beats the Serbian in ten out of ten matches. PageRank is a terrible tool for sporting applications!
Hopefully I have convinced you by this point that axiomatic characterizations are important. But how does one actually go about breaking down a metric to its most rudimentary components? To illustrate this, I would like a provide a brief characterization of a simple measure of the quality of a T20 innings: the strike-rate. We want a function Q which can take an input such as (x, y), where x and y are positive quantities which stand for the number of runs and the number of balls respectively, and map it to an output z on the real number line, namely quality. So, our domain will be the first quadrant of the standard Cartesian space and the range will be the real line. Two axioms one might find desirable in such a world are what we will call “linearity” and “scale invariance”. According to linearity, scoring a constant number of runs more in the same number of balls should reflect a linear increase in quality. Formally, Q(kx, y) = kQ(x, y). According to scale invariance, scoring a constant fraction of runs more in a constant fraction of balls more should keep the measured quality of the innings unchanged. Formally, Q(x, y) = Q(kx, ky).
The main theorem is that the strike-rate metric, defined as SR(x, y) = Kx/y, where K is some normalization constant, is the only possible measure which satisfies both these axioms. I will offer a sketch of the proof. Firstly, it is easy to se that SR satisfies both these properties. Now, to see that any measure which satisfies both these properties must be the strike-rate, start from Q(x, y) = Q(xy/y , y). Using linearity, we have Q(x, y) = x/y * Q(y, y). Finally, scale invariance guarantees that Q(y, y) = yQ(1, 1). Define Q(1, 1) as K, and we finish with Q(x, y) = Kx/y, which is exactly the definition of the strike-rate! So we have proved not only that the strike-rate metric satisfies both our axioms, but also that any metric which satisfies them must collapse to the strike-rate. This is a simple proof because the strike-rate is a simple measure. But it outlines exactly why characterization theorems are useful: if you are the sort of person who believes that linearity and scale invariance are important properties, then strike-rate is the only measure of quality that should tickle your bones. But if you’re not, then it teaches us that using strike-rate as the measure of quality prima-facie grounds us to certain assumptions. If those assumptions are not passable, then we need better metrics.
In a previous post, I raised the point that as the quantity of cricket analysis increases the quality we expect of it must also increase. But I barely provided actionable means for doing this, so in some ways this article could be considered a follow-up to that post. With its recent focus on metrification, cricket analytics has firmly pivoted from being a largely empirical field to also having a thriving theoretical belt. In doing so, it finds itself in a highly exciting place, much like software engineering found itself in the 1980s, or welfare economics in the 1970s. Not unlike these fields, the job of a theory field must first be to closely examine the theoretical prescriptions it makes, only then to be followed up by an application of said rule to real-world data. It is only by doing so that the contemporary analytics community can conclusively demonstrate that the methods they prefer are improvements over the holy trinity of cricket statistics.