Jazz Refs Bot

A Twitter bot that posted referee stats before Utah Jazz games.

Back in 2021, when X was Twitter and bots were adorable little cron jobs, I built a bot that scraped the NBA's Last Two Minute Reports and used them to rank the referees.

I then alerted Jazz fans to what they had to look forward to that night.

The job did the following:

  1. Checked the NBA's officiating site for tonight's Jazz game and its assigned crew.
  2. Scraped any Last Two Minute Reports published since the last run and appended them to a running dataset.
  3. Scored every referee in the league on bad calls per minute reviewed, where a bad call is an incorrect call or an incorrect non-call.
  4. Wrote the post and posted it.

The stats

The Twitter bot just used a simple average (bad calls per minute while they were on the floor) for ranking the refs. This is obviously imperfect for a few reasons:

  • Sample sizes vary a lot. One ref had been reviewed for 4 minutes and another for 70. A ref with 4 clean minutes would be ranked first without a prior.
  • Blame is shared. Three refs are on the floor for every bad call, so a ref who works with error-prone crews gets blamed for their mistakes.

To improve this, I wrote some (now defunct) blog posts where I built a hierarchical Bayesian Poisson model to properly estimate error rates. Each ref has their own bad-call rate, and a game's expected number of bad calls is the sum of its crew's rates. A shared prior pulls refs with little data toward the league average.

The main takeaway from this model is that the Last Two Minute Reports don't provide nearly enough data to definitively rank NBA refs' skill, as can be seen in how wide the posterior credible intervals on the refs' rankings are in the graphic below:

Each row is one referee, sorted by average rank. Lines span the 95% credible interval for that referee's rank; most span more than half the league. Tre Maddox ranges from 1st to 39th, Curtis Blair from 52nd to 79th.League rank (1 = fewest bad calls)Hover or tap a row11020304050607079Tre MaddoxTre Maddox: rank 1–39, median 9Mousa Dagher: rank 1–54, median 14Jacyn Goble: rank 1–57, median 16Evan Scott: rank 1–62, median 18Josh Tiven: rank 1–57, median 20Kevin Cutler: rank 1–66, median 19Natalie Sago: rank 1–68, median 21JT Orr: rank 1–74, median 20Ed Malloy: rank 1–69, median 21Mitchell Ervin: rank 2–64, median 23Ben Taylor: rank 1–66, median 23Aaron Smith: rank 2–65, median 25Brett Nansel: rank 1–70, median 24Dedric Taylor: rank 2–66, median 26Haywoode Workman: rank 2–69, median 26Brandon Schwab: rank 1–73, median 25Brent Barnaky: rank 2–71, median 26Zach Zarba: rank 2–67, median 27Leon Wood: rank 2–71, median 27Sean Corbin: rank 2–67, median 28Karl Lane: rank 2–68, median 28Jonathan Sterling: rank 2–68, median 28Eric Lewis: rank 2–67, median 30Dannica Mosher: rank 2–73, median 29Phenizee Ransom: rank 2–72, median 30Ray Acosta: rank 2–71, median 31Mark LindsayMark Lindsay: rank 3–68, median 32Rodney Mott: rank 2–71, median 32James CapersJames Capers: rank 3–67, median 33Bill Spooner: rank 2–78, median 30Jenna Schroeder: rank 3–72, median 34Tony Brown: rank 2–74, median 34Scott Foster: rank 3–72, median 34Suyash Mehta: rank 2–74, median 35Nick Buchert: rank 3–73, median 35John Goble: rank 3–70, median 36Tyler Ford: rank 3–74, median 36Matt Myers: rank 3–74, median 37Scott Wall: rank 4–72, median 38John Butler: rank 3–74, median 37John Conley: rank 3–75, median 37Bill Kennedy: rank 3–76, median 37Lauren Holtkamp: rank 3–77, median 37Leroy Richardson: rank 3–74, median 40Justin Van Duyne: rank 3–75, median 41James Williams: rank 6–72, median 42Nate Green: rank 4–76, median 42CJ Washington: rank 4–75, median 43Michael Smith: rank 4–76, median 45Marc Davis: rank 6–74, median 45Jason Goldenberg: rank 7–74, median 47Matt Boland: rank 5–77, median 47Eric Dalen: rank 7–76, median 48Scott Twardoski: rank 5–77, median 49Danielle Scott: rank 6–77, median 49Pat Fraher: rank 4–79, median 50Derek Richardson: rank 5–78, median 51Andy Nagy: rank 6–78, median 51JB DeRosa: rank 10–76, median 52Tom Washington: rank 6–78, median 53Kane Fitzgerald: rank 7–77, median 53Simone Jelks: rank 7–78, median 53Mark Ayotte: rank 9–77, median 53Bennie Adams: rank 4–79, median 55Gediminas Petraitis: rank 13–76, median 54Kevin Scott: rank 11–78, median 58David Guthrie: rank 11–78, median 58Tony Brothers: rank 15–77, median 59Robert Hussey: rank 6–79, median 62Matt Kallio: rank 8–79, median 62Brian Forte: rank 21–78, median 62Derrick Collins: rank 21–78, median 62Sean Wright: rank 23–78, median 65Brandon Adair: rank 19–79, median 66Marat Kogut: rank 24–79, median 66Ken Mauer: rank 15–79, median 68Ashley Moyer-Gleich: rank 27–79, median 69Courtney Kirkland: rank 37–79, median 72Curtis BlairCurtis Blair: rank 52–79, median 76
Each row is one of the 79 referees in the 2020–21 Last Two Minute Reports, sorted by average rank. The line is the 95% credible interval on that ref's league rank and the dot is the median.
The Stan model
data {
  int<lower=0> N_games;
  int<lower=0> N_refs;
  int bad_calls[N_games];
  vector[N_games] minutes;
  matrix[N_games, N_refs] refs;   // 1 if the ref worked that game
}
parameters {
  vector<lower=0>[N_refs] r;      // each ref's bad calls per minute
  real<lower=0> theta;            // league-wide average rate
}
transformed parameters {
  vector<lower=0>[N_games] mu;
  for (i in 1:N_games)
    mu[i] = minutes[i] * dot_product(r, row(refs, i));
}
model {
  for (i in 1:N_games)
    target += poisson_lpmf(bad_calls[i] | mu[i]);
  target += exponential_lpdf(r | 1 / theta);
  target += exponential_lpdf(theta | 10);
}
Code on GitHub