This year’s NCAA tournament finally brought the mother of all upsets: 16seeded UMBC (the Retrievers!) beat the Virginia Cavaliers by 21 points!^{1} Just how unlikely was UMBC’s upset?
To get a more precise without delving deeply into basketball metrics we can use Empirical Bayes as a framework for estimating the likelihood of a 16over1 upset. Our NCAA problem turns out to be especially suited for this approach because we’re asking a question about probabilities (this involves a Beta distribution) using winloss data (“Bernoulli trials”). Under this setup, the probability of a 16over1 upset is described by a probability distribution . The function gives us us the “probability of a probability”: it tells us what the probability of a 16over1 upset could be. Maybe it’s 1%, maybe 10%, etc.
We need two pieces of information to apply Empirical Bayes: an initial guess at the probability of an upset (a “prior” belief) and some hard data. The data is easy to gather; in the 33 years since the NCAA tournament format expanded to include 64 teams, 16 seeds had won zero games in attempts.
Our “initial guess” is subjective, but it won’t turn out to matter much. For fun, let’s pick two extreme guesses and see how they affect our estimates.
 Prior 1: We have no idea what the probability of an upset is; it’s just as likely for a 16 to beat a 1 as for the reverse outcome. This is clearly not a realistic position, but it (in a sense) keeps us from imposing our point of view on the problem. This belief corresponds to a distribution.
 Prior 2: We’re pretty sure 1’s will beat 16’s; maybe call it 20to1 odds against an upset. This is a much stronger initial position and corresponds to an initial distribution.^{2}
Our two guesses look like this:
The first guess is completely flat; before seeing hard data, we do not claim to know if 16 seeds win 0% of the time, 55%, or 99%, etc. The second guess, though, is much more opinionated: we’re pretty sure that the 16 seed is going to lose.
Guess  99% CI Lower  99% CI Upper 

Flat Prior  0  1.000 
Opinionated Prior  0  0.197 
With the “flat” prior, we don’t claim to know the probability of an upset: The 99% credible interval for the upset probability ranges from 0% to 100%. On the other hand, our 20to1 guess really does make an impact: The 99% CI ranges between 0% and 20%. That seems more reasonable.
Now, let’s add some data and answer the question: Going into this year, how unlikely was it that any individual 16 seed would beat a 1 seed?^{3} Since 1985 we’d observed 132 failures and zero successes, so we can update our initial guesses very easily: and . This is possible because of the Beta distribution/Bernoulli trial problem type—they are conjugate priors:
The resulting probability distributions are below—we have to really zoom in to even see where the distribution is nonzero.
Two takeaways:
 These are very similar distributions.
 The probability of a 16over1 upset is very small (we are zoomed in 20 times on the axis).
Just how small is that probability?
Guess  Most Likely  99% CI Lower  99% CI Upper 

Flat Prior  0  0  0.0340 
Opinionated Prior  0  0  0.0298 
Even for the flat prior, our 99% CI only ranges up to a 3.4% chance of upset. But, as promised, the opinionated prior hasn’t made much of a difference: incorporating our 20to1 guess only shrinks the probability to a 3% chance. In both cases, the most likely probability of an upset (the mode) is 0%!
So—we could not have ruled out an upset being impossible, given these results. That’s good, given what happened next! But, also happily, the estimated probability of an upset is very small and matches our intuition. Pretty cool!
Coda
So…what about after this year? To find out, we just follow the same procedure as before, adding three nonupsets and our one shiny new upset. The posterior distributions are now:
 Flat Prior
 Opinionated Prior
And as a result, our updated probabilities are:
Guess  Most Likely  99% CI Lower  99% CI Upper 

Flat Prior  0.0074  1e04  0.0474 
Opinionated Prior  0.0065  1e04  0.0417 
Still small, with the 99% CI ranging from about 0.01% to 4.7% and the most likely probability somewhere around 0.7%. But notice—now that we have finally observed an upset—that our posterior distribution no longer allows for a 0% chance of upset. As it shouldn’t!

Even worse: Virginia was the overall number one seed, the highestranked team in the nation. ↩︎

This is the only constraint given for a twoparameter problem; it’s only enough to guarantee and are related: . In keeping with the flat prior, we set so that . ↩︎

The probability that any of the four 16 seeds would win is slightly different: . ↩︎