Franchise analytics claims and the gap between a model and a decision

Two things that get merged

A model produces an estimate, a ranking or a projection, and that output is one input into a conversation involving coaches, scouts and owners. What eventually happens is a decision, and decisions incorporate constraints, relationships and judgements the model never saw. Crediting an outcome to analytics therefore assumes the model output survived that conversation intact, which is an assumption about a room nobody watched.

It is equally possible that the analysis was overruled and the successful selection came from a scout who disagreed with it. From outside, those two histories are indistinguishable and both are consistent with the same result.

The claiming problem after a success

When a signing works, every group involved has an incentive to describe their contribution as decisive, and none of them is lying about having contributed. The account that reaches the public is usually the one that is most communicable, which favours a clean story about data. That story is attractive because it implies a repeatable process rather than a fortunate judgement, which is what organisations wish to project.

None of this requires anyone to be dishonest; it only requires the usual asymmetry between what is said after success and after failure. The asymmetry alone is enough to make published accounts of analytical success unreliable evidence.

The unpublished denominator

Assessing whether a data department adds value requires knowing how many of its recommendations were taken and how those fared collectively. That denominator is never published, so the visible record consists entirely of the cases somebody chose to describe. A process with a mediocre overall record can produce an impressive list of successes if the list is selected after the fact.

This is the ordinary structure of survivorship, and it applies to scouting departments and coaching methods in exactly the same way. The absence of the denominator is not a flaw in analytics; it is a flaw in how analytics is discussed.

Where models genuinely help

Models are strongest where the question is repetitive, the data is plentiful and the human alternative is a fallible memory of past examples. Identifying which matchups have historically been favourable, or which roles a squad has left uncovered, is exactly that kind of question. They are weakest where the sample is small, the situation is novel and the important variables were never recorded.

Most high-profile squad decisions sit closer to the second case than the first, which is why they remain contested. A department that knows which of its questions is which is far more valuable than one that answers everything.

Judging the process rather than the season

The defensible way to evaluate an analytical operation is to look at whether its recommendations were specified in advance and checked afterwards. That requires an internal record of predictions and outcomes, which serious departments keep and which is not available to anyone outside. Public claims about analytical success should therefore be treated as marketing until the record behind them is described.

This is not a criticism of the discipline, which is capable of exactly the rigour that its public representation lacks. It is a criticism of reasoning from a highlight reel to a competence. The same reasoning would credit any department with any success, which is exactly why it establishes nothing about this one.