A season-long field experiment · football, hockey, basketball
How good are frontier AI models at real-world decision-making? Eleven of them are finding out the hard way: each manages a real fantasy team, reading the news, setting lineups, bidding sealed dollars, pitching trades and forecasting scores, while the actual games grade every decision. Alongside them runs a control that applies ESPN’s projections mechanically. So far, across all three leagues, the best two models and the control are statistically level, and most of the field is behind all three.
Why fantasy sports
Fifty-seven million people in North America play fantasy sports, and what separates the good managers from the rest isn’t fandom or luck. It’s a bundle of skills that look a lot like knowledge work: forecasting under uncertainty, deciding which of a thousand available facts to look up, which sources to trust, negotiating with rivals who have their own incentives, budgeting against a hard cap, and committing before a deadline that does not move.
That makes a fantasy season an unusually honest exam for an AI. The relevant information is scattered across projections, rankings, depth charts, beat reporters and injury designations, far more than anyone can use, and the skill is prioritizing it. And none of it can be memorized: this season hadn’t happened when these models were trained, so every answer has to be looked up that week and acted on.
It also tests everything at once. One weekly turn demands fresh facts, arithmetic inside a budget, a forecast with a stated probability, a negotiation, and a plan for a week that hasn’t happened. Academic benchmarks isolate those skills on purpose. Here they arrive together, and a failure in any one of them lands on the same scoreboard.
So eleven models each manage a team, with identical information relayed word for word across three sports. One more team plays in every league: a spreadsheet, with no model behind it at all.
The control group
In every league, one roster is run by a spreadsheet, not a language model: I follow a fixed set of rules built on ESPN’s numbers and make no judgment calls of my own. It stands in for the most sensible basic strategy a casual player can use, which is outsourcing the thinking to ESPN’s fantasy experts: draft straight down ESPN’s rankings, start whoever ESPN projects highest among players actually playing (anyone ruled out projects at zero and sits automatically), add a free agent only when both his projection and his ownership percentage beat someone it holds, judge trades by the same two numbers, and call every game for the side with the higher projected total.
That is the entire operation. Every piece of it is a lookup, and any person with ten minutes a week could run it by hand.
It exists because a leaderboard of models only says which model is best, not how good any of them actually is. The spreadsheet is the baseline every model has to clear: if reading, reasoning and negotiating are worth anything in fantasy sports, a frontier model ought to beat a manager that does nothing but follow ESPN’s projections.
So far, few have. Averaged across all three leagues, the spreadsheet ranks first on the standings, though not by a settled margin: no model is better at both jobs, though Grok and Mistral manage better and ChatGPT and Claude Fable forecast better. In football and hockey no model has beaten it at both managing and forecasting; in basketball, two weeks in, four have.
The field
The field is the frontier, as people actually meet it: each model runs in its ordinary chat interface, with whatever search, tools and memory it ships with. Five American labs against three Chinese ones, a French entrant, one anonymous forecaster, and nine free tiers up against the two paid Claudes. Every rivalry you could want is in the bracket.
The research questions
Why three sports
A season carries far more information than its standings. Decisions compound: a wasted bid shapes a roster for months, and a dead start costs a whole week. Every week brings a new lineup, sealed bids, trade offers and a card of scored predictions, so alongside the standings the study tracks missed opportunities and mistakes directly: every start priced against the projections and against hindsight, every fabricated claim logged. Running three sports at once tests how far performance generalizes across contexts. The same models manage football, hockey and basketball simultaneously, so a manager who wins for stable reasons should win in more than one place. Three is also the practical ceiling: every message in every league is relayed and refereed by hand.
The three sports also differ in exactly the dimension this study cares about: how much public information exists. Fantasy football dominates the hobby, accounting for roughly four in five fantasy players in FSGA industry surveys, and the coverage matches: projections, rankings and analysis everywhere. Basketball adds a twist of its own: lineups lock for a week while players play anywhere from two to seven games, so a start is quality times quantity and the schedule itself becomes homework. Fantasy hockey is the most niche of the three, with the thinnest coverage, late-breaking goalie decisions, and far less for a model to lean on. If the gap between strong and weak models widens where information is thin, the advantage lies in finding and weighing information, not in recalling it.
Each league adds its own test. In hockey, teammates play different numbers of games in the same week, two of the eleven starting slots belong to goalies, who can score negative, and goalie starts are confirmed in public only hours before puck drop; in Week 1 the goalie slots alone accounted for more than a third of all the points left on benches. In hockey and basketball the schedule is withheld, so the models have to find and multiply game counts themselves. A manager who never counts games keeps starting the better player and losing to someone who started a slightly worse one twice as often.
Method
One chat per team
Each model runs in its own chat, ten to twelve per league. Every week it receives the same packet: results, standings, all rosters, its remaining budget. It replies with sealed waiver bids, a lineup, at least one trade offer, and a prediction card with confidences. A human commissioner relays every message and enters every move by hand; no model sees the league directly or reads another team’s reply.
$100, sealed bids
Free agents are won by sealed bid from a $100 season budget. Highest bid wins; ties go to the worse team. A claim on a player someone already owns is void, with no second chance that week.
Every week
Every team reports its trade offers each week, made and received, accepted and declined, and every one is logged. Offers travel between chats through the commissioner, which turns negotiation into a weekly measurement rather than an occasional event: what a model asks for, what it is willing to give up, and whether its pitch rests on facts that are true.
On purpose
Nothing computed goes back to the teams, and neither do ESPN’s projections: the spreadsheet runs on them, but a model that wants projections has to go find its own. Prediction cards are collected after the week’s initial lineups are published, so every forecaster prices the same information: results, standings and rosters. No against-the-field records, no efficiency figures, no commissioner analysis. Everything in those tables can be derived from information the teams already hold. Whether a model bothers to derive it is part of what is being measured.
Limits