How a daily scanner tracks AI model rankings without an LLM

GHC runs a scanner that records who leads each AI model leaderboard and reports only the changes. It uses no language model. The whole pipeline is 1,068 lines of Python, four scrapers, and a diff. Most days the report says nothing changed, which is the result that matters: no one has to read ten leaderboards to learn that.

What the scanner tracks

Every run records the top 10 models in 10 categories. Eight come from LMArena, a public site where people vote between two model answers and each model earns an ELO rating (the chess ranking system) from those votes. Two come from specialist boards for speech.

A fourth collector pulls price per token and context length (how much text a model accepts at once) for 400+ models from OpenRouter, an API marketplace. It needs a free API key. When the key is missing, the collector reports skipped and the run continues.

How a run flows

The run is one command, python3 scanner.py, and it moves through four steps.

cron wrapper
   └─ scanner.py
        ├─ 1. has today's scan?  no → run 4 collectors → data/scans/{date}.json
        ├─ 2. load today + most recent prior day
        ├─ 3. diff the top 10 per category
        └─ 4. write ai-scan-latest.json + {date}.md report

Step 1 decides whether to collect. If data/scans/2026-08-22.json already exists, the scanner reuses it, so a rerun costs nothing. The --force flag overrides that check.

Step 2 picks the comparison day. The store sorts every file in data/scans/ by name and takes the latest date before today. If the job missed a week, the diff covers the week instead of failing.

Step 3 is the diff. For each category, the scanner checks four conditions:

• A different model holds rank 1.
• A model appears in the top 10 that wasn't there before.
• A model's ELO moved by 50 points or more.
• A category exists today that had no prior data.

Step 4 writes two files. data/ai-scan-latest.json is for other tooling. outputs/daily-intel/ai-landscape/{date}.md is for reading and lists every change, then the full top 10 per category with the score in ELO, or WER for speech-to-text.

Why there's no LLM in the loop

A diff is a deterministic comparison, and a model adds nothing to it except cost and the chance of a hallucinated rank. The scanner reads structured data (JSON or HTML tables) and compares strings and numbers. Each run takes seconds and costs zero API calls. A language model is only useful after the diff, when a person decides whether a change matters to a client.

The same reasoning shaped storage. The first version used SQLite. At roughly 900 rows a day, with no schedule wired up yet, a database added a schema and a dependency for no query the scanner ever makes. One JSON file per day replaced it: 41 lines of code, and ls data/scans/ is the whole history.

Where it broke

The scrapers are the fragile part, and two failures are worth recording.

TTS Arena moved to client-side rendering, and the HTML scraper began returning an empty page. The fix was to find the JSON endpoint the site's own frontend calls, GET /api/leaderboard, and read that instead, filtering out entries marked suspended. An API the frontend depends on is more stable than the markup around it.

The scheduler is the other weak point. A cron wrapper at cron-jobs/run_ai_landscape_scan.sh runs the scan and appends stderr to a timestamped log. On Windows Subsystem for Linux (WSL), cron doesn't start with the machine, so nothing is confirmed to fire on schedule. The last scan on disk is 2026-08-09. Until the job moves to a host that stays on, the scanner is a manual daily command.

What to copy from it

The pattern transfers to any recurring check a business does by hand: competitor prices, a supplier's stock page, a review site.

• Collect into a dated file. Never overwrite yesterday.
• Compare against the most recent prior file, not against a fixed "yesterday".
• Report only the changes and define the thresholds in code, with names.
• Let each collector fail on its own. One broken source skips; the other three still write.
• Keep the model out of the comparison and bring it in for judgment.

The business version of this for a client is the same 309-line scanner, pointed at whatever page they currently refresh every morning.