v19.1: The Home Page Was Reading 13 Megabytes
A cost review measured what the site actually pulls from the database on a single page load: 750 documents and about 13 MB, before anything appeared on screen. One component accounted for 98% of it. Moving three aggregations to the server cut a page load to 5 documents and roughly 60 KB, with no change to what the page shows.
The site had never been measured against its own database. Doing so produced an uncomfortable number: loading the home page read 750 documents and transferred roughly 13 megabytes before the first chart could paint. On a phone that is a visible wait. Projected against traffic, it was also the overwhelming majority of the running cost — and it scaled linearly, so every new visitor made it worse.
| Source | Reads per visit | Bytes | Share of bytes |
|---|---|---|---|
| 30-day trend chart | 721 | 13.41 MB | 98% |
| AI panel (mounted but hidden) | 25 | 0.03 MB | under 1% |
| Geo heatmap | 1 | 0.24 MB | 2% |
| Live meter subscription | 1 | 0.02 MB | under 1% |
| Yesterday’s meter, space weather | 2 | 0.02 MB | under 1% |
The trend chart was reading 721 snapshots to draw 120 points
The chart requested every hourly snapshot in its range — 721 documents for the default 30-day view — then averaged them down into the 120 points it actually plots. Eighty-three per cent of what it paid for was discarded before anything was drawn. Worse, each hourly snapshot carries the full list of top signals for that hour, roughly 18 KB of article headlines and AI summaries that the chart never reads.
That bucketing now happens once an hour on the server instead of once per visitor. The scheduled job that writes hourly snapshots also maintains three small documents — one per range — containing the finished 120 points. A page load reads one of them.
The detail panel that opens when you click a point still shows that snapshot’s top signals. Each precomputed point records which snapshot it represents, and that single document is fetched on click. The chart is drawn from exactly the same arithmetic as before — the bucketing logic is shared between the server job and the browser, and was verified to produce byte-identical output across every range and boundary case, because a chart that quietly changes shape would be a worse outcome than the cost it fixed.
The heatmap was shipping article lists nobody opened
The country heatmap had already been moved server-side, replacing a 500-article scan with a single pre-aggregated document. But that document stored five full articles for every country — 240 KB of headlines, links and summaries — and the map only needs a country code, a score and a count to colour itself. The article lists were being downloaded by every visitor to support a panel that opens for at most one country per session.
The rollup is now split. The map reads a small document containing just the render data. The article lists live in a second document fetched on the first country click and cached for the rest of the session. This also moves the data well clear of the database’s per-document size ceiling, which the combined version had been approaching as country coverage grew.
The AI panel was loading for everyone
The AI signals panel was kept permanently mounted and hidden, so it could compute a fallback score for the tab strip — 25 article reads for every visitor, including those who never opened that tab. The backend already publishes an AI score computed from the full article pool, which is both more accurate than the fallback and already present in the data the page loads. The panel now renders only when its tab is selected.
Result
before: 750 documents, 13.7 MB per page load after: 5 documents, ~60 KB per page load
Nothing about the interface changed. The same charts, the same detail panels, the same data. The difference is that the aggregation now happens once an hour on a server that already has the data in memory, instead of once per visitor across the network. The page also loads considerably faster on mobile connections, which was never the goal but is the more valuable outcome.
What was left alone
Two further optimisations were identified and deliberately not made, because both would change behaviour rather than plumbing. The article-creation trigger re-scores the meter roughly as often as the five-minute schedule already does, which is duplicated work — but removing it changes how quickly the meter reacts. And raising the 500-article scoring cap would let slow institutional feeds back into the pools they are currently crowded out of, which is a scoring change and belongs in a versioned release rather than a performance one.