This is the second part of Fresh when it matters. Part 1 covered the pages. Now the server.
One small function, once a minute
Behind the pages is one small server function (a Supabase Edge Function) that a scheduler wakes up every minute. Each run it does two jobs for live shows. First it re-checks shows it already knows about: the ones live now, or about to start. Then it goes looking for new ones by reading creators' public feeds.
It can't read every creator's feed every minute. A Substack activity feed is about 400 KB, and hundreds of creators would mean hundreds of those every minute. So each run it decides who goes first.
The order
- Creators with a page open right now. While a reader page is visible, it tells the server which people in the reader's My Feed are on screen. For the next 5 minutes, those people jump the queue and are checked about every 3 minutes, as long as we haven't already read them in the last 2 and a half minutes. Up to 8 of these per run, on top of the normal quota.
- Creators who are due. Everyone else carries a "next check" time. Whoever is overdue, oldest first, fills the rest of the run.
- Quiet creators, rarely. Someone with no stream in the last 60 days isn't ignored. They are simply due much less often (the exact numbers are in Part 3).
The effect: attention is the signal. The creators people are reading stay fresh, and the long tail costs almost nothing.
Only creators someone asked about
There is also a gate in front of all of this. The server only watches a creator if some page asked about them in the last 14 days. A creator nobody has looked at in two weeks drops out of the rotation and costs nothing, until someone looks again.
Posts work the same way. The scheduled refresh keeps the publications people asked for in the last 14 days up to date, and it paces itself by how active each one is. A publication whose newest post is under 2 days old is rechecked about every hour. Under 14 days, every 4 hours. Under 90 days, every 12. Older than that, once a day. At most 12 publications are refreshed per run, 3 at a time.
A shared cache, with a freshness dial
Everything the server reads is stored in a database and served from there. A page doesn't get its own copy. It asks for a publication, and the server answers from the stored copy if it is fresh enough.
A page can say how fresh it needs the answer to be. A creator page asks for posts "no older than 3 minutes". The server won't go below 2 minutes for that, and only honors the request when the page asks about 4 publications or fewer. YouTube channels get the same dial with a floor of 10 minutes, for 3 channels or fewer.
So the cache has three outcomes:
- The stored copy is recent enough: serve it, and make no request to Substack.
- It is older than the page asked for: fetch once, store it, serve the new one.
- The platform has asked us to slow down: serve the old copy anyway. Part 4 covers that.
A hundred people reading the same creator means one request every few minutes, not a hundred. The dial lets a page that cares about minutes get minutes, while a page that doesn't just reads what's there.
Part 2 in one line: one run a minute, open pages first, a 14-day memory of who matters, and one shared copy of everything.