This is the last part of Fresh when it matters. The first three decided when to look. These guardrails make sure looking stays polite.
- One fetch serves everyone. Posts are cached per publication on our side (Part 2). A hundred readers of the same creator cost one request to Substack every few minutes, not a hundred. YouTube channels are cached the same way, with a 10 minute floor.
- Hard caps per run. Each minute, the server reads at most 18 Substack activity feeds (10 that are due plus up to 8 for open pages), 3 at a time. For YouTube it reads at most 28 channel feeds (20 due plus up to 8 open), 6 at a time. Streams we already know about are re-checked on Substack 4 at a time.
- Free feeds before paid APIs. New videos come from a channel's public RSS feed, which has no quota. That feed has been patchy since late 2025, so when it fails we fall back to the API's list of uploads, at 1 unit a channel. Otherwise YouTube's Data API is only used for details of the live shows, and it takes up to 50 videos per call. The API has a daily quota (10,000 units a day), and a call for 50 videos costs 1 unit, so a whole lineup is one unit.
- Back off together. If Substack or YouTube answers "429, slow down" (or YouTube says the quota is used up), every part of the system pauses that service for 10 minutes. Substack publications, profiles and live shows each have their own pause, so one being throttled doesn't stop the others. The pause is stored in the database, so no other copy of the function keeps knocking. Pages keep getting the cached answer meanwhile. A Substack request that gets a 429 is retried once after a short wait before it counts.
- Jitter. That means adding a little randomness so checks don't all land at once. Each next-check time for a live show is nudged up to 20% either way, so hundreds of creators don't come due on the same second.
- Short leashes. A request that takes more than 8 seconds is dropped, so one slow website can't hold up the whole run.
What a bad day looks like
Say Substack starts answering 429 to the live-show reads. The run that sees it stops reading Substack feeds, and writes a 10 minute pause into the database. The next run, and any other copy of the function, reads that pause before it makes a single Substack request, and skips the reads while it lasts. Meanwhile pages keep working. Posts and shows already stored are served as they were, so a reader sees slightly older data instead of an error.
When the pause ends, the normal order resumes. Open pages go first, then whoever is overdue. There is no pile-up of retries to make things worse, because we never queued any.
The rule behind all of it is that a platform telling us to slow down is information. We would rather be a few minutes behind for a while than argue with it.
Did we invent this? No, and that's the point
Every piece has a long history. Many web apps pause background tabs when nobody is looking. Feed readers check popular feeds more often than quiet ones. Search engines visit busy sites more often and slow down when a server pushes back. HTTP caching has had "serve it if it's fresh enough" for decades.
We are not claiming a new trick. What we like is the fit. Reader attention drives server effort, and learned habits fill the gaps when nobody is looking. The result: live shows appear within minutes, and the whole thing runs on small pieces. If you want the money side of that, see What it costs to run.
If you're building anything that reads someone else's data, here's our takeaway: let attention set the schedule. Check hard where people are looking, gently everywhere else, and keep a memory so you know when to look again.