Pass Muster Media My Social Feed

The Tech

When Substack changed under us

One day in November 2026 the public archive we read from Substack changed shape, and almost every publication we followed suddenly looked unreadable.

4 min read

Everything we make starts from public data that belongs to someone else. My Social Feed and every creator site read a writer's archive from Substack. We don't get a notice when that data changes. We find out when something stops working.

Around November 5, 2026, something stopped working.

What changed

For a publication's archive, Substack used to answer with a bare list of posts. Around November 5 it started answering with an object instead: {posts, nextCursor}, the posts inside it, plus a bookmark for the next page. At the same time the old way of paging, asking for "posts 20 to 40" with an offset, was refused. The new way is to hand back the bookmark: &cursor=<nextCursor>.

It's a small change. A reasonable one, even. But three of our own readers assumed the old shape.

What broke

Notice the difference. One failure was loud, and two were quiet: one looked like a normal "this site isn't a Substack" answer, the other just came back empty. The quiet ones are the dangerous ones.

How we found it

We found it while building the first creator site, whose crawler had to read one writer's whole archive page by page. That crawl hit the new cursor behavior head on. Once we looked closely, the same shape change explained the other two failures. The creator notes put it plainly: the archive "changed shape around 2026-11-05," and it "also broke" the other readers.

The fix, in one idea

Accept both shapes. The rule we now use everywhere is the same one-liner in spirit: if the answer is a list, use it; if it's an object with a posts list, use that; otherwise treat it as unreadable. Old and new both work, so if Substack ever reverts, or rolls the change out unevenly, nothing breaks. All three readers were fixed and deployed on November 8. The rows the server function had marked unreadable were reset to "due now," so they refilled on their own.

The creator-site crawler got the cursor treatment: it asks for 20 posts a page, follows the bookmark, and pauses a little between requests. One writer's archive of 2,515 posts took about 130 requests. If a page ever comes back unreadable, it says so and stops that publication, rather than quietly saving nothing.

Making it louder next time

The real lesson wasn't the format. It was that our health check only noticed when Substack was slowing us down. It had no opinion on "most answers came back unreadable." Now it fails loudly in that case too. Throttled and unreadable are different problems, and the check now tells them apart.

How we handle the other kind of trouble

Format changes are rare. Being told to slow down is routine, and we plan for it. If Substack answers "429, too many requests," the server function waits a moment and tries once more. If it's still refused, it starts a pause of 10 minutes, stored in a shared table so every copy of the function respects it. We keep separate pauses for separate doors, because we measured on October 5 that they have different limits: the publications' own archives read 677 of 683 in one burst, while the central profile lookups were throttled after about 440. One pause must not stop the other.

During a pause nothing goes blind. Pages are served the last saved copy. A network failure doesn't erase a good copy either; it just pushes the next check an hour out. The cost is that a fix can look unfinished for a few minutes while a pause runs. We'd take that over hammering a service that has asked us to stop. (More on that attitude in Being a good guest.)

If you build on someone else's data: tolerate both shapes, tell "slow down" apart from "I can't read this," and always keep a last good copy.

It's why My Social Feed and the creator sites can sit on top of platforms we don't control. We can't stop them changing. We can make sure a change costs us a bad afternoon instead of a bad week.

← All posts