Everything we make starts from public data that belongs to someone else. My Social Feed and every creator site read a writer's archive from Substack. We don't get a notice when that data changes. We find out when something stops working.
Around November 5, 2026, something stopped working.
What changed
For a publication's archive, Substack used to answer with a bare list of posts. Around November 5 it started answering with an object instead: {posts, nextCursor}, the posts inside it, plus a bookmark for the next page. At the same time the old way of paging, asking for "posts 20 to 40" with an offset, was refused. The new way is to hand back the bookmark: &cursor=<nextCursor>.
It's a small change. A reasonable one, even. But three of our own readers assumed the old shape.
What broke
- The server function that keeps a shared copy of every publication. It got a perfectly good answer, looked for a list, found an object, and decided this wasn't a Substack at all. It stored every publication as "HTTP 415" (our label for "answered, but not with what we expected"). On the day we noticed, only 2 of 984 publications were readable.
- A nightly script that reads the lists of people a reader follows. It crashed every night from November 6 with a Python error about a string having no
get. - A third collector simply read zero posts.
Notice the difference. One failure was loud, and two were quiet: one looked like a normal "this site isn't a Substack" answer, the other just came back empty. The quiet ones are the dangerous ones.
How we found it
We found it while building the first creator site, whose crawler had to read one writer's whole archive page by page. That crawl hit the new cursor behavior head on. Once we looked closely, the same shape change explained the other two failures. The creator notes put it plainly: the archive "changed shape around 2026-11-05," and it "also broke" the other readers.
The fix, in one idea
Accept both shapes. The rule we now use everywhere is the same one-liner in spirit: if the answer is a list, use it; if it's an object with a posts list, use that; otherwise treat it as unreadable. Old and new both work, so if Substack ever reverts, or rolls the change out unevenly, nothing breaks. All three readers were fixed and deployed on November 8. The rows the server function had marked unreadable were reset to "due now," so they refilled on their own.
The creator-site crawler got the cursor treatment: it asks for 20 posts a page, follows the bookmark, and pauses a little between requests. One writer's archive of 2,515 posts took about 130 requests. If a page ever comes back unreadable, it says so and stops that publication, rather than quietly saving nothing.
Making it louder next time
The real lesson wasn't the format. It was that our health check only noticed when Substack was slowing us down. It had no opinion on "most answers came back unreadable." Now it fails loudly in that case too. Throttled and unreadable are different problems, and the check now tells them apart.
How we handle the other kind of trouble
Format changes are rare. Being told to slow down is routine, and we plan for it. If Substack answers "429, too many requests," the server function waits a moment and tries once more. If it's still refused, it starts a pause of 10 minutes, stored in a shared table so every copy of the function respects it. We keep separate pauses for separate doors, because we measured on October 5 that they have different limits: the publications' own archives read 677 of 683 in one burst, while the central profile lookups were throttled after about 440. One pause must not stop the other.
During a pause nothing goes blind. Pages are served the last saved copy. A network failure doesn't erase a good copy either; it just pushes the next check an hour out. The cost is that a fix can look unfinished for a few minutes while a pause runs. We'd take that over hammering a service that has asked us to stop. (More on that attitude in Being a good guest.)
If you build on someone else's data: tolerate both shapes, tell "slow down" apart from "I can't read this," and always keep a last good copy.
It's why My Social Feed and the creator sites can sit on top of platforms we don't control. We can't stop them changing. We can make sure a change costs us a bad afternoon instead of a bad week.
← All posts