A loop is not an upload pipeline
The naive version of "upload 3,500 files to an API" is a for loop with a requests.post inside it. It works for the first forty files and then something real happens — the token expires, the network blips, the process gets killed — and you're left with no idea what landed and what didn't. You re-run it and now you have duplicates.
I moved a few thousand audio masters to a third-party API this week. Here's the architecture that made it boring instead of a two-day fire drill.
1. State is a append-only log, not a counter
The single most important decision: every completed unit writes one JSON line to a state file the moment it succeeds.
{"ok": true, "path": "/…/track.wav", "id": 2359316534, "t": "2026-07-14 02:11:03"}
On startup the engine reads that log, builds a set of already-done paths, and skips them. This makes the whole job idempotent and resumable — kill it at file 900, restart, and it picks up at 901 with zero duplicates and zero bookkeeping. Failures get logged too ("ok": false with the HTTP code), so you can retry exactly the ones that failed without re-touching the 3,499 that worked.
Crucially: verify success by the ID the server returns, not by "the request didn't throw." A 200 with an empty body is not a success. Store the returned ID; later you can audit for distinctness and catch silent duplicates.
2. OAuth refresh tokens rotate — respect it or get locked out
This one bit me, so learn it free. Many OAuth providers issue single-use refresh tokens: every time you exchange your refresh token for a new access token, they hand you a new refresh token and invalidate the old one. If your code refreshes but forgets to write the new refresh token back to storage, the next refresh fails with invalid_grant and you're locked out until a human re-authorizes.
The rule: refresh and persist atomically. Read token → refresh → immediately write both the new access and new refresh token back to the database → then proceed. And never let two processes share one refresh token; they'll race and mutually invalidate each other.
3. Backoff is not optional, and neither is throttling
Be a polite API citizen or get rate-limited into oblivion:
- Throttle deliberately between calls (I use a fixed floor, ~25s for big media) rather than hammering.
- On
429/500/502/503, exponential backoff — 60s, 120s, 240s — up to a few attempts before you log the failure and move on. One bad file must never stall the other 3,000. - Refresh the token on a timer before it expires (say at 45 min of a 60-min life), not reactively after a 401.
4. The watchdog has to measure progress, not liveness
I put a watchdog in front of the whole thing, and it taught me a subtle lesson: "is the process alive?" is the wrong question. A wedged process is alive. A relay that stopped accepting connections is alive.
The watchdog has to check progress — has the state file grown in the last N minutes? If the process is up but the file is stale, it's stuck; kill and restart it. And when it declares "done," it must count distinct completed units, not log lines. My first version counted lines, and because retry sweeps re-log the same items, it declared victory while dozens of items were still unfinished. Measure the thing you actually care about.
The payoff
With those four pieces — append-only idempotent state, refresh-token persistence, backoff plus throttle, and a progress-based watchdog — a multi-thousand-item upload becomes something you start and walk away from. It survives token expiry, network drops, and a hard crash, and it can text you only when a human is genuinely needed. That's the whole game with long-running automation: make the boring path bulletproof so the interesting problems get your attention.