When a vendor API slows down, a helpful pipeline often becomes a distributed denial of that same API. Fixed-interval retries across dozens of workers can turn a twenty-minute blip into a two-hour lockout.
Symptoms
- Error rates climb, then suddenly every job starts succeeding after a mysterious pause — because the vendor blocked you, then lifted the block.
- On-call sees “intermittent network” tickets that correlate with your own retry schedule.
- Partial files appear with overlapping primary keys after overlapping retry windows.
Habits that calm the storm
Use jittered exponential backoff. Cap concurrency per upstream. Prefer idempotent upserts so a deliberate re-pull does not duplicate facts. Record the retry budget in the runbook so humans do not “just kick it harder” during an incident.
Coordination with vendors
If you are a meaningful fraction of a supplier’s traffic during Taiwan evening hours, tell them. Some will share maintenance windows; others will at least recognize your User-Agent when you ask for a higher quota during quarter close.
Where consulting helps
Incident post-mortem facilitation is often where retry storms surface as a cultural issue: bravery points for “keeping the job alive” quietly override the quieter virtue of backing off.