Exponential backoff and retry for InfluxDB client writes
A fleet-telematics gateway buffers CAN-bus and GPS fixes from a few thousand vehicles while they roam through cellular dead zones, then flushes the backlog in large batches the moment connectivity returns. Those synchronized reconnect bursts push the InfluxDB write endpoint past its per-second limits, and it answers 429 Too Many Requests or 503 Service Unavailable. The gateway’s first-generation writer treated any failure the same way — an immediate, tight retry loop — so every rejected batch was resubmitted in milliseconds, multiplying load exactly when the server was already shedding it. The result is a self-inflicted retry storm: throughput collapses, the write buffer overflows, and points that were merely delayed get dropped. The fix is a disciplined retry policy — exponential backoff with jitter, capped and bounded, that honours the server’s own Retry-After guidance and routes genuinely un-retryable batches to a dead-letter store instead of looping forever. This page builds that policy on top of the Python client orchestration patterns that govern how the gateway talks to InfluxDB.
Prerequisites
The backoff formula
Every retry policy on this page derives its wait time from one bounded expression. For attempt number $n$ (starting at zero), a base interval $b$, a growth factor of two, a random jitter term drawn uniformly from $[0, j)$, and a hard ceiling $c$, the sleep duration is:
$$ t_n = \min!\left(c,; b \cdot 2^{n} + U(0, j)\right) $$
The min with $c$ stops the interval from doubling indefinitely — without a cap, attempt 12 would sleep for over an hour. The jitter term $U(0, j)$ is not decoration: when thousands of gateways all trip on the same 503 at the same instant, an identical deterministic backoff makes them retry in lockstep, reproducing the thundering herd one delay later. Adding independent random jitter smears the retries across a window so the server drains its backlog instead of facing a second synchronized wave.
Solution walkthrough
1. Turn on the client’s built-in retry for batched writes
The influxdb-client batching writer already implements exponential backoff — but only if you configure WriteOptions deliberately. The library’s defaults retry a handful of times over a short window, which is rarely tuned to a bursty gateway. Set the parameters that map directly onto the formula above.
import os
from influxdb_client import InfluxDBClient, WriteOptions
write_options = WriteOptions(
batch_size=5_000, # points per flush
flush_interval=10_000, # ms between automatic flushes
max_retries=5, # bounded attempts (n = 0..4)
retry_interval=1_000, # base b, milliseconds
exponential_base=2, # growth factor in b * base^n
max_retry_delay=32_000, # cap c, milliseconds
max_retry_time=180_000, # total wall-clock budget across all retries
jitter_interval=2_000, # upper bound j of the uniform jitter term
)
client = InfluxDBClient(
url=os.environ["INFLUX_URL"],
token=os.environ["INFLUX_TOKEN"],
org=os.environ["INFLUX_ORG"],
)
write_api = client.write_api(write_options=write_options)
retry_interval is the base $b$, exponential_base fixes the doubling, max_retry_delay is the ceiling $c$, and jitter_interval supplies $j$. Two bounds matter most under sustained backpressure: max_retries caps the number of attempts, and max_retry_time caps the total wall-clock time spent retrying a single batch — whichever trips first ends the retry. The client automatically retries 429, 503, and 504, and it reads the Retry-After response header when present, sleeping for that server-specified interval instead of its own computed backoff. This built-in path covers the automatic batching writer and should be your default for steady ingestion.
2. Wrap explicit flushes with a jittered backoff that honours Retry-After
When the gateway flushes a buffered backlog on reconnect, it often writes synchronously so it can confirm durability before clearing the on-disk buffer. Synchronous writes bypass the batching retry, so wrap them yourself. Classify the response precisely: retry only on 429/503/504, respect any Retry-After header, and let all other 4xx errors fail fast.
import random
import time
from influxdb_client import WritePrecision
from influxdb_client.client.exceptions import InfluxDBError
RETRYABLE = {429, 503, 504}
BASE_S, FACTOR, CAP_S, JITTER_S, MAX_RETRIES = 1.0, 2, 32.0, 2.0, 5
def _sleep_seconds(attempt: int, retry_after: str | None) -> float:
# Server guidance always wins over our computed backoff.
if retry_after:
try:
return float(retry_after)
except ValueError:
pass
return min(CAP_S, BASE_S * FACTOR ** attempt + random.uniform(0, JITTER_S))
def write_batch(write_api, bucket: str, org: str, records: list[str]) -> None:
for attempt in range(MAX_RETRIES + 1):
try:
write_api.write(bucket=bucket, org=org,
record=records, write_precision=WritePrecision.NS)
return
except InfluxDBError as exc:
status = getattr(exc.response, "status", None)
if status not in RETRYABLE or attempt == MAX_RETRIES:
raise # permanent, or attempts exhausted
retry_after = exc.headers.get("Retry-After") if exc.headers else None
delay = _sleep_seconds(attempt, retry_after)
time.sleep(delay)
The Retry-After check is the piece most hand-rolled loops omit. A 429 from InfluxDB Cloud frequently carries this header telling you exactly how long the rate-limit window has left; sleeping for a locally computed two seconds when the server said thirty guarantees another rejection. Honouring the header first, and only falling back to $t_n$ when it is absent or malformed, keeps the gateway aligned with the server’s own throttle. Note that non-retryable statuses and exhausted attempts both re-raise — the caller in the next step decides what to do with a batch that cannot be written, which is where this write path connects to the wider automated task scheduling and orchestration layer.
3. Dead-letter the un-retryable and the exhausted
Retrying a 400 Bad Request (malformed line protocol) or a 401 Unauthorized (a rotated-away token) is pure waste — the batch will never succeed, and each retry only burns budget that a genuinely retryable batch needs. Both permanent failures and retry-exhausted batches must leave the hot path immediately and land in a dead-letter store, so ingestion continues and a human or a replay job can deal with them later.
import json
import pathlib
import time
DEAD_LETTER = pathlib.Path("/var/lib/gateway/dead_letter")
def flush_with_dead_letter(write_api, bucket: str, org: str,
records: list[str], batch_id: str) -> None:
try:
write_batch(write_api, bucket, org, records)
except InfluxDBError as exc:
status = getattr(exc.response, "status", None)
reason = "exhausted" if status in RETRYABLE else f"permanent_{status}"
DEAD_LETTER.mkdir(parents=True, exist_ok=True)
payload = {
"batch_id": batch_id,
"reason": reason,
"status": status,
"failed_at": time.time(),
"line_protocol": records,
}
(DEAD_LETTER / f"{batch_id}.json").write_text(json.dumps(payload))
Tagging each dead-lettered batch with its reason and original status lets a replay job apply the right treatment: re-submit an exhausted batch once the server has recovered, but quarantine a permanent_400 for schema inspection rather than looping it back into the same failure. Ordering the replay so a batch is only retried after its upstream dependencies have themselves succeeded is a scheduling concern covered in dependency mapping and DAG construction. The invariant to protect is that the on-disk buffer is cleared only after either a confirmed write or a successful dead-letter — never before — so no telemetry is silently lost between the two.
Gotchas and edge cases
Retrying on every exception, including permanent 4xx. A bare except InfluxDBError: retry loop treats a malformed-payload 400 and a rate-limit 429 identically, so a poison batch retries until it exhausts the budget and starves healthy traffic behind it. Always classify on the HTTP status and retry only the 429/503/504 family; everything else in the 4xx range is a client bug that no amount of waiting will fix, and it belongs in the dead-letter store on the first failure.
Backoff with no jitter reproduces the thundering herd. If every gateway uses the identical deterministic schedule of one, two, four, eight seconds, a fleet that trips on the same 503 retries in perfect unison, and the server faces the same synchronized load one interval later. The uniform jitter term $U(0, j)$ decorrelates the retries so the backlog drains smoothly. Full jitter — sleeping a random value in $[0, b \cdot 2^{n}]$ rather than adding a small $j$ — spreads retries even more aggressively and is worth adopting for very large fleets.
An unbounded retry budget masks a real outage. Without both max_retries and max_retry_time, a batch caught during a multi-minute InfluxDB outage retries forever, the write buffer fills behind it, and fresh telemetry is dropped at the source — the opposite of what retrying was meant to protect. Bound the retry with a total wall-clock budget so the writer gives up in a bounded time, dead-letters the batch, and keeps accepting new data. A retry policy is a smoothing mechanism for transient backpressure, not a substitute for handling a sustained outage.
Verification
Drive the classification with a fault injection or a deliberately over-large burst, then confirm the retryable batches eventually landed and the permanent ones were dead-lettered. First check that the expected points made it into the bucket:
from(bucket: "vehicle_telemetry_raw")
|> range(start: -15m)
|> filter(fn: (r) => r._measurement == "can_signals")
|> group(columns: ["batch_id"])
|> count()
|> yield(name: "points_written_per_batch")
Then assert that no batch was lost in the gap between failure and recovery — the dead-letter directory plus the written counts should reconcile to the number of batches the gateway flushed. A non-empty dead-letter directory whose entries are all reason: "permanent_400" is a healthy outcome: backpressure was absorbed by backoff, and only genuinely malformed batches were set aside. Entries tagged exhausted during a period the server was healthy, by contrast, point at a retry budget set too low for your reconnect burst size — raise max_retry_time before you raise max_retries.
FAQ
Should I use the client’s built-in retry or a hand-rolled loop?
Prefer the built-in WriteOptions retry for the automatic batching writer — it is well tested and already honours Retry-After. Hand-roll a loop only for synchronous flushes that bypass batching, or when you need custom classification such as routing specific statuses to different dead-letter queues. The two coexist: batching for steady ingestion, an explicit wrapper for the reconnect-burst flush path.
Does InfluxDB always send a Retry-After header?
No. InfluxDB Cloud commonly returns Retry-After on 429 rate-limit responses, but OSS 503 backpressure may arrive without one. Your code must handle both: use the header when present and valid, and fall back to the computed $t_n$ otherwise. Never assume the header exists, and never ignore it when it does.
How many retries is the right number?
Bound by total time, not just count. A max_retries of five with a 32-second cap and a jitter term spans a couple of minutes of transient backpressure, which absorbs most reconnect bursts. Pair it with a max_retry_time budget so a genuine outage ends the retry promptly and dead-letters the batch instead of stalling the whole buffer.
Related
- Python Client Orchestration Patterns — the reusable-client, batching, and error-handling conventions this retry policy plugs into.
- Dependency Mapping & DAG Construction — how to order a dead-letter replay after its upstream stages have succeeded.
- Automated Task Scheduling & Orchestration — the surrounding scheduling layer that decides when buffered batches flush.