How to configure retention policies in InfluxDB 2.x
You inherit a telemetry_raw bucket that has quietly grown to fill its volume, so you set retention to 30 days and move on — only to discover weeks later that expiry is reclaiming storage in coarse, unpredictable jumps, and that the oldest hours are being dropped a few hours before the nightly rollup task ever reads them. InfluxDB 2.x no longer has a separate “retention policy” object: version 1.x nested retention policies inside databases, but 2.x collapses that hierarchy into a single bucket that carries its own expiry window and its own shard-group duration. Getting retention right therefore means provisioning both durations explicitly, in code, and ordering expiry after the tasks that feed off the data. This page shows how to configure retention deterministically through the CLI, the HTTP API, and the Python client, and how to pair it with a downsampling task so nothing is dropped before it is aggregated. It puts the staged-lifecycle theory from Retention Policy Design into runnable form.
Prerequisites
How retention and shard groups relate
Before writing any commands, name the one relationship that governs everything downstream. The Time-Structured Merge (TSM) engine does not delete individual points on a timer — it groups incoming data into time-bound shard groups, and the background retention service drops a shard group whole once its entire time window falls outside the bucket’s retention period. The shard-group duration is therefore the granularity at which storage is reclaimed:
$$ D_{shard} \approx \frac{R_{retention}}{10} $$
If the shard duration is too large relative to the retention window, expiry happens in clumsy blocks and disk usage sawtooths instead of holding flat; if it is too small, you multiply file handles and compaction work and degrade query performance. A 30-day retention window with a 1-day shard duration reclaims storage a day at a time — smooth and predictable. Omitting the shard duration lets InfluxDB pick a default from the retention length (roughly 7 days for a 30-day window, 30 days for a 180-day window), which is almost always coarser than a high-velocity IoT fleet wants. This is the same shard-sizing logic that underlies bucket partitioning for IoT telemetry; here we apply it to a single retention tier.
Solution walkthrough
Step 1 — Create the bucket with explicit retention and shard duration (CLI)
The influx CLI is the most reliable interface for infrastructure-as-code and manual provisioning. Set both --retention and --shard-group-duration explicitly so the engine never falls back to a default you did not choose.
influx bucket create \
--name iot-telemetry-prod \
--retention 30d \
--shard-group-duration 1d \
--org engineering-ops \
--token $INFLUX_TOKEN
--retention 30d sets the expiry window: any shard group whose whole time range is older than 30 days becomes eligible for deletion. --shard-group-duration 1d fixes the reclaim granularity at one day, so a fleet writing millions of points per hour has its data swept in predictable daily batches rather than one large block every week. Duration strings accept h, d, and w suffixes; a value of 0 (or omitting --retention) creates an infinite bucket — see the gotchas before you rely on that.
Step 2 — Provision the same rule through the HTTP API (CI/CD)
For pipelines that stand up environments automatically, the /api/v2/buckets endpoint takes a structured retentionRules array with the same two durations expressed in seconds.
curl -X POST "https://influxdb.example.com/api/v2/buckets" \
-H "Authorization: Token $INFLUX_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"orgID": "054b8c3f2a1d9000",
"name": "sensor-aggregated-1m",
"retentionRules": [
{
"type": "expire",
"everySeconds": 2592000,
"shardGroupDurationSeconds": 86400
}
]
}'
everySeconds: 2592000 is 30 days and shardGroupDurationSeconds: 86400 is one day — the same policy as Step 1, expressed for automation. The endpoint returns 201 Created with the bucket metadata; always parse the retentionRules array back out of the response body and assert the durations match what you sent. Programmatic validation here is what catches a silent misconfiguration that would otherwise surface as storage bloat weeks later. See the InfluxDB API reference for the full schema.
Step 3 — Provision idempotently from Python
Dynamic environments need bucket creation that is safe to re-run. The influxdb-client library lets you build the retention rule as an object and handle the “already exists” case rather than failing a whole deployment on a conflict.
import os
from influxdb_client import InfluxDBClient, BucketRetentionRules
from influxdb_client.rest import ApiException
def provision_retention_bucket(org_id: str, bucket_name: str,
retention_days: int, shard_days: int) -> str:
client = InfluxDBClient(
url=os.environ["INFLUX_URL"],
token=os.environ["INFLUX_TOKEN"],
org=org_id,
)
buckets_api = client.buckets_api()
try:
rule = BucketRetentionRules(
type="expire",
every_seconds=retention_days * 86400,
shard_group_duration_seconds=shard_days * 86400,
)
bucket = buckets_api.create_bucket(
bucket_name=bucket_name, org_id=org_id, retention_rules=[rule]
)
print(f"Provisioned {bucket.name} (id={bucket.id})")
return bucket.id
except ApiException as e:
if e.status == 409: # bucket already exists — treat as success
print(f"'{bucket_name}' exists; reusing.")
return buckets_api.find_bucket_by_name(bucket_name).id
raise RuntimeError(f"Failed to provision '{bucket_name}': {e}")
finally:
client.close()
provision_retention_bucket(
org_id="054b8c3f2a1d9000",
bucket_name="edge-telemetry-raw",
retention_days=90,
shard_days=1,
)
The 409 branch makes the function idempotent — a re-run reconciles to the existing bucket instead of aborting the pipeline. For production, wrap the call in a retry/backoff decorator and log the returned metadata for an audit trail; the reusable client, batching, and backoff patterns are covered in Python client orchestration patterns.
Step 4 — Pair retention with a downsampling task
Retention only expires raw data; it does not preserve the trend inside it. Before a hot bucket’s window closes, a scheduled task must roll high-resolution samples into a longer-lived bucket — otherwise expiry is data loss, not data lifecycle. This Flux task reads the hot bucket, computes 5-minute means, and writes them into the aggregated bucket from Step 2.
option task = {name: "downsample-iot-metrics", every: 1h, offset: 10m}
from(bucket: "iot-telemetry-prod")
|> range(start: -task.every)
|> filter(fn: (r) => r._measurement == "sensor_readings")
|> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
|> to(bucket: "sensor-aggregated-1m", org: "engineering-ops")
The offset: 10m lets late-arriving edge batches close their window before the task fires, and createEmpty: false stops sparse sensors from emitting null-filled series. The non-negotiable rule is ordering: the destination bucket’s retention must be longer than the source’s, and the task cadence must comfortably beat the source’s expiry so no window is dropped before it is read. This is the entry point to the full downsampling aggregation pipeline; for anchoring windows and surviving retries, see writing robust Flux scripts for automated data rollups.
Gotchas and edge cases
A retention window shorter than your rollup cadence silently deletes data. If iot-telemetry-prod expires at 24 hours but the downsampling task in Step 4 runs only nightly, a deploy that delays the task by a few hours means the oldest samples are already gone before it reads them — and the aggregate for those hours is permanently empty, with no error anywhere. Always size the hot retention to comfortably exceed the rollup interval plus its worst-case delay, and treat retention and task cadence as one design, not two unrelated settings.
--retention 0 creates an infinite bucket that bypasses the sweeper. Omitting the retention rule or setting it to 0 is valid and useful for archival tiers, but such a bucket never expires anything — the retention service skips it entirely. Reach for it deliberately for cold/compliance data you will lifecycle by other means, and never as an accidental default on a high-velocity hot bucket, where it will grow until the volume fills.
A default shard-group duration makes expiry lumpy. Create a 30-day bucket without --shard-group-duration and InfluxDB picks roughly a 7-day shard, so storage is reclaimed a week at a time and disk usage sawtooths by ~25% of the bucket. Set the shard duration to about a tenth of the retention window (1 day for 30 days) so reclaim is smooth and queries with tight range() bounds scan fewer files.
Verification
Confirm the engine accepted the exact durations you specified — a mismatch here is a provisioning bug, not a rounding artifact:
influx bucket list --org engineering-ops --token $INFLUX_TOKEN
# Check the Retention and Shard-group-duration columns for iot-telemetry-prod:
# Retention 720h0m0s (30d) and Shard-group-duration 24h0m0s (1d).
Then prove expiry actually fires by writing a point with a timestamp just past the boundary and confirming it is gone after the sweeper runs:
from(bucket: "iot-telemetry-prod")
|> range(start: -31d, stop: -30d)
|> filter(fn: (r) => r._measurement == "sensor_readings")
|> count()
|> yield(name: "past_boundary_should_be_zero")
A zero count in the window just outside retention confirms the sweeper is reclaiming shard groups on schedule; a non-zero count means the shard-group duration is too coarse for that window to have fully aged out yet — the signal to shorten the shard duration, not the retention.
Related
- Retention Policy Design — the staged-lifecycle model that orders expiry, downsampling, and archival so data is transformed before it is dropped.
- Best Practices for Bucket Partitioning in IoT Telemetry — how the shard-to-retention sizing here extends across tenant and metric-class partitions.
- Bucket Architecture & Tiering Boundaries — the hot/warm/cold tiering these retention windows implement.