# Retention Policy Design

A time-series platform almost never loses data at the moment someone sets a retention window. It loses data three weeks later, when a hot bucket set to expire at 7 days quietly drops the raw samples that a daily rollup task had not yet aggregated, and nobody notices until a monthly report shows a hole. Retention in InfluxDB is not a single storage-expiration toggle — it is a staged lifecycle in which expiry windows, downsampling cadence, and archival must be ordered so that data is always transformed before it is dropped. This page sits under InfluxDB Data Lifecycle & Architecture Fundamentals and covers how to design retention windows that survive the tasks feeding off them, automate selective purges and archival, and continuously verify that expiry is happening as intended rather than silently deleting data you still needed.

# The failure scenario this solves

Consider a fleet of 20,000 industrial sensors writing vibration and temperature at 15-second resolution into a bucket named telemetry_raw. An engineer, worried about disk growth, sets the bucket retention to 7 days. Separately, a nightly downsampling task rolls the raw data into hourly aggregates in telemetry_hourly. The two changes were made by different people in different sprints, and neither checked the other.

For the first week everything looks correct: raw data is present, aggregates are being written, disk usage flattens. Then the failures begin. The nightly task runs at 02:00 UTC and reads the previous day with a fixed range(start: -24h), but on the night a deploy delayed it by several hours, its window no longer lined up with what still existed — the oldest hours had already been expired by the 7-day boundary before the task read them. The hourly aggregate for those hours is permanently empty. Worse, the compliance team later asks for raw samples from 30 days ago to investigate a bearing failure, and there is nothing to give them: the raw tier was never meant to hold that history, but no cold archive was ever created to catch it.

The root cause is not the 7-day number. It is that retention was treated as an isolated database setting instead of the last stage of a pipeline. The remainder of this page builds retention deliberately: expiry windows sized to outlive every task that reads the tier, an archival hop for data that must persist beyond a bucket’s working life, and automated purges plus validation so that expiry is observable rather than a silent background sweep.

# Prerequisites

• InfluxDB 2.7+ (or InfluxDB 3.x Cloud/Dedicated) with the native task engine enabled.
• Flux 0.x (bundled with the versions above; no separate install) for downsampling and validation queries.
• Python 3.9+ and influxdb-client 1.36+ for programmatic bucket provisioning and boundary checks.
• An operator or all-access token, plus per-tier tokens scoped to read/write only on the buckets each task touches.
• The tier buckets already defined per Bucket Architecture & Tiering Boundaries — this page assumes a hot/warm/cold split and sets the expiry windows for each.
• Timestamps written in RFC 3339 / UTC so retention math and shard boundaries are unambiguous; see the RFC 3339 timestamp specification.

# Core concept: retention as an ordered lifecycle

In InfluxDB 2.x the legacy database-plus-retention-policy model collapses into the bucket. Each bucket carries a retention period (the every_seconds duration after which data is eligible for deletion) and a shard group duration (the time span of each internally immutable shard). Retention does not delete individual points — the background service drops whole shard groups once their newest point falls outside the retention window. That means the shard duration sets the granularity of expiry: a 7-day bucket with 7-day shards can only reclaim space in one large clump, whereas 1-day shards let it drop cleanly each day.

The design principle that prevents the failure above is an ordering constraint. For every bucket, its retention window must be at least as long as the coverage of the slowest task that reads it, plus a margin for late-arriving data and delayed runs. If a rollup reads the last 24 hours and can be delayed a few hours by a deploy, the source tier must retain well beyond 24 hours — not exactly 24. Formally, for a source bucket feeding a downsampling job:

$$R_{source} \ge W_{task} + O_{offset} + M_{margin}$$

where $R_{source}$ is the source retention window, $W_{task}$ the task’s read window, $O_{offset}$ its late-data offset, and $M_{margin}$ a safety buffer for delayed or retried runs. Retention flows the same direction as data temperature: short windows on the hot tier, longer on warm, longest (or infinite, backed by archival) on cold.

# Step-by-step implementation

# 1. Set expiry windows that outlive their consuming tasks

Provision each tier so its retention window clears the ordering constraint above. The hot tier holds raw data just long enough for real-time reads and the downsampling task to safely consume it; the warm tier holds aggregates for trend analysis; the cold tier holds compliance-grade history. Define the shard group duration explicitly rather than accepting the engine’s default so expiry granularity matches the window.

# Hot: 15s raw, retained 7d, 1d shards so expiry reclaims daily.
influx bucket create --name telemetry_raw \
  --retention 7d --shard-group-duration 1d --org iot-platform

# Warm: hourly aggregates, retained 1y, 7d shards.
influx bucket create --name telemetry_hourly \
  --retention 52w --shard-group-duration 7d --org iot-platform

# Cold: daily compliance aggregates, retained 5y, 30d shards.
influx bucket create --name telemetry_audit \
  --retention 260w --shard-group-duration 30d --org iot-platform

The 7-day hot window is deliberately far longer than the nightly rollup’s 24-hour read window — that gap is the margin that keeps a delayed or retried run from ever reading past the expiry boundary. Full CLI, HTTP, and Terraform provisioning variants are covered step by step in how to configure retention policies in InfluxDB 2.x.

# 2. Provision the same windows as code

Keep retention definitions in version control so a review catches a window shortened below what a task needs. This Terraform module encodes the three-tier topology; because the durations live beside the tasks that read them, a pull request that shrinks a source window becomes visible in review rather than surfacing as a data hole in production.

terraform {
  required_providers {
    influxdb = {
      source  = "influxdata/influxdb"
      version = "~> 2.0"
    }
  }
}

provider "influxdb" {
  url   = var.influxdb_url
  token = var.influxdb_token
}

resource "influxdb_bucket" "iot_raw" {
  name = "telemetry_raw"
  org  = var.influxdb_org
  retention_rules {
    every_seconds         = 604800   # 7 days
    shard_group_duration_seconds = 86400  # 1 day
  }
}

resource "influxdb_bucket" "iot_hourly" {
  name = "telemetry_hourly"
  org  = var.influxdb_org
  retention_rules {
    every_seconds = 31536000  # 365 days
  }
}

resource "influxdb_bucket" "iot_audit" {
  name = "telemetry_audit"
  org  = var.influxdb_org
  retention_rules {
    every_seconds = 157680000 # 5 years
  }
}

# 3. Aggregate before the boundary fires

Retention is only safe when a task guarantees high-resolution data is rolled up before its window closes. The task below downsamples 15-second vibration data into hourly means and writes them to the warm tier. Two parameters carry the safety: offset waits past the boundary for late edge packets, and tasks.lastSuccess() anchors the read to the last good run so a delayed execution resumes from where it left off instead of skipping a window that is about to expire. The retry-safe Flux discipline behind this pattern is developed in writing robust Flux scripts for automated data rollups.

import "influxdata/influxdb/tasks"

option task = {
  name: "downsample_vibration_hourly",
  every: 1h,
  offset: 10m,      // wait for late IoT packets before rolling up
  concurrency: 1,   // never overlap runs
}

from(bucket: "telemetry_raw")
  |> range(start: tasks.lastSuccess(orTime: -3h))   // resume, don't skip
  |> filter(fn: (r) => r._measurement == "vibration_sensor")
  |> filter(fn: (r) => r._field == "acceleration_mg")
  |> aggregateWindow(every: 1h, fn: mean, createEmpty: false)
  |> set(key: "_measurement", value: "vibration_sensor_hourly")
  |> to(bucket: "telemetry_hourly", org: "iot-platform")

The choice of cadence — every versus a calendar-aligned cron — determines whether the rollup window lines up with the retention boundary predictably; that interaction is the subject of cron & interval scheduling logic, and a mismatch here is a classic cause of the missing-window failure.

# 4. Automate archival before cold data expires

A bucket’s retention window still ends eventually, even on the cold tier. For data that must outlive the bucket — regulatory audit trails, forensic history — schedule an export before expiry rather than relying on the retention sweep to be the final state. This Python job, built with the Python client orchestration patterns used elsewhere on the platform, pages the oldest window out of the cold bucket to object storage on a schedule, so deletion is always preceded by a durable copy.

import os
from influxdb_client import InfluxDBClient

def export_expiring_window(bucket: str, older_than: str = "-4y"):
    client = InfluxDBClient(
        url=os.environ["INFLUX_URL"],
        token=os.environ["INFLUX_TOKEN"],
        org=os.environ["INFLUX_ORG"],
    )
    # Pull the window approaching the 5y cold boundary, oldest first.
    flux = f'''
        from(bucket: "{bucket}")
          |> range(start: {older_than}, stop: -3y)
          |> filter(fn: (r) => r._measurement == "vibration_sensor_hourly")
    '''
    csv = client.query_api().query_csv(flux, org=os.environ["INFLUX_ORG"])
    with open(f"/exports/{bucket}_archive.csv", "w") as fh:
        for row in csv:
            fh.write(",".join(str(c) for c in row) + "\n")
    client.close()

if __name__ == "__main__":
    export_expiring_window("telemetry_audit")

# Configuration reference

Duration literals accept ns, us, ms, s, m, h, d, w, mo, y. Infinite retention (0) is legitimate only on a cold tier that is backed by an archival export — never leave a hot bucket infinite.

# Common failure modes and fixes

1. Source retention shorter than the task that reads it. Symptom: aggregate tier has gaps for the oldest hours of each window; a delayed run makes it worse. Root cause: the expiry boundary fires before a delayed or retried rollup reads the data. Fix: size the source window to $W_{task} + O_{offset} + M_{margin}$, not to the read window, and anchor the read with tasks.lastSuccess().

# Give a 24h-read rollup a 7d source window, not 24h.
influx bucket update --id "$RAW_BUCKET_ID" --retention 7d

2. Shard group duration mismatched to retention. Symptom: disk barely drops when data expires, or shard counts climb into the hundreds with sluggish compaction. Root cause: shards far larger than the window reclaim space in coarse clumps; shards far smaller multiply file handles. Fix: set the shard group to roughly one seventh to one tenth of the retention window; recreate the bucket if the mismatch is severe, since existing shards keep their original duration.

3. Infinite retention with no archival. Symptom: storage grows without bound and no query ever reads the oldest data. Root cause: an every_seconds = 0 bucket used as a substitute for a real archive. Fix: set a finite cold window and schedule an export-before-expiry job so deletion is always preceded by a durable copy.

4. Retention change assumed to be instantaneous. Symptom: disk usage does not drop immediately after shortening a window, prompting a second, over-aggressive cut. Root cause: expiry is asynchronous — data is tombstoned, shards are compacted, then disk is reclaimed on the engine’s schedule. Fix: treat expiry latency as a planning input; verify reclamation over hours, not seconds, before making further cuts.

5. Retention conflicts with compliance minimums. Symptom: an audit finds required history was expired. Root cause: a window tuned only for cost ignores a regulatory minimum retention. Fix: record both the minimum and maximum retention per measurement and set the cold window to the maximum of the two; scope the tokens that can change retention using data ingestion security frameworks.

# Verification and testing

Retention must be observable, not a silent sweep. Verify three things: that each bucket’s actual oldest point matches its intended window, that rollups are landing before the source expires, and that a stalled rollup raises an alert before the boundary drops unaggregated data.

Check the oldest surviving point against the intended window for a bucket:

from(bucket: "telemetry_raw")
  |> range(start: -30d)
  |> filter(fn: (r) => r._measurement == "vibration_sensor")
  |> first()
  |> keep(columns: ["_time", "_measurement"])

If that query returns data older than the retention window, expiry is lagging; if it returns nothing near the window edge, the window is behaving as designed. Add a deadman health check so a stalled rollup pages an operator while the source data still exists to reprocess:

import "influxdata/influxdb/monitor"
import "experimental"

from(bucket: "telemetry_hourly")
  |> range(start: -3h)
  |> filter(fn: (r) => r._measurement == "vibration_sensor_hourly")
  |> monitor.deadman(t: experimental.subDuration(from: now(), d: 2h))
  |> filter(fn: (r) => r.dead == true)

From the CLI, confirm the buckets carry the retention and shard sizing you intended before trusting the pipeline:

influx bucket list --org iot-platform

A scheduled Python check can assert the boundary invariant in CI, iterating every bucket and flagging any whose oldest point drifts past its declared retention:

import os
from influxdb_client import InfluxDBClient

client = InfluxDBClient(
    url=os.environ["INFLUX_URL"],
    token=os.environ["INFLUX_TOKEN"],
    org=os.environ["INFLUX_ORG"],
)
buckets_api = client.buckets_api()
query_api = client.query_api()

for bucket in buckets_api.find_buckets().buckets:
    if not bucket.retention_rules:
        continue
    window = bucket.retention_rules[0].every_seconds
    flux = f'''
        from(bucket: "{bucket.name}")
          |> range(start: -{window}s)
          |> first()
          |> keep(columns: ["_time"])
    '''
    tables = query_api.query(flux, org=os.environ["INFLUX_ORG"])
    if not tables:
        print(f"[OK] {bucket.name}: window clean ({window}s)")
    else:
        oldest = tables[0].records[0].get_time()
        print(f"[CHECK] {bucket.name}: oldest {oldest} within {window}s window")

client.close()

# Integration points

Retention is the last stage of the lifecycle, so it depends on almost everything upstream of it. The expiry windows set here are only meaningful against the tier definitions in bucket architecture & tiering boundaries — the shard sizing there determines how cleanly the windows here reclaim disk. The ordering constraint that keeps a source window ahead of its consumer ties retention directly to the downsampling & aggregation pipeline design: a retention window is safe only if the aggregates that outlive it are already written. Which tokens are even permitted to change a retention window, and how they are scoped to least privilege, belongs to data ingestion security frameworks. And when a tier is briefly unreachable during a maintenance window, telemetry must not be dropped just because its destination is degraded — buffering and secondary routing are covered in fallback routing & high availability. Programmatic task management endpoints are documented in the InfluxDB Task API reference.

# FAQ

# How long should a hot bucket’s retention actually be?

Set it to the longest range your real-time reads and rollups need, plus a margin for late data and delayed runs — not to the exact rollup read window. If a nightly task reads 24 hours and can slip a few hours on a bad deploy, a 7-day hot window gives the safety gap that prevents the boundary from ever expiring data the task has not yet consumed.

# Does shortening a retention window delete data immediately?

No. Expiry is asynchronous: data is tombstoned, shard groups are compacted, and disk is reclaimed on the engine’s schedule, which can take hours. Plan for that latency and verify reclamation over hours rather than cutting the window again when disk does not drop in seconds.

# Can retention alone satisfy compliance archival?

Only if the cold tier’s window is at least the regulatory minimum and you accept that expiry is a hard delete. For history that must outlive the bucket, schedule an export-before-expiry job so a durable copy always precedes deletion, and record both the minimum and maximum retention per measurement.

# Why did my aggregate tier develop gaps after I set retention?

Almost always because the source window was set to the rollup’s read window instead of comfortably longer. A delayed or retried run then reads past the expiry boundary and finds nothing. Fix it by sizing the source window to the read window plus offset plus margin and anchoring reads with tasks.lastSuccess().

# Should the shard group duration change when I change retention?

Usually yes. Expiry drops whole shard groups, so the shard duration sets how granularly disk is reclaimed. Keep it around one seventh to one tenth of the retention window; a 7-day bucket wants ~1-day shards, a 1-year bucket wants ~7-day shards.

# Related

• How to configure retention policies in InfluxDB 2.x — CLI, HTTP, and Terraform provisioning of bucket expiry, step by step.
• Bucket Architecture & Tiering Boundaries — the hot/warm/cold tiers whose shard sizing governs how cleanly retention reclaims disk.
• Fallback Routing & High Availability — keep telemetry flowing when a tier is briefly unreachable.
• Data Ingestion Security Frameworks — scope the tokens allowed to change retention to least privilege.
• Downsampling & Aggregation Pipeline Design — the rollups that must complete before any source window expires.

Up one level: InfluxDB Data Lifecycle & Architecture Fundamentals