# Storage Tier Migration Automation

Defining hot, warm, and cold buckets does nothing on its own — data does not walk between them. Something has to read the aging high-resolution points out of the hot tier, downsample them, write them into the warm tier, and later fold the warm tier into the cold tier, all on a schedule that finishes before each tier’s expiry window drops the source. This is storage tier migration: the recurring copy-and-aggregate tasks that physically relocate data across tiers so every tier holds the resolution appropriate to its age and total disk stays bounded. This page sits under InfluxDB Data Lifecycle & Architecture Fundamentals and covers how to build those migration tasks — the Flux jobs that move hot to warm and warm to cold, a Python orchestrator for large cross-tier copies, and the watermark and verification discipline that stops a migration from silently losing or double-writing data. It is a distinct concern from the two design layers on either side of it: where bucket architecture & tiering boundaries defines the tiers and retention policy design sets the expiry windows, this page is about the scheduled machinery that moves data between those tiers before expiry can drop it.

# The failure scenario this solves

Consider a utility running an Advanced Metering Infrastructure (AMI) deployment of 500,000 smart electricity meters. Each meter reports instantaneous load in kilowatts every 30 seconds into a hot bucket named meter_load_hot, retained 3 days. A separate warm bucket, meter_load_warm, holds 5-minute means retained 90 days for billing analytics, and a cold bucket, meter_load_cold, holds hourly means retained 3 years for grid-planning and regulatory reporting. The tiers were provisioned correctly. The retention windows were sized correctly. But nobody built the tasks that actually move data between them.

For the first three days the platform looks healthy: the hot bucket fills, dashboards query it directly, disk climbs as expected. On day four the hot tier’s 3-day expiry begins dropping the oldest shard groups — and because no migration task ever ran, those 30-second readings are simply gone. The warm and cold tiers are empty. The billing team pulls a monthly load profile and finds three days of data, then nothing. Worse, when a migration task is finally bolted on in a hurry, it is written to copy the raw 30-second points straight into the warm bucket without downsampling, so the warm tier inherits the full 500,000-series firehose at 30-second resolution and blows through its disk budget in a week — the tiering was defeated because migration and downsampling were treated as separate steps instead of one operation.

The root cause is that tier definitions and retention windows describe a desired state but do not enforce it. Migration is the enforcement: a scheduled task must aggregate and relocate each window of data out of the hot tier while the source still exists, and it must reduce resolution in the same pass so each tier’s footprint matches its age. The rest of this page builds that machinery — migration tasks paced to beat expiry, downsampling folded into the move, and a watermark so no window is ever copied twice.

# Prerequisites

• InfluxDB 2.7+ (or InfluxDB 3.x Cloud/Dedicated) with the native task engine enabled.
• Flux 0.x (bundled with the above) for the migration and downsampling tasks.
• Python 3.9+ and influxdb-client 1.36+ for the cross-tier orchestrator and backfills.
• Three tier buckets already provisioned — meter_load_hot, meter_load_warm, meter_load_cold — per bucket architecture & tiering boundaries, with expiry windows set per retention policy design.
• A task token scoped to read the source tier and write the target tier — never a blanket all-access token on a migration job.
• Timestamps written in RFC 3339 / UTC so migration windows align with shard boundaries; see the RFC 3339 timestamp specification.

# Core concept: migration is a scheduled race against expiry

Each tier holds data at a resolution appropriate to its age, and a migration task carries data from one tier to the next by aggregating it down and writing it forward. The defining constraint is temporal: for any source tier, the migration that empties it must complete before that tier’s retention boundary drops the data. Retention design guarantees the source window is long enough to survive a delayed run; migration design guarantees the task actually runs often enough and finishes fast enough to stay ahead of it. If the migration cadence plus its runtime plus its late-data offset ever exceeds the source retention minus a safety margin, the boundary wins and data is lost before it moves.

For a source tier with retention $R_{src}$, a migration cadence $C_{mig}$, a per-run duration $D_{run}$, and a late-data offset $O_{late}$, the safe-migration invariant is:

$$C_{mig} + D_{run} + O_{late} \le R_{src} - M_{margin}$$

The second half of migration is resolution reduction. A tier’s disk footprint is roughly its series cardinality times its retention divided by its sample interval. Moving data forward without lengthening the interval defeats the whole tiering scheme, so every migration hop must downsample in the same pass — 30-second raw to 5-minute means into warm, 5-minute means to 1-hour means into cold.

# Step-by-step implementation

# 1. Migrate hot to warm with a watermark-anchored Flux task

The core migration is a task that reads a bounded window of the hot tier, downsamples it, and writes the result to the warm tier. The window is anchored to the last successful run with tasks.lastSuccess() — this is the watermark. Anchoring to the watermark rather than a fixed range(start: -15m) is what makes the task idempotent: a delayed or retried run resumes exactly where the previous success ended, so no window is skipped and no overlapping window is re-copied. The to() call downsamples in the same pass, so raw resolution never lands in the warm tier.

import "influxdata/influxdb/tasks"

option task = {
  name: "migrate_meter_hot_to_warm",
  every: 15m,
  offset: 2m,       // let late meter packets settle before the window closes
  concurrency: 1,   // one run at a time so windows never overlap
}

// Watermark: resume from the last successful run, capped at 6h of catch-up.
start = tasks.lastSuccess(orTime: -6h)

from(bucket: "meter_load_hot")
  |> range(start: start)
  |> filter(fn: (r) => r._measurement == "meter_load")
  |> filter(fn: (r) => r._field == "active_power_kw")
  |> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
  |> set(key: "_measurement", value: "meter_load_5m")
  |> to(bucket: "meter_load_warm", org: "grid-ops")

The orTime: -6h cap bounds how far a stalled task will reach back on its first recovery run, so a task that was disabled for a day does not attempt to re-aggregate the entire hot tier in a single execution. The offset: 2m waits for late-arriving meter packets before the window is considered complete; because the hot tier retains 3 days, this offset is trivially inside the safe-migration invariant. The per-window mechanics of this task — cardinality control, boundary alignment, empty-window handling — are worked through in automating hot-to-warm bucket migration with Flux tasks.

# 2. Migrate warm to cold on a coarser cadence

The warm-to-cold hop follows the same shape but runs far less often and downsamples further. Warm data changes slowly and its retention is 90 days, so a daily migration comfortably satisfies the invariant. Aligning the task to a calendar boundary with cron rather than every keeps each day’s aggregate window clean and reproducible — the interaction between calendar-aligned and interval scheduling is covered in cron & interval scheduling logic.

import "influxdata/influxdb/tasks"

option task = {
  name: "migrate_meter_warm_to_cold",
  cron: "20 1 * * *",   // 01:20 UTC daily, after the day's 5m data has settled
  offset: 0s,
  concurrency: 1,
}

start = tasks.lastSuccess(orTime: -3d)

from(bucket: "meter_load_warm")
  |> range(start: start)
  |> filter(fn: (r) => r._measurement == "meter_load_5m")
  |> filter(fn: (r) => r._field == "active_power_kw")
  |> aggregateWindow(every: 1h, fn: mean, createEmpty: false)
  |> set(key: "_measurement", value: "meter_load_1h")
  |> to(bucket: "meter_load_cold", org: "grid-ops")

Reading from meter_load_5m rather than the raw hot tier is deliberate: cold aggregates are built from warm aggregates, so each hop reduces resolution by one step and the cold tier never has to touch the 30-second firehose. Building rollups from already-aggregated tiers rather than re-reading raw is a broader principle of downsampling & aggregation pipeline design.

# 3. Deploy the tasks and confirm cadence beats expiry

Apply the tasks with the influx CLI and record which token each runs under. Keep the migration cadence, offset, and expected runtime documented against the source retention so the safe-migration invariant is auditable, not assumed.

# Register both migration tasks from their Flux files.
influx task create --org grid-ops --file migrate_meter_hot_to_warm.flux
influx task create --org grid-ops --file migrate_meter_warm_to_cold.flux

# Confirm they are active and note their schedules.
influx task list --org grid-ops

# Sanity-check the hot tier's retention against the 15m migration cadence:
# 15m cadence + a few minutes runtime + 2m offset must be well under 3d.
influx bucket list --org grid-ops --name meter_load_hot

# 4. Orchestrate large cross-tier copies in Python

Native tasks are ideal for the steady-state 15-minute and daily hops, but a bulk backfill — re-tiering a year of history after a schema change, or seeding a new cold bucket — needs explicit chunking, a durable watermark, and retry control that a single Flux task cannot give. This Python orchestrator walks the source window in fixed chunks, aggregates each chunk, writes it to the target tier, and only advances a persisted watermark after a chunk succeeds. Because the watermark is written after the copy, an interrupted run resumes at the last completed chunk instead of restarting or double-writing. The batching and retry discipline here follows the platform’s Python client orchestration patterns.

import os
from datetime import datetime, timedelta, timezone
from influxdb_client import InfluxDBClient
from influxdb_client.client.write_api import SYNCHRONOUS

WATERMARK = "/var/lib/meter-migration/warm_to_cold.watermark"

def read_watermark(default: datetime) -> datetime:
    try:
        with open(WATERMARK) as fh:
            return datetime.fromisoformat(fh.read().strip())
    except FileNotFoundError:
        return default

def write_watermark(ts: datetime) -> None:
    os.makedirs(os.path.dirname(WATERMARK), exist_ok=True)
    with open(WATERMARK, "w") as fh:
        fh.write(ts.isoformat())

def migrate_chunk(client, start: datetime, stop: datetime) -> None:
    # Downsample this chunk of warm data and write it into cold in one pass.
    flux = f'''
        from(bucket: "meter_load_warm")
          |> range(start: {start.isoformat()}, stop: {stop.isoformat()})
          |> filter(fn: (r) => r._measurement == "meter_load_5m")
          |> filter(fn: (r) => r._field == "active_power_kw")
          |> aggregateWindow(every: 1h, fn: mean, createEmpty: false)
          |> set(key: "_measurement", value: "meter_load_1h")
          |> to(bucket: "meter_load_cold", org: "grid-ops")
    '''
    client.query_api().query(flux, org=os.environ["INFLUX_ORG"])

def run_backfill(start: datetime, end: datetime, chunk=timedelta(days=1)):
    client = InfluxDBClient(
        url=os.environ["INFLUX_URL"],
        token=os.environ["INFLUX_TOKEN"],
        org=os.environ["INFLUX_ORG"],
    )
    cursor = read_watermark(default=start)
    while cursor < end:
        stop = min(cursor + chunk, end)
        migrate_chunk(client, cursor, stop)   # idempotent: overwrites same 1h points
        write_watermark(stop)                  # advance only after success
        print(f"[OK] migrated {cursor.isoformat()} -> {stop.isoformat()}")
        cursor = stop
    client.close()

if __name__ == "__main__":
    origin = datetime(2025, 7, 1, tzinfo=timezone.utc)
    run_backfill(origin, datetime.now(timezone.utc))

The chunk boundaries land on the same 1-hour grid the aggregation produces, so re-running an already-migrated chunk rewrites identical timestamps rather than creating duplicate points — the write is idempotent by construction. Data that must ultimately leave InfluxDB entirely for durable long-term storage is handed off to cold-storage archival to object store, which picks up where the cold tier ends.

# Configuration reference

Each row is one migration hop. The cadence must satisfy the safe-migration invariant against the source tier’s retention, and the resolution must step down at every hop so no tier inherits its predecessor’s footprint.

Key parameters that govern each hop:

# Common failure modes and fixes

1. Migration slower than expiry — data dropped before it moves. Symptom: the target tier is missing the oldest slice of every source window, and the gap grows whenever the task is delayed. Root cause: the migration cadence plus runtime plus offset has crept past the source retention minus margin, so the expiry boundary drops raw data before the task reads it — the safe-migration invariant is violated. Fix: either shorten the cadence, speed up the run (narrower filters, fewer series per pass), or lengthen the source retention. Re-derive the invariant and leave headroom.

# Widen the hot window so a slow or delayed 15m migration always has slack.
influx bucket update --id "$HOT_BUCKET_ID" --retention 3d

2. Migrating raw resolution into warm instead of downsampling first. Symptom: the warm tier’s disk usage tracks the hot tier’s and its retention budget is exhausted far early; queries against warm are as slow as against raw. Root cause: the migration task copies points with to() but omits aggregateWindow, so the full-resolution firehose lands in a tier sized for aggregates. Fix: always fold the downsample into the migration pass so resolution steps down at the boundary.

// WRONG: copies 30s points straight into warm.
// from(bucket: "meter_load_hot") |> range(start: start) |> to(bucket: "meter_load_warm")

// RIGHT: downsample in the same pass.
from(bucket: "meter_load_hot")
  |> range(start: start)
  |> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
  |> to(bucket: "meter_load_warm", org: "grid-ops")

3. No watermark — a re-run re-copies overlapping windows. Symptom: the target tier shows inflated counts or subtly wrong means after a task retries or a backfill is re-run; the same hour appears migrated twice. Root cause: the task uses a fixed range(start: -15m) with no watermark, so an overlapping retry re-reads and re-aggregates a window that was already moved. When aggregation lands on a stable grid the re-write is harmless overwrite, but a shifted or offset window instead creates duplicate points. Fix: anchor the read to tasks.lastSuccess() (native tasks) or a persisted watermark advanced only after success (Python), and align aggregate windows to a fixed grid so any overwrite is idempotent.

4. Unbounded catch-up after an outage. Symptom: a migration task re-enabled after a long pause runs for a very long time, spikes memory, and may OOM. Root cause: tasks.lastSuccess() with no orTime cap reaches all the way back to the last success, attempting to migrate the entire backlog in one pass. Fix: cap the watermark with orTime and, for large gaps, run the chunked Python orchestrator instead so the backlog is walked in bounded pieces.

5. Overlapping runs double-writing aggregates. Symptom: intermittent duplicate or averaged-twice points that correlate with slow runs. Root cause: concurrency left above 1, so a long run and its successor overlap and migrate the same window simultaneously. Fix: set concurrency: 1 on every migration task; if runs routinely exceed the cadence, that is a signal to lengthen the interval or narrow the per-run window.

# Verification and testing

Migration must be observable: confirm that each window that left the source arrived at the target, that the watermark is advancing, and that a stalled migration pages an operator while the source data still exists to reprocess.

Compare the count of source points migrated against the aggregates that landed in the target for a recent window — the target count should equal the number of aggregate windows, not the raw sample count:

warm = from(bucket: "meter_load_warm")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "meter_load_5m")
  |> filter(fn: (r) => r._field == "active_power_kw")
  |> group()
  |> count()

warm

Confirm the migration is keeping pace by checking that the newest point in the target tier is recent — a target whose latest timestamp lags the cadence means the task is falling behind the expiry race:

from(bucket: "meter_load_warm")
  |> range(start: -2h)
  |> filter(fn: (r) => r._measurement == "meter_load_5m")
  |> last()
  |> keep(columns: ["_time", "_measurement"])

Add a deadman health check so a stalled hot-to-warm migration alerts before the 3-day hot boundary starts dropping unmigrated raw data. This fires when no fresh 5-minute aggregate has landed in the warm tier within the expected window:

import "influxdata/influxdb/monitor"
import "experimental"

from(bucket: "meter_load_warm")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "meter_load_5m")
  |> monitor.deadman(t: experimental.subDuration(from: now(), d: 45m))
  |> filter(fn: (r) => r.dead == true)

A 45-minute deadman on a 15-minute migration gives two missed runs of tolerance before paging — comfortably inside the 3-day hot retention, so an operator has days, not minutes, to intervene before the invariant is breached.

# Integration points

Migration is the moving part that connects the static design layers of the lifecycle. It only makes sense against the tier definitions in bucket architecture & tiering boundaries, which decide how many tiers exist and what resolution each holds, and it is bounded by the windows in retention policy design, whose whole safety argument assumes a migration task actually empties each tier before expiry fires. The downsampling folded into every hop is an application of downsampling & aggregation pipeline design, and when data finally leaves the cold tier for durable storage it is handed to cold-storage archival to object store. Programmatic control of the migration tasks themselves — enabling, pausing, inspecting run logs — is documented in the InfluxDB Task API reference.

# FAQ

# How is tier migration different from retention policy design?

Retention design sets when data in each tier becomes eligible for deletion; migration is the scheduled task that moves and downsamples data to the next tier before that deletion happens. Retention defines the deadline, migration is the work that has to beat it. You need both — a retention window with no migration task just deletes data on schedule, and a migration task with a mis-sized source window loses data before it can run.

# Why downsample during migration instead of after?

Because the target tier is sized for the coarser resolution. If you migrate raw 30-second points into a warm bucket meant for 5-minute means and downsample later, the warm tier carries the full-resolution footprint in the meantime and can exhaust its disk budget before the second step runs. Folding aggregateWindow into the same pass as to() means resolution steps down exactly at the tier boundary, which is the entire point of tiering.

# What stops a re-run from copying the same window twice?

A watermark. Native tasks anchor the read to tasks.lastSuccess(); the Python orchestrator persists a watermark it advances only after a chunk succeeds. Combined with aggregate windows aligned to a fixed time grid, a re-run rewrites identical timestamps rather than appending duplicates, so migration is idempotent even under retries.

# How often should the hot-to-warm task run?

Often enough that the cadence plus the run duration plus the late-data offset stays comfortably below the hot tier’s retention. For a 3-day hot window a 15-minute cadence leaves enormous headroom; the cadence is usually driven instead by how fresh you need the warm aggregates for analytics, not by the expiry race. Tighten it only if downstream queries need lower-latency aggregates.

# Can I use one task to migrate hot straight to cold?

You can, but you lose the warm tier’s mid-resolution history and you force the cold aggregation to read the raw firehose every run, which is expensive at high cardinality. Chaining hop by hop — hot to warm, then warm to cold reading from the already-aggregated warm tier — keeps each run cheap and gives you a 5-minute tier for analytics that a direct hot-to-cold jump would skip.

# Related

• Automating hot-to-warm bucket migration with Flux tasks — the per-window mechanics, cardinality control, and boundary alignment of the core migration task.
• Bucket Architecture & Tiering Boundaries — how the tiers migration moves between are defined and sized.
• Retention Policy Design — the expiry windows every migration task must beat.
• Cold-Storage Archival to Object Store — where data goes once it has aged out of the cold tier.
• Downsampling & Aggregation Pipeline Design — the resolution-reduction logic folded into every migration hop.

Up one level: InfluxDB Data Lifecycle & Architecture Fundamentals