Task Automation

Automating hot-to-warm bucket migration with Flux tasks

A grid-scale battery energy storage site writes cell-level telemetry at 1 Hz: for every one of its 9,600 lithium cells the battery management system (BMS) reports voltage_v, temp_c, and soc_pct once a second into a bms_hot bucket kept at seven-day retention. That raw fidelity is priceless during a thermal event and worthless a week later, so the plan is a warm tier at one-minute resolution and ninety-day retention. The trap is timing: if the migration is a naive nightly task and one run fails, the seven-day window keeps sweeping while the migration falls behind, and the oldest minutes are reclaimed from bms_hot before they are ever aggregated into bms_warm — silent, permanent loss with no error in any log. The fix is a task that advances a watermark from its own last successful run, reads exactly the un-migrated window, and writes idempotently so a re-run never double-counts. This page builds that task step by step as a concrete instance of storage tier migration automation.

Prerequisites

Core relationship: lag budget versus hot retention

The whole design reduces to keeping one inequality true on every run. Let $R_{hot}$ be the hot retention window, $\Delta_{lag}$ the gap between now and the watermark the task is about to resume from, and $\Delta_{run}$ the wall-clock a single run takes to read and write its window. The migration is safe only while:

$$ \Delta_{lag} + \Delta_{run} < R_{hot} $$

With $R_{hot} = 7,\text{d}$ and a task that runs every 15 minutes, a healthy watermark sits minutes behind now, leaving days of margin. The danger is not steady state — it is accumulated failure: every skipped or errored run adds its interval to $\Delta_{lag}$, and once the sum approaches $R_{hot}$ the task begins reading a window whose oldest edge has already been reclaimed. The watermark is what makes that lag visible and recoverable instead of catastrophic.

Watermark stays ahead of the hot retention boundary The task resumes from tasks.lastSuccess (the watermark) and reads up to now minus a small offset. The safety margin is the distance between the hot retention boundary at now minus seven days and the watermark; it must remain positive or the oldest minutes are reclaimed before migration reads them. bms_hot · 1 Hz raw · 7-day retention already reclaimed already migrated this run reads watermark → now−offset hot boundary (now−7d) watermark (lastSuccess) now safety margin · must stay positive bms_warm · 1-min aggregates · 90 days ← older

Solution walkthrough

Step 1 — Provision the two tiers with a deliberate retention gap

The warm bucket must outlive the hot bucket by a wide margin, and its shard-group duration should match its coarser cadence. Create both explicitly so neither falls back to a default.

bash
# Hot: raw 1 Hz, short life, fine shards for smooth reclaim
influx bucket create --name bms_hot  --retention 7d   --shard-group-duration 6h \
  --org grid-storage-ops --token $INFLUX_TOKEN

# Warm: 1-minute aggregates, long life, coarser shards
influx bucket create --name bms_warm --retention 90d  --shard-group-duration 1d \
  --org grid-storage-ops --token $INFLUX_TOKEN

The 7-day hot window gives the migration days of slack, and the 6-hour hot shard duration keeps expiry smooth so a stalled migration is not competing with a huge weekly compaction. The durations here follow the same sizing discipline described in retention policy design; the only new constraint is that warm retention strictly exceeds hot retention.

Step 2 — Write the watermarked migration task

The heart of the design is tasks.lastSuccess, which returns the timestamp of the task’s most recent successful run. Resuming from it — rather than from a fixed -task.every window — means a run that was delayed or that had to be retried still picks up exactly where the last success left off, with no gap and no overlap.

flux
import "influxdata/influxdb/tasks"

option task = {name: "migrate-bms-hot-to-warm", every: 15m, offset: 2m}

// Resume from the last successful run; on the very first run fall back
// one interval so the task has a defined starting window.
watermark = tasks.lastSuccess(orTime: -task.every)

from(bucket: "bms_hot")
    |> range(start: watermark, stop: -1m)
    |> filter(fn: (r) => r._measurement == "cell_metrics")
    |> filter(fn: (r) => r._field == "voltage_v" or r._field == "temp_c" or r._field == "soc_pct")
    |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)
    |> set(key: "tier", value: "warm")
    |> to(bucket: "bms_warm", org: "grid-storage-ops")

Three parameters carry the load. offset: 2m delays the trigger so late 1 Hz batches from cells on flaky cellular backhaul have landed before the window is read. stop: -1m excludes the current, still-filling minute so no partial bucket is written and then never corrected. createEmpty: false keeps genuine sensor gaps — a cell taken offline for a fault — as gaps rather than fabricating null or zero minutes. Because aggregateWindow stamps each output point at a deterministic minute boundary and the cell_id/pack_id tags pass through unchanged, re-writing the same window produces identical series keys and timestamps, so InfluxDB overwrites in place instead of duplicating — the property that makes the whole task idempotent.

Step 3 — Preserve counts so downstream rollups stay honest

A one-minute mean discards how many raw samples it summarised. That is fine until bms_warm is itself rolled to daily figures, where a minute built from 60 samples must not weigh the same as a minute built from 3 during a partial outage. Carry the sample count alongside the mean so later tiers can compute count-weighted aggregates.

flux
import "influxdata/influxdb/tasks"

option task = {name: "migrate-bms-hot-to-warm", every: 15m, offset: 2m}

src =
    from(bucket: "bms_hot")
        |> range(start: tasks.lastSuccess(orTime: -task.every), stop: -1m)
        |> filter(fn: (r) => r._measurement == "cell_metrics")
        |> filter(fn: (r) => r._field == "voltage_v" or r._field == "temp_c")

means = src |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)

counts =
    src
        |> aggregateWindow(every: 1m, fn: count, createEmpty: false)
        |> map(fn: (r) => ({r with _field: r._field + "_n"}))

union(tables: [means, counts])
    |> set(key: "tier", value: "warm")
    |> to(bucket: "bms_warm", org: "grid-storage-ops")

Now every migrated minute carries both voltage_v and voltage_v_n (its sample count). A daily rollup can then compute a true count-weighted mean instead of a lossy mean-of-means — the exact weighting problem worked through in multi-tier rollup scheduling. Emitting the count is cheap insurance against distortion that only becomes visible once the raw tier has expired and can no longer be recomputed.

Step 4 — Bound the catch-up window after downtime

When the task has been down long enough that the watermark trails far behind, an unbounded range(start: watermark) can try to read hours or days in a single run — a query heavy enough to time out and fail, which prevents the watermark from advancing and wedges the task permanently. Cap how far back any one run reaches so recovery proceeds in safe increments.

flux
import "influxdata/influxdb/tasks"
import "date"

option task = {name: "migrate-bms-hot-to-warm", every: 15m, offset: 2m}

raw = tasks.lastSuccess(orTime: -task.every)
// Never look back more than 6h in a single run, even after long downtime.
floor = date.sub(from: now(), d: 6h)
start = if raw < floor then floor else raw

from(bucket: "bms_hot")
    |> range(start: start, stop: -1m)
    |> filter(fn: (r) => r._measurement == "cell_metrics")
    |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)
    |> set(key: "tier", value: "warm")
    |> to(bucket: "bms_warm", org: "grid-storage-ops")

Each run migrates at most six hours and the watermark advances a step at a time, so a task that fell a day behind heals over several intervals without a single crushing query. Any minutes older than the six-hour floor that were also older than hot retention are unrecoverable — which is precisely why Step 5 alerts on lag long before it reaches that point.

Gotchas and edge cases

Silent watermark drift wedges the migration. tasks.lastSuccess advances only on success, so a task failing every run — a revoked token, a warm bucket at quota — leaves the watermark frozen while bms_hot keeps expiring. Nothing errors loudly; data simply stops arriving in warm and the oldest hot minutes are reclaimed before the next success reads them. Treat rising lag as a first-class alert, not a log line, using the health check below.

Mean-of-means silently distorts uneven minutes. If you skip the count field from Step 3 and later average one-minute means into daily values, a minute assembled from three samples counts as much as one from sixty, skewing the day. Always carry the sample count when a warm tier feeds a further rollup; the mean alone is not a sufficient summary for additive downstream math.

createEmpty: true fabricates data across real outages. Setting createEmpty: true writes a null or zero-backed point for every minute a cell was silent, turning an honest gap into what looks like a live-but-flat reading — dangerous when the gap coincides with a fault you later investigate. Keep createEmpty: false on migration tasks so absence stays absence; detect the silence separately with a deadman check rather than papering over it in the warm tier.

Verification

First confirm the watermark is fresh — a lag far below the seven-day boundary proves the migration is keeping up:

bash
influx task list --org grid-storage-ops --token $INFLUX_TOKEN
# Note the task ID, then inspect its most recent run's finishedAt/status:
influx task run list --task-id <TASK_ID> --org grid-storage-ops --token $INFLUX_TOKEN
# 'success' with a finishedAt within the last ~15m means the watermark is advancing.

Then prove the migration is neither dropping nor duplicating minutes by comparing a completed hour. The raw count divided by 60 should closely match the number of warm minutes, and each warm minute should be unique:

flux
warm = from(bucket: "bms_warm")
    |> range(start: -2h, stop: -1h)
    |> filter(fn: (r) => r._measurement == "cell_metrics" and r._field == "voltage_v")
    |> filter(fn: (r) => r.cell_id == "A17")
    |> count()
    |> yield(name: "warm_minutes_should_be_~60")

Finally, guard against the wedge failure directly with a monitor.deadman pattern that fires if no fresh point has reached the warm tier — the canonical signal that the watermark has stopped advancing:

flux
import "influxdata/influxdb/monitor"
import "influxdata/influxdb/schema"

from(bucket: "bms_warm")
    |> range(start: -45m)
    |> filter(fn: (r) => r._measurement == "cell_metrics" and r._field == "voltage_v")
    |> monitor.deadman(t: experimental.subDuration(from: now(), d: 30m))
    |> filter(fn: (r) => r.dead)
    |> yield(name: "migration_stalled_if_any_rows")

Any rows from that query mean warm ingestion has been silent for over thirty minutes — a stalled migration to page on before lag erodes the safety margin. For the anchoring and retry-safety patterns these tasks depend on, see writing robust Flux scripts for automated data rollups.

FAQ

Why use tasks.lastSuccess instead of a fixed range like -task.every?

A fixed -task.every window assumes every run fires exactly on time and succeeds. The moment one run is delayed or retried, a fixed window either skips the minutes between the missed slot and the next run or re-reads an overlapping slice. tasks.lastSuccess anchors each run to the last successful one, so the migrated window is always contiguous and gap-free regardless of scheduling jitter or transient failures.

Is re-running the task safe, or will it double-count?

Re-running is safe because the write is idempotent. aggregateWindow stamps each output at a fixed minute boundary and the cell_id/pack_id tags are unchanged, so a re-migrated minute has an identical series key and timestamp and InfluxDB overwrites it in place. Duplicates only appear if you change the tag set or the window alignment between runs, which you should avoid on a live migration.

How do I backfill data that predates the task?

Run a one-off task or client script with an explicit range(start: ..., stop: ...) covering the historical window, writing to bms_warm with the identical measurement, tags, fields, and one-minute alignment the live task uses. Because the write is idempotent, the backfill and the live task can safely overlap at the seam without producing duplicate points.

Up: Storage Tier Migration Automation