Bounding series cardinality in Flux rollup tasks
A ride-share e-scooter operator runs a ride_metrics measurement in a hot bucket tagged with device_id (about 48,000 live scooters), region (14 cities), model (3 hardware revisions), and — the silent killer — a session_id and trip_id minted fresh for every ride. The raw bucket ingests fine because each series lives briefly and ages out under retention. Then someone writes a 5-minute mean rollup task that pipes those same tags straight into a long-lived aggregate bucket, and within a week the task starts failing with runtime error: memory allocation limit reached, the aggregate bucket’s index balloons past the raw bucket’s, and dashboard queries that used to take 200 ms crawl. The rollup did not aggregate the fleet — it faithfully copied millions of unique trip series into a tier meant to be small and permanent. Bounding cardinality is the discipline of stripping unbounded tags and regrouping on a stable key before aggregateWindow runs, so the aggregate tier’s series count stays flat no matter how many trips the fleet takes. This page is the runnable companion to Flux Scripting for Task Automation; it turns the “keep your rollups bounded” rule into a task you can deploy today.
Prerequisites
Why cardinality is a product, not a sum
Series cardinality in InfluxDB is the number of distinct combinations of measurement, tag set, and field key that the storage index must track. It is multiplicative: every tag you keep multiplies the series count by the number of distinct values that tag can take. For $n$ tags whose value counts are $|T_1|, |T_2|, \dots, |T_n|$, the series count for one measurement and field is
$$ C = \prod_{i=1}^{n} |T_i| $$
For the scooter fleet, a rollup that keeps region (14) and model (3) produces at most $14 \times 3 = 42$ series per field — trivial and permanent. Add device_id (48,000) and you jump to about 2 million. Add session_id, which is unbounded because a new value appears on every trip, and $C$ has no ceiling at all: it grows without limit for as long as the task runs, because yesterday’s trip series never come back but never leave the index either. That last term is the difference between a rollup that costs kilobytes and one that OOMs the task executor. The fix is never “tune the memory limit” — it is to remove the unbounded factors from the product before the aggregate is written.
Solution walkthrough
1. Measure the cardinality you are about to inherit
Before changing the task, quantify what the source is carrying so you know which tag is the culprit and can confirm the fix later. influxdb.cardinality() reports the exact series count over a time range without scanning the field data.
import "influxdata/influxdb"
influxdb.cardinality(bucket: "scooter_telemetry_raw", start: -1h)
|> yield(name: "raw_series_last_hour")
Run the same call against the aggregate bucket. If the aggregate count is climbing hour over hour — rather than holding at a fixed number like 42 or 84 — an unbounded tag is leaking through the rollup. To pinpoint which tag, count distinct values per tag with schema.tagValues and look for the one whose count tracks trip volume rather than fleet size. session_id and trip_id will grow every hour; region and model will not.
2. Drop the unbounded tags before anything aggregates
The instinct to fix cardinality after aggregation is wrong: aggregateWindow preserves the existing group key, so any unbounded tag still in the key at that point survives into the output. Strip the offending columns first with drop, which removes them from the group key and from the written series entirely.
from(bucket: "scooter_telemetry_raw")
|> range(start: -task.every)
|> filter(fn: (r) => r._measurement == "ride_metrics")
|> filter(fn: (r) => r._field == "battery_pct" or r._field == "speed_kph")
|> drop(columns: ["session_id", "trip_id", "device_id", "firmware_build"])
Here device_id is dropped too because the aggregate tier only needs city-and-model trends; if you genuinely need per-device rollups later, keep device_id (bounded at fleet size) but never keep session_id or trip_id. Prefer drop (an explicit deny-list of the columns you are removing) or its inverse keep(columns: [...]) (an explicit allow-list) over hoping a downstream stage will collapse them.
3. Regroup on a bounded key, then aggregate and write
With the unbounded tags gone, set an explicit group key of only bounded tags plus _field, then run the windowed mean and write it out. The complete task:
option task = {name: "rollup-scooter-telemetry-5m", every: 5m, offset: 30s}
from(bucket: "scooter_telemetry_raw")
|> range(start: -task.every)
|> filter(fn: (r) => r._measurement == "ride_metrics")
|> filter(fn: (r) => r._field == "battery_pct" or r._field == "speed_kph")
|> drop(columns: ["session_id", "trip_id", "device_id", "firmware_build"])
|> group(columns: ["region", "model", "_field"])
|> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
|> to(bucket: "scooter_telemetry_5m", org: "mobility-ops")
The group(columns: ["region", "model", "_field"]) line is what caps the product at $14 \times 3 \times 2 = 84$ series. aggregateWindow now computes one mean per city-model-field bucket per 5-minute window, and to writes exactly those tags — the aggregate index can never outgrow the bounded key. createEmpty: false keeps sparse windows from fabricating null series, and offset: 30s lets late edge writes land before the window closes. For the broader anatomy of a resilient rollup task — anchoring the window and surviving retries — see writing robust Flux scripts for automated data rollups.
4. Enforce a cardinality budget as a guard
A rollup that is bounded today can regress the moment someone adds a tag upstream. Codify the ceiling as a guard task that fails loudly instead of silently bloating the index. This task computes the aggregate bucket’s series count and errors if it exceeds the agreed budget.
import "influxdata/influxdb"
budget = 200
count = influxdb.cardinality(bucket: "scooter_telemetry_5m", start: -1h)
|> findColumn(fn: (key) => true, column: "_value")
if count[0] > budget then
die(msg: "cardinality budget exceeded: aggregate bucket carries a leaking tag")
else
count
Schedule this alongside the rollup so a cardinality regression pages you within the hour rather than surfacing as an out-of-memory crash days later. Treating the budget as an explicit number — not a hope — is the same defensive posture that runs through the rest of task scheduling and orchestration.
Gotchas and edge cases
Dropping a tag after aggregateWindow does nothing for cardinality. aggregateWindow inherits the group key, so if session_id is still in the key when it runs, the engine allocates one series per trip during aggregation — that is where the OOM happens, before any later drop executes. The order is non-negotiable: strip unbounded tags and regroup first, aggregate second. A task that “drops session_id” but does it after the window will still crash.
Bounding the aggregate does not shrink the raw bucket’s index. Removing tags in the rollup only protects the destination tier. The source scooter_telemetry_raw bucket still carries every trip series until retention ages it out, so keep the raw window short and let expiry reclaim it — a concern that belongs to how you draw bucket architecture and tiering boundaries, not to the rollup. If the raw bucket itself is straining the index, the answer is retention and partitioning, not a busier task.
Collapsing series changes what the mean means. When you drop device_id, every scooter in a city-model group folds into one series, so mean now averages across the whole sub-fleet rather than per device. That is usually the intent for a city-level dashboard, but if a downstream alert assumed per-device granularity it will misfire. Decide explicitly which dimensions the aggregate tier must preserve, and keep exactly those bounded tags — no more, no fewer.
Verification
Confirm the aggregate bucket’s cardinality is flat and matches the bounded product you designed for. Run this a few hours after the task has been live:
import "influxdata/influxdb"
influxdb.cardinality(bucket: "scooter_telemetry_5m", start: -1h)
|> yield(name: "aggregate_series_now")
A steady result at or below your budget (84 series for two fields across 14 regions and 3 models) proves the unbounded tags are no longer leaking; a number that climbs each time you run it means a tag survived the drop — recheck that session_id, trip_id, and device_id are all in the drop list and that group runs before aggregateWindow. Comparing the count against the previous hour turns cardinality from an invisible failure mode into a single number you can watch.
FAQ
Should I drop device_id or keep it in the rollup?
Keep it only if the aggregate tier genuinely serves per-device queries, because device_id is bounded at fleet size (about 48,000) and adds a large but finite factor. If your dashboards and alerts operate at the city or model level, drop it — you shrink the index by four orders of magnitude and lose nothing you query. Never keep session_id or trip_id regardless, since those are unbounded and have no ceiling.
Why not just raise the task’s memory limit?
Because the memory demand grows with trip volume, so any limit you set is a limit you will hit again. Cardinality is a product of tag value counts; an unbounded tag makes that product unbounded, and no fixed memory ceiling survives an unbounded input. Removing the unbounded factor is the only fix that scales — tuning memory just delays the same crash.
How is bounding cardinality different from downsampling?
Downsampling reduces cardinality along the time axis by collapsing many timestamps into fewer windowed points; bounding cardinality reduces it along the tag axis by collapsing many series into fewer. A rollup task usually needs both: aggregateWindow handles time, while drop and group handle tags. Doing only the time part is exactly how a “downsampling” task still explodes the index.
Related
- Flux Scripting for Task Automation — the parent guide to authoring safe, idempotent Flux tasks that this cardinality discipline plugs into.
- Writing Robust Flux Scripts for Automated Data Rollups — window anchoring, retries, and watermarks for the surrounding rollup task.
- Bucket Architecture & Tiering Boundaries — where raw versus aggregate index pressure is designed away with retention and partitioning.
Up one level: Flux Scripting for Task Automation