# Sizing shard group duration for IoT retention windows

A fleet of 8,000 refrigerated shipping containers streams reefer telemetry — supply and return air temperature, setpoint, humidity, compressor state, and GPS — every 60 seconds into a reefer-hot bucket set to a 14-day window. Retention behaves, but the storage engine does not: disk usage sawtooths by a fifth of the volume every few days, and roughly every 84 hours a compaction cycle stalls writes long enough that the ingest gateway starts buffering and paging on-call. Nothing in the retention rule is wrong. The problem is the shard-group duration — left at the engine’s default, so an entire tranche of high-velocity data is being reclaimed and compacted in one lump instead of in small, predictable increments. This page shows how to size shard-group duration against the retention window and write rate, using a heuristic and a lookup table, and how to recognise the failure modes at both extremes. It applies the tiering model from Bucket Architecture & Tiering Boundaries to the single most consequential storage knob on a bucket.

# Prerequisites

• InfluxDB 2.7+ (OSS or Cloud) with bucket and retention-rule APIs available.
• The influx CLI 2.7+ with an active --host/--token/--org profile, or those values exported as environment variables.
• An operator or all-access token scoped to create, read, and update buckets.
• A known retention window per tier and a rough points-per-second ingest estimate for the bucket you are sizing (fleet size × fields × sample rate).
• A retention plan agreed up front — see Retention Policy Design — because shard duration is only meaningful relative to the window it subdivides.

# The sizing rule in one relationship

A shard group is the physical unit the Time-Structured Merge (TSM) engine reclaims, compacts, and skips over during queries. Retention never deletes individual points; it drops a shard group whole once that group’s entire time range has aged past the window. So the shard-group duration is simultaneously the reclaim granularity, the compaction batch size, and the coarsest skip unit a range()-bounded query can prune. A good starting point is one order of magnitude below the retention window, clamped to a sane band so you neither reclaim in giant lumps nor drown in tiny files:

$$ D_{shard} \approx \operatorname{clamp}!\left(\frac{R_{retention}}{10},\ 1\text{h},\ 4\text{w}\right) $$

The lower clamp keeps sub-day hot buckets from creating hundreds of shard groups; the upper clamp stops multi-year archival tiers from building shard groups so large that a single compaction reads tens of gigabytes at once. The write rate then breaks ties within the band: at 8,000 containers × ~6 fields × 1 sample/60s the reefer-hot bucket takes roughly 800 points/second, so within the 1-to-3-day range that $R/10$ suggests for a 14-day window, you lean toward the smaller end to keep each shard group’s TSM footprint modest.

# Solution walkthrough

# Step 1 — Estimate the target duration from the window and write rate

Start from the retention window, apply the clamped $R/10$ heuristic, then use the write rate to pick within the resulting band. The table below is the lookup for common IoT tiers; the “InfluxDB default” column is what the engine chooses when you omit --shard-group-duration, and the mismatch in the last two rows is exactly what causes lumpy reclaim.

For the reefer-hot bucket — 14-day window, ~800 points/second — the table lands on a 1-day shard, and the write rate keeps it there rather than pushing to 12h. For the paired reefer-warm bucket at 400 days, the default 7-day shard produces roughly 57 groups; the recommended 4-week shard produces about 15, cutting open file handles and compaction bookkeeping without making any single group unwieldy.

# Step 2 — Create the bucket with the chosen shard duration

Set both durations explicitly so the engine never substitutes a default. The CLI is the most reliable interface for provisioning and infrastructure-as-code.

influx bucket create \
  --name reefer-hot \
  --retention 14d \
  --shard-group-duration 1d \
  --org maritime-telemetry \
  --token $INFLUX_TOKEN

--retention 14d fixes the expiry window; --shard-group-duration 1d fixes the reclaim-and-compaction granularity at one day, so the 800-points-per-second stream is swept and compacted in daily increments rather than one large weekly block. Duration strings accept h, d, and w suffixes. Provision the warm tier alongside it with the wider shard:

influx bucket create \
  --name reefer-warm \
  --retention 400d \
  --shard-group-duration 4w \
  --org maritime-telemetry \
  --token $INFLUX_TOKEN

# Step 3 — Re-size an existing bucket safely

Shard-group duration can be updated on a live bucket, but the new value applies only to shard groups created after the change — existing groups keep the duration they were written with. That means a re-size is a forward-looking correction, not a rewrite, so you plan it around the window rolling over.

# Widen the warm tier's shard duration from the coarse-default 7d to 4w.
influx bucket update \
  --id 0a91ff_reefer_warm_id \
  --shard-group-duration 4w \
  --token $INFLUX_TOKEN

Because only new shard groups adopt 4w, the bucket runs mixed durations until every pre-existing 7-day group has aged out of the 400-day window. The mixed state is harmless — retention still drops each group whole on its own boundary — but it means you should verify the change took effect on freshly written data, not on historical shard groups. If you need the whole bucket at one duration immediately, the clean path is to create a new correctly-sized bucket and migrate, which is the job of the automated Storage Tier Migration Automation tasks rather than an in-place edit.

# Gotchas and edge cases

Shard duration larger than the retention window disables incremental reclaim. If a 14-day bucket carries a 30-day shard duration, no shard group’s full range ever ages past the window before the next one starts, so the engine holds far more data than the retention rule implies and then reclaims it in one violent lump. Keep the shard duration strictly below the window — the $R/10$ heuristic guarantees this — and never let a copy-pasted long-tier duration land on a short hot bucket.

Too-fine shards exhaust file handles and trigger compaction storms. Each shard group is a set of open TSM and index files. Slice a 400-day window into 1-day shards and you carry ~400 groups and their descriptors at once; under sustained ingest the compactor spends its time merging tiny files instead of serving queries, and the process can hit the OS open-file limit. This is the mirror image of the coarse failure and the reason the reefer-warm tier uses 4-week shards, not the fine default.

Changing the duration does not touch existing shard groups. As in Step 3, bucket update is forward-only. Teams routinely “fix” a lumpy bucket, see no change in disk behaviour that week, and assume the setting was ignored — when in fact the old-duration groups simply have not rolled over yet. Confirm the new duration on data written after the change, and give the window a full turnover before judging the result.

# Verification

Confirm the engine accepted the durations you set — a mismatch here is a provisioning bug, not a rounding artifact:

influx bucket list --org maritime-telemetry --token $INFLUX_TOKEN
# reefer-hot  -> Retention 336h0m0s (14d)   Shard-group-duration 24h0m0s (1d)
# reefer-warm -> Retention 9600h0m0s (400d) Shard-group-duration 672h0m0s (4w)

Then check that reclaim is actually incremental: the count in the day-wide window just past the boundary should be zero once its shard group has aged out, while the window just inside stays populated.

from(bucket: "reefer-hot")
    |> range(start: -15d, stop: -14d)
    |> filter(fn: (r) => r._measurement == "reefer_status")
    |> count()
    |> yield(name: "just_past_boundary_should_be_zero")

A zero here with a healthy count in the -14d-to--13d window confirms the sweeper is dropping one day-sized shard group at a time. A non-zero count means the group straddling the boundary has not fully aged — expected transiently — but a persistently non-zero count across several runs is the signal that the shard duration is coarser than you think for that window.

# FAQ

# Does shard-group duration affect query speed, or only storage reclaim?

Both. Because a range()-bounded query can skip any shard group whose time span lies entirely outside the requested window, the shard duration is the coarsest unit of pruning. On the reefer-hot bucket, a dashboard asking for the last two hours touches a single 1-day group; the same query against a bucket with a 30-day shard would scan the whole month of files to answer a two-hour question. Sizing for smooth reclaim usually sizes well for pruning too, which is why the balanced middle of the diagram serves both goals.

# Should every tier of the same fleet use the same shard duration?

No — size each tier to its own retention window, not to the fleet. The reefer-hot bucket at 14 days wants a 1-day shard, while the reefer-warm bucket at 400 days wants roughly 4 weeks, even though both carry the identical device stream. Copying the hot tier’s fine shard onto the warm tier is precisely what produces hundreds of open file handles and constant small compactions across a year of data.

# Related

• Bucket Architecture & Tiering Boundaries — the hot/warm/cold tiering model this shard sizing serves.
• Retention Policy Design — how to set the retention windows that the shard durations here subdivide.
• Storage Tier Migration Automation — automated tasks for moving data between correctly-sized buckets when an in-place re-size is not enough.

Up: Bucket Architecture & Tiering Boundaries