# Cold-Storage Archival to Object Store

Retention in InfluxDB is a hard delete: once a shard group ages past its window the background sweep drops it, and there is no undo. That is fine for data you have already downsampled and no longer need at full resolution — but for a regulated operation, the raw history you are obligated to keep for years is far too expensive to leave sitting in a live time-series engine. The answer is a durable, cheap, re-loadable copy in object storage that is written before the retention boundary fires, so expiry only ever deletes data that already exists somewhere else. This page sits under InfluxDB Data Lifecycle & Architecture Fundamentals and covers how to automate the export of aged partitions to S3-compatible object storage in a format that survives a decade, records a manifest and watermark so gaps are impossible to miss, and can be re-ingested for a compliance audit or a forensic investigation years later.

# The failure scenario this solves

An offshore hydrocarbon operator runs 4,200 accelerometers across 14 North Sea platforms, each streaming turbine and pump vibration at 10 ms resolution into a bucket named rig_vibration_cold. The regulator requires that raw machinery-health data be retrievable for seven years after a well is decommissioned — but keeping seven years of 10 ms samples for 4,200 channels online would need roughly 90 TB of hot storage and would make the live engine crawl. So the platform holds 90 days of cold data live and sets the bucket to expire at 90 days, on the assumption that “the archive job has it.”

The archive job did not have all of it. It ran nightly, exported “the last day” with a fixed range(start: -24h), and wrote every export to the same object key, rig_vibration/archive.csv, overwriting the previous night’s file. For months nobody looked. Then a gearbox on Platform 7 threw a bearing, an incident investigation was opened, and the regulator asked for the full-resolution vibration trace from 14 months earlier. The team went to object storage and found a single 40 MB file containing one day of data — every prior night had been overwritten. The 90-day live window was long gone. There was no manifest, so no alarm had ever fired for the 400-plus days of history that had quietly expired without a durable copy. The raw record the operator was legally required to produce simply did not exist.

Three design defects caused this: a single object key instead of date-partitioned keys, no manifest to make gaps observable, and — most dangerous — no verification that the object was durable before the retention sweep was allowed to delete the source. The rest of this page fixes all three: it exports the oldest window (not an arbitrary fixed range), writes date-partitioned keys, records a manifest watermark, and verifies the object exists and matches the source count before anything is eligible for deletion.

# Prerequisites

• InfluxDB 2.7+ (or InfluxDB 3.x Cloud/Dedicated) with the native task engine and a finite retention window on the cold bucket.
• Flux 0.x (bundled) for the count and watermark queries used to verify each export.
• Python 3.9+ with influxdb-client 1.36+ and boto3 1.28+ (or any S3-compatible SDK) for the export job.
• An S3-compatible object store bucket with versioning and a lifecycle rule; a read/write access key scoped to that bucket only.
• An InfluxDB token scoped to read on the cold bucket and read/write on a small archive_state bucket that holds the watermark.
• Timestamps written in RFC 3339 / UTC so date-partition keys and windows are unambiguous; see the RFC 3339 timestamp specification.

# Core concept: a durable copy must precede deletion

Archival to object storage is governed by one invariant: for every window of source data, a verified durable copy exists in the object store before that window becomes eligible for retention deletion. Break the invariant in either direction and you get the failure above — either expiry outruns the export, or the export “succeeds” but the object was never actually written. The whole design is a discipline for keeping the invariant true continuously and provably.

The mechanics are a loop. A scheduled export job reads the oldest un-archived window from the cold bucket — anchored not to a fixed -24h but to a watermark that records how far archival has progressed. It serializes that window into a portable, columnar-friendly format (line protocol, CSV, or Parquet), writes it to a date-partitioned object key such as year=2026/month=03/day=09/, records a manifest row (key, time range, row count, checksum), verifies the object is readable and its row count matches the source, and only then advances the watermark. Retention on the source bucket is deliberately set longer than the archival lag so the sweep never reaches a window the job has not yet confirmed. The safety condition is:

$$R_{cold} \ge L_{archive} + P_{period} + M_{margin}$$

where $R_{cold}$ is the cold-bucket retention window, $L_{archive}$ the worst-case archival lag (how far behind the newest data the job runs), $P_{period}$ the export period, and $M_{margin}$ a buffer for a failed or retried run. Keep the retention window comfortably above the right-hand side and the object store is always ahead of the delete.

# Step-by-step implementation

# 1. Provision the cold tier and an export watermark

Give the cold bucket a finite retention window that clears the safety condition, and create a tiny archive_state bucket to hold the watermark — the timestamp up to which archival has been verified. Keeping the watermark inside InfluxDB (rather than a local file) means the job is stateless and can run from any worker; the cold-tier sizing itself follows the tier boundaries set in retention policy design.

# 90-day cold window, 7-day shards so expiry reclaims weekly.
influx bucket create --name rig_vibration_cold \
  --retention 90d --shard-group-duration 7d --org rig-platform

# Tiny state bucket, never expires — holds the archival watermark.
influx bucket create --name archive_state \
  --retention 0 --org rig-platform

The 90-day window is far longer than the daily export period plus its worst-case lag, so the retention sweep can never reach an un-archived window. Never set the cold bucket to infinite retention as a substitute for archival — an infinite bucket grows without bound and still gives you no cheap, portable copy.

# 2. Page the oldest window out to a portable file

Read the oldest un-archived window, bounded by the watermark, one day at a time. Serialize to Parquet (columnar, compresses vibration data ~8:1, re-loadable by any analytics engine) with line protocol as the fallback format when you need a byte-exact re-ingest. Paging one day per run keeps memory flat regardless of fleet size and makes each object independently restorable. This job uses the same connection-and-retry discipline described in Python client orchestration patterns.

import os, datetime as dt
import pyarrow as pa, pyarrow.parquet as pq
from influxdb_client import InfluxDBClient

ORG = os.environ["INFLUX_ORG"]
COLD_BUCKET = "rig_vibration_cold"

def read_watermark(client) -> dt.datetime:
    flux = '''from(bucket: "archive_state")
        |> range(start: 0)
        |> filter(fn: (r) => r._measurement == "archive_wm" and r._field == "watermark")
        |> last()'''
    tables = client.query_api().query(flux, org=ORG)
    if tables and tables[0].records:
        return tables[0].records[0].get_value()
    return dt.datetime(2026, 1, 1, tzinfo=dt.timezone.utc)  # cold-tier genesis

def export_oldest_day(client) -> tuple[dt.datetime, str, int]:
    start = read_watermark(client)
    stop = start + dt.timedelta(days=1)
    flux = f'''from(bucket: "{COLD_BUCKET}")
        |> range(start: {start.isoformat()}, stop: {stop.isoformat()})
        |> filter(fn: (r) => r._measurement == "turbine_vibration")'''
    records = [
        {"time": r.get_time().isoformat(), "platform": r.values.get("platform"),
         "channel": r.values.get("channel"), "field": r.get_field(), "value": r.get_value()}
        for tbl in client.query_api().query(flux, org=ORG) for r in tbl.records
    ]
    local = f"/tmp/part-{start:%Y-%m-%d}.parquet"
    pq.write_table(pa.Table.from_pylist(records), local, compression="zstd")
    return start, local, len(records)

The window boundaries are exclusive of stop, so consecutive days never overlap and a re-run of the same watermark produces the identical file — the property that makes the export idempotent in step 4.

# 3. Write date-partitioned object keys with a manifest

Compute a date-partitioned key so every day lands at its own address, then write the object and append a manifest row. The manifest — one JSON line per exported window with its key, time range, row count, and checksum — is what turns a silent gap into a detectable one: a missing day is a missing manifest line, trivially found by the verification query later. Deriving where cold data even comes from, and how it is promoted between tiers, is the job of storage tier migration automation; this step assumes the data has already landed in the cold bucket.

import hashlib, json, boto3

S3_BUCKET = os.environ["ARCHIVE_BUCKET"]
s3 = boto3.client("s3", endpoint_url=os.environ.get("S3_ENDPOINT"))

def object_key(day: dt.datetime) -> str:
    # Hive-style partitioning: cheap to prefix-list and range-scan by date.
    return (f"rig_vibration/year={day:%Y}/month={day:%m}/"
            f"day={day:%d}/part.parquet")

def upload_and_manifest(day, local_path, row_count):
    key = object_key(day)
    with open(local_path, "rb") as fh:
        body = fh.read()
    checksum = hashlib.sha256(body).hexdigest()
    s3.put_object(Bucket=S3_BUCKET, Key=key, Body=body,
                  Metadata={"rows": str(row_count), "sha256": checksum})
    manifest_line = json.dumps({
        "key": key, "day": day.strftime("%Y-%m-%d"),
        "rows": row_count, "sha256": checksum,
        "written_at": dt.datetime.now(dt.timezone.utc).isoformat(),
    })
    s3.put_object(Bucket=S3_BUCKET,
                  Key=f"rig_vibration/manifest/{day:%Y-%m-%d}.json",
                  Body=manifest_line.encode())
    return key, checksum

Because the key is derived purely from the day, re-uploading the same window is an overwrite to the same address — not a new file — so retries never fan out into duplicate objects. Enable versioning on the object-storage bucket so even an overwrite keeps the prior copy recoverable.

# 4. Verify the object before allowing expiry

This is the step the failure scenario omitted. Before advancing the watermark — and therefore before the source becomes eligible for deletion — read the object back and confirm its row count matches the source count for that window. Only a passing parity check writes the new watermark. If verification fails, the watermark stays put, the source data remains protected by the still-longer retention window, and the next run retries the same day.

def source_count(client, start, stop) -> int:
    flux = f'''from(bucket: "{COLD_BUCKET}")
        |> range(start: {start.isoformat()}, stop: {stop.isoformat()})
        |> filter(fn: (r) => r._measurement == "turbine_vibration")
        |> count()
        |> sum()'''
    tables = client.query_api().query(flux, org=ORG)
    return tables[0].records[0].get_value() if tables and tables[0].records else 0

def verify_and_advance(client, day, key, row_count):
    start, stop = day, day + dt.timedelta(days=1)
    head = s3.head_object(Bucket=S3_BUCKET, Key=key)   # object must exist
    stored_rows = int(head["Metadata"]["rows"])
    src = source_count(client, start, stop)
    if not (stored_rows == row_count == src):
        raise RuntimeError(
            f"parity failed {day:%Y-%m-%d}: object={stored_rows} "
            f"file={row_count} source={src} — watermark NOT advanced")
    # Passed: advance the watermark to the end of the verified window.
    lp = (f"archive_wm watermark={int(stop.timestamp())}i "
          f"{int(dt.datetime.now(dt.timezone.utc).timestamp() * 1e9)}")
    client.write_api().write(bucket="archive_state", org=ORG, record=lp)

The parity check compares three independent counts — the count reported by the object’s metadata, the count of rows the job serialized, and a fresh count queried from the source. Requiring all three to agree catches a truncated upload, a partial read, and a silently mutated source alike. The watermark advances by exactly one verified day per run, so archival progress is monotonic and auditable.

# 5. Restore and re-ingest from the archive

An archive that cannot be re-loaded is not an archive. When a regulator or an incident team needs a window back, list the date-partitioned keys for the range, pull the Parquet objects, and re-ingest into a scratch bucket (never overwrite the live cold tier). Because keys are Hive-partitioned by date, restoring “March 2026” is a single prefix list.

import pyarrow.parquet as pq
from influxdb_client import Point, WritePrecision

def restore_range(client, first_day: str, last_day: str, into="rig_vibration_restore"):
    resp = s3.list_objects_v2(Bucket=S3_BUCKET, Prefix="rig_vibration/year=")
    write_api = client.write_api()
    for obj in resp.get("Contents", []):
        key = obj["Key"]
        if not key.endswith("part.parquet"):
            continue
        # crude date filter; production code parses the year=/month=/day= tokens
        if not (first_day <= _key_date(key) <= last_day):
            continue
        body = s3.get_object(Bucket=S3_BUCKET, Key=key)["Body"].read()
        table = pq.read_table(pa.BufferReader(body))
        for row in table.to_pylist():
            p = (Point("turbine_vibration")
                 .tag("platform", row["platform"]).tag("channel", row["channel"])
                 .field(row["field"], row["value"])
                 .time(row["time"], WritePrecision.NS))
            write_api.write(bucket=into, org=ORG, record=p)
    write_api.flush()

def _key_date(key: str) -> str:
    parts = dict(seg.split("=") for seg in key.split("/") if "=" in seg)
    return f"{parts['year']}-{parts['month']}-{parts['day']}"

Re-ingesting into a dedicated restore bucket keeps forensic reconstruction fully isolated from production retention, and — because the original timestamps are preserved — the restored series lines up exactly with whatever live data still overlaps. The step-by-step mechanics of the export half of this loop are expanded in the walkthrough on exporting cold partitions to S3 before expiry.

# Configuration reference

Choose the coldest storage class deliberately: deep-archive tiers can impose hours of restore latency and a minimum billable duration, which is fine for a seven-year regulatory copy but wrong for data you might query next week.

# Common failure modes and fixes

1. Expiry beats the export. Symptom: windows vanish from the cold bucket with no matching object; the manifest has gaps at the oldest end. Root cause: the retention window is too close to the archival lag, so the sweep reaches a window before the job verifies it. Fix: enforce $R_{cold} \ge L_{archive} + P_{period} + M_{margin}$ and let the watermark, not the retention sweep, be the thing that authorizes deletion. If the job falls behind, retention (being longer) still protects the backlog.

# Widen the cold window if the export lag ever approaches it.
influx bucket update --id "$COLD_BUCKET_ID" --retention 120d

2. No manifest, so gaps go unnoticed. Symptom: an audit years later finds missing days that no alarm ever caught. Root cause: the job wrote objects but recorded no index of what it wrote, so absence is invisible. Fix: append a manifest line per window and run a scheduled query that asserts one manifest entry exists for every day inside the retention window; alert on any missing day.

3. One giant object instead of date-partitioned keys. Symptom: restoring a single day forces a download of the entire history; a re-run overwrites yesterday’s export. Root cause: a fixed object key (archive.csv) or an unbounded range in one blob. Fix: derive the key from the window’s date (year=/month=/day=) so every window has its own immutable address and restore-by-date is a prefix list.

# WRONG: every run clobbers the same address.
# key = "rig_vibration/archive.csv"
# RIGHT: address is a pure function of the day.
key = f"rig_vibration/year={day:%Y}/month={day:%m}/day={day:%d}/part.parquet"

4. Export not verified before deletion. Symptom: an object exists but is truncated or empty; the source was already expired, so the data is unrecoverable. Root cause: the job advanced the watermark on a bare upload response without reading the object back. Fix: require count parity across object metadata, the serialized file, and a fresh source count before advancing the watermark; a failed check leaves the source protected for the next retry.

5. Idempotency lost on retry. Symptom: duplicate or fragmented objects accumulate after a crashed run. Root cause: keys keyed on run time or an appended sequence rather than on the window’s date. Fix: make the key a pure function of the window so a retry overwrites the same address, and enable object-store versioning so the prior copy is still recoverable.

# Verification and testing

Verification has two jobs: prove the archive is complete for every day inside the retention window, and prove the export job itself is still alive. First, a parity check that lists objects and compares their count to the days that should exist — run it on a schedule and alert on any shortfall. This CLI parity check compares the number of daily objects under a month’s prefix to the number of days that month should hold:

# Count archived objects for a month vs. days that should be present.
DAYS_IN_MONTH=31
OBJECTS=$(aws s3 ls s3://$ARCHIVE_BUCKET/rig_vibration/year=2026/month=03/ \
  --recursive | grep -c 'part.parquet')
echo "archived=$OBJECTS expected=$DAYS_IN_MONTH"
test "$OBJECTS" -eq "$DAYS_IN_MONTH" || echo "GAP: missing daily objects"

Second, put a monitor.deadman on the export task itself so a stalled job pages an operator while the source data still exists to re-archive. The watermark is written to archive_state on every successful run, so a deadman on that series fires precisely when archival stops advancing:

import "influxdata/influxdb/monitor"
import "experimental"

from(bucket: "archive_state")
  |> range(start: -6h)
  |> filter(fn: (r) => r._measurement == "archive_wm" and r._field == "watermark")
  |> monitor.deadman(t: experimental.subDuration(from: now(), d: 3h))
  |> filter(fn: (r) => r.dead == true)

Because the retention window is far longer than three hours, this alert gives an operator days of runway to fix a broken export before the boundary ever reaches un-archived data. Full task-management and query endpoints used by these checks are documented in the InfluxDB API reference. Finally, rehearse a restore quarterly: pull a random historical day back into a scratch bucket and confirm its row count matches the manifest — an archive is only proven by a successful re-ingest, never by the upload alone.

# Integration points

Cold-storage archival is the terminal stage of the lifecycle, so it depends on the tiers upstream of it. The hot, warm, and cold tiers whose boundaries decide what is even eligible for export are laid out in bucket architecture & tiering boundaries; the retention windows and watermark ordering that this job stays ahead of come from retention policy design, and the promotion of aged partitions into the cold tier is handled by storage tier migration automation before the export job ever sees them. Which tokens are permitted to read the cold bucket and write to the object store — and how they are scoped to the minimum needed — is the concern of data ingestion security frameworks, since an over-broad archival key is a standing exfiltration risk. When the object store or a tier is briefly unreachable mid-export, the run must fail safe rather than advance the watermark, which ties archival to the buffering and secondary-routing patterns in fallback routing & high availability. And the aggregates that let the cold tier hold a coarser, cheaper resolution in the first place are produced by downsampling & aggregation pipeline design. Together these turn retention from a lossy delete into a two-tier durability guarantee: cheap object storage behind an expensive live engine.

# FAQ

# Should I export line protocol, CSV, or Parquet?

Parquet for almost everything: it compresses columnar sensor data far better than text, carries its own schema, and is readable by any analytics engine years later without an InfluxDB running. Keep line protocol as an option when you need a byte-exact round-trip back into InfluxDB, and CSV only as a lowest-common-denominator fallback for a consumer that can read nothing else.

# How do I guarantee the object is durable before InfluxDB deletes the source?

Never let the retention sweep be the thing that authorizes deletion — let the watermark be. The job advances the watermark only after a parity check confirms the object exists and its row count matches the source, and the retention window is set longer than the archival lag so the sweep can never reach an un-advanced window. If verification fails, the watermark stays put and the still-longer retention keeps the source alive for the retry.

# What makes the export idempotent on retry?

The object key is a pure function of the window’s date, so re-running the same window overwrites the same address rather than creating a new file, and the watermark advances by exactly one verified day per run. A crashed run leaves the watermark where it was, so the next run reprocesses the same day and lands on the identical key. Enable versioning on the object-storage bucket so even an overwrite keeps the prior copy.

# How far behind real time should the archival job run?

Far enough that late-arriving data has settled but comfortably inside the retention window — typically a day or two behind the newest cold data, with the retention window several times that lag. Size it with $R_{cold} \ge L_{archive} + P_{period} + M_{margin}$: if the daily job can slip a day on a bad deploy, a 90-day cold window leaves enormous headroom.

# Can I query the archive without re-ingesting it?

Yes, if you keep it in Parquet with Hive-style date partitioning: most query engines can range-scan the object store directly by prefix. For anything that needs Flux, InfluxDB functions, or the exact original series semantics, re-ingest the relevant days into a scratch bucket instead — that keeps forensic work isolated from live retention.

# Related

• Retention Policy Design — sets the cold-tier expiry window that archival must stay ahead of.
• Storage Tier Migration Automation — promotes aged partitions into the cold tier this job exports.
• Exporting Cold Partitions to S3 Before Expiry — the day-by-day export mechanics in full.
• Python Client Orchestration Patterns — the connection, retry, and batching discipline behind the export and restore jobs.

Up one level: InfluxDB Data Lifecycle & Architecture Fundamentals