# Kubernetes CronJob Orchestration

There is a gap between what InfluxDB’s native task engine can do and what a full workflow engine is worth deploying. When a rollup outgrows a Flux task — it needs Python libraries, a call to an external inventory API, or a container image your platform team already ships — but does not yet need an in-cluster DAG scheduler with a database and a web UI, a Kubernetes CronJob is the right-sized tool. It gives you a declarative schedule, a container to run arbitrary code, automatic retries, and a manifest that lives in Git alongside the rest of your infrastructure. This page sits under External Orchestration & Workflow Engines and covers exactly how to run an InfluxDB maintenance job as a CronJob: how to build the job container, write a manifest whose concurrency and deadline settings actually prevent double-writes, mount the token from a Secret instead of baking it into the image, and verify the whole thing under GitOps.

# The failure scenario this solves

A cold-chain logistics operator runs a fleet of 4,200 refrigerated-container sensors — each reefer reports compartment temperature, door-open events, and compressor duty cycle every 20 seconds into a bucket called reefer_raw. A nightly Python job enriches the raw stream by joining each reading against a shipment-manifest API (to attach the SKU and destination that the sensor itself does not know), then writes hourly means into reefer_hourly. The job was too gnarly for a Flux task — it needed an HTTP client, retry logic, and a pandas transform — so an engineer ran it as a Kubernetes CronJob with schedule: "0 2 * * *" and nothing else specified.

For six weeks it worked. Then a control-plane upgrade took the API server offline for about forty minutes across the 02:00 UTC window on a Sunday. When the scheduler came back, two things happened at once. First, the missed 02:00 run never fired — Kubernetes saw the schedule time had passed and, with no startingDeadlineSeconds set, silently declined to start it, so Sunday’s enrichment simply never ran. Second, on the following night the manifest API was slow, the 02:00 job ran long, and because concurrencyPolicy defaults to Allow, the 03:00-triggered catch-up run started while the first was still writing — two jobs computed overlapping hourly windows and both called to(), double-writing the compressor-duty aggregates. The daily cold-chain compliance report showed a gap for one day and impossibly high duty-cycle averages for the next. The root cause was never the Python; it was a CronJob manifest that accepted every unsafe default. The rest of this page writes that manifest so neither failure can happen.

# Prerequisites

• A Kubernetes cluster 1.25+ (CronJob is stable/GA from 1.21; startingDeadlineSeconds and concurrencyPolicy behave as described here on 1.25+).
• InfluxDB 2.7+ (or InfluxDB 3.x Cloud/Dedicated) reachable from the Kubernetes cluster, with the target buckets already provisioned.
• Python 3.11+ and influxdb-client 1.36+ baked into the job image, plus requests for the manifest API call.
• A container registry your cluster can pull from, and permission to create CronJob, Job, and Secret objects in the target namespace.
• An InfluxDB token scoped to read on reefer_raw and write on reefer_hourly only — never an all-access token. Least-privilege scoping is covered in data ingestion security frameworks.
• A GitOps controller (Flux or Argo CD) reconciling the namespace, so the manifest is the single source of truth.
• Timestamps handled in UTC end to end so the schedule field and the query range agree; see the RFC 3339 timestamp specification.

# Core concept: the CronJob → Job → Pod chain

A CronJob is not itself the thing that runs your code. It is a controller that, on each schedule tick, stamps out a new Job object from its jobTemplate. The Job controller in turn creates one or more Pods, and a Pod runs your container. Each layer owns a distinct piece of the reliability contract, and the settings that prevent the failure above live at different layers: schedule, concurrencyPolicy, and startingDeadlineSeconds are CronJob-level; backoffLimit and activeDeadlineSeconds are Job-level; restartPolicy and the Secret mount are Pod-level. Understanding which layer owns which knob is what keeps you from setting restartPolicy where a backoffLimit was needed.

The scheduling model is deliberately simple: a CronJob is a level-triggered controller, not an event queue. It compares the current time against the schedule and decides whether a run is due, then checks whether it is allowed to start (concurrency) and still worth starting (deadline). Because it is level-triggered, a run that is skipped is gone — there is no backlog it drains later. That single property is why the missing-run failure is a configuration choice, not a bug. If you think of the CronJob as issuing at most one Job per tick, the number of live Jobs a schedule produces over a window is bounded by:

$$N_{jobs} = \left\lceil \frac{T_{window}}{P_{schedule}} \right\rceil \quad\text{when}\quad concurrencyPolicy = \texttt{Allow}$$

With concurrencyPolicy: Forbid that count collapses toward one active Job regardless of how long a run takes, which is exactly the invariant an idempotent rollup needs.

# Step-by-step implementation

# 1. Write the idempotent job container

The container runs one bounded rollup and exits. Two disciplines make it safe to retry: it derives its read window from wall-clock time floored to the hour (so a retry recomputes the same window rather than a drifting one), and it writes with a deterministic aggregate that overwrites rather than appends. In InfluxDB, a point is uniquely identified by measurement, tag set, and timestamp — so writing the same hourly-mean point twice overwrites it in place. That is what makes an overlapping run at worst wasteful, not corrupting, and it is the property the manifest then reinforces with Forbid.

import os
import sys
from datetime import datetime, timedelta, timezone

import requests
from influxdb_client import InfluxDBClient, Point, WritePrecision
from influxdb_client.client.write_api import SYNCHRONOUS

def previous_hour_bounds() -> tuple[datetime, datetime]:
    now = datetime.now(timezone.utc).replace(minute=0, second=0, microsecond=0)
    return now - timedelta(hours=1), now

def enrich(sku_by_reefer: dict[str, str], reefer_id: str) -> str:
    # Manifest lookup attaches shipment SKU the sensor cannot know.
    return sku_by_reefer.get(reefer_id, "unassigned")

def main() -> int:
    start, stop = previous_hour_bounds()
    client = InfluxDBClient(
        url=os.environ["INFLUX_URL"],
        token=os.environ["INFLUX_TOKEN"],   # injected from a Secret, see step 4
        org=os.environ["INFLUX_ORG"],
    )
    manifest = requests.get(os.environ["MANIFEST_API"], timeout=10).json()

    flux = f'''
        from(bucket: "reefer_raw")
          |> range(start: {start.isoformat()}, stop: {stop.isoformat()})
          |> filter(fn: (r) => r._measurement == "reefer_metrics")
          |> filter(fn: (r) => r._field == "compartment_c" or r._field == "compressor_duty")
          |> aggregateWindow(every: 1h, fn: mean, createEmpty: false)
    '''
    write_api = client.write_api(write_options=SYNCHRONOUS)
    points = []
    for table in client.query_api().query(flux, org=os.environ["INFLUX_ORG"]):
        for rec in table.records:
            reefer = rec.values["reefer_id"]
            points.append(
                Point("reefer_metrics_hourly")
                .tag("reefer_id", reefer)
                .tag("sku", enrich(manifest, reefer))
                .field(rec.get_field(), rec.get_value())
                .time(start, WritePrecision.S)   # fixed window bound = deterministic ID
            )
    write_api.write(bucket="reefer_hourly", record=points)
    client.close()
    print(f"wrote {len(points)} hourly points for {start.isoformat()}")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Pinning .time(start, ...) to the floored hour is the crux: every run and every retry that targets the same window produces the same series identity, so to-equivalent writes are upserts. This is the same idempotency contract that native Flux rollups get from tasks.lastSuccess(), applied by hand in Python — the parallel is discussed under Python client orchestration patterns.

# 2. Write the CronJob manifest with safe defaults

Every field here overrides a default that would otherwise reproduce the opening failure. concurrencyPolicy: Forbid stops an overrunning job from being lapped by the next tick. startingDeadlineSeconds tells the controller how late a missed run may still start, so a short control-plane outage does not silently drop a run. backoffLimit and activeDeadlineSeconds bound retries and total runtime so a wedged job cannot run forever.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: reefer-hourly-rollup
  namespace: telemetry
spec:
  schedule: "0 2 * * *"          # 02:00 in .spec.timeZone below, NOT node-local time
  timeZone: "Etc/UTC"            # pin the zone explicitly (k8s 1.27+)
  concurrencyPolicy: Forbid      # never run two rollups at once
  startingDeadlineSeconds: 900   # a run up to 15 min late still fires
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2            # retry a failed Pod at most twice
      activeDeadlineSeconds: 1200  # kill a Job that runs past 20 min
      template:
        spec:
          restartPolicy: Never   # let the Job controller own retries, not the kubelet
          containers:
            - name: rollup
              image: registry.internal/reefer-rollup:1.4.0
              env:
                - name: INFLUX_URL
                  value: "https://influx.telemetry.svc:8086"
                - name: INFLUX_ORG
                  value: "cold-chain"
                - name: MANIFEST_API
                  value: "http://manifest.logistics.svc/v1/skus"

restartPolicy: Never paired with backoffLimit is deliberate: if you instead set restartPolicy: OnFailure, the kubelet restarts the container in place and backoffLimit no longer governs the retry count the way you expect. Keep retries at the Job layer. The schedule/timeZone pairing is examined below and is a frequent source of “it ran an hour early” incidents; the general theory of aligning cron expressions to a zone is covered in cron & interval scheduling logic.

# 3. Bound retries and total runtime deliberately

The two limits interact. backoffLimit: 2 means the Job tolerates three Pod attempts (initial + two retries) before it is marked Failed; activeDeadlineSeconds: 1200 caps the wall-clock life of the Job across all attempts. Set the deadline to comfortably more than one healthy run but less than the schedule period, so a hung run is reaped well before the next tick is due. For the nightly reefer rollup — which finishes in about ninety seconds when the manifest API is healthy — twenty minutes is generous headroom that still guarantees the Job is gone long before 02:00 the next day.

# jobTemplate.spec excerpt — retry and lifetime bounds
backoffLimit: 2              # 3 attempts total, then Job = Failed
activeDeadlineSeconds: 1200  # hard ceiling across all attempts
ttlSecondsAfterFinished: 3600  # garbage-collect the Job object 1h after it ends

ttlSecondsAfterFinished keeps finished Job and Pod objects from accumulating in the namespace, which otherwise slowly clutters kubectl get jobs and the GitOps controller’s view. The retry count itself should match the failure profile of your dependencies: a job that calls a flaky external API benefits from a small backoff, but for write-heavy jobs prefer application-level exponential backoff and retry for InfluxDB client writes over relying on whole-Pod restarts, which repeat the entire read as well.

# 4. Mount the token from a Secret, never the image or plaintext env

The token must not live in the image, in the manifest’s plaintext value:, or in your Git history. Create it as a Kubernetes Secret and inject it with valueFrom.secretKeyRef so it arrives in the container’s environment at runtime only. Under GitOps, store the Secret encrypted (SOPS or Sealed Secrets) so the encrypted form is safe to commit while the plaintext never touches the repository.

apiVersion: v1
kind: Secret
metadata:
  name: influx-token
  namespace: telemetry
type: Opaque
stringData:
  token: "REPLACE_VIA_SOPS_OR_SEALED_SECRETS"
---
# add to the container spec from step 2:
env:
  - name: INFLUX_TOKEN
    valueFrom:
      secretKeyRef:
        name: influx-token
        key: token

Scope that token to read on reefer_raw and write on reefer_hourly and nothing else, and rotate it on a schedule so a leaked token has a short blast radius. A read-write token scoped to two buckets that leaks is an incident; an all-access operator token that leaks is a breach.

# Configuration reference

# Common failure modes and fixes

1. concurrencyPolicy default Allow overlaps runs and double-writes. Symptom: aggregate values spike impossibly (sums doubled, means skewed) whenever a run is slow. Root cause: an overrunning Job is still writing when the next tick stamps a second Job, and both call the InfluxDB write path over the same window. Fix: set concurrencyPolicy: Forbid so the controller skips a new run while one is active, and keep the job idempotent so even a Replace transition cannot corrupt.

spec:
  concurrencyPolicy: Forbid   # the single most important line in the manifest

2. Missed schedules after control-plane downtime with no startingDeadlineSeconds. Symptom: a run silently never happens after a Kubernetes control-plane upgrade or API-server outage that spanned the schedule time. Root cause: the CronJob controller counts missed start times; with no deadline it will not start a run whose slot has passed, and if more than 100 starts are missed it stops scheduling entirely and logs Cannot determine if job needs to be started. Fix: set startingDeadlineSeconds to a value shorter than your schedule period but long enough to cover a routine outage, so a late run still fires.

spec:
  startingDeadlineSeconds: 900   # a run up to 15 minutes late still runs

3. Timezone assumptions in spec.schedule. Symptom: the rollup fires an hour early or late, or shifts twice a year at DST boundaries. Root cause: without spec.timeZone the schedule is evaluated in the control plane’s local zone, which is undefined across managed clusters and can differ from the UTC your query ranges assume. Fix: set spec.timeZone: "Etc/UTC" and write all query bounds in UTC so schedule and range never disagree. Never encode a DST-observing zone unless you genuinely want the job to shift with local clocks.

4. Token in env plaintext or baked into the image. Symptom: an InfluxDB token appears in kubectl describe cronjob, in the image layers, or in Git history. Root cause: the token was set as a literal value: or ENV in the Dockerfile instead of referenced from a Secret. Fix: move it to a Kubernetes Secret referenced via secretKeyRef, encrypt the Secret for GitOps, and rotate the exposed token immediately.

5. restartPolicy: OnFailure defeats backoffLimit expectations. Symptom: a failing job retries far more often than backoffLimit suggests, hammering a downstream API. Root cause: with OnFailure the kubelet restarts the container in place on the same Pod, and the interaction with backoffLimit is not the clean “N attempts then stop” you assumed. Fix: use restartPolicy: Never so each attempt is a fresh Pod counted cleanly against backoffLimit.

# Verification and testing

Verify at three levels: that the CronJob exists with the schedule and policies you intend, that its recent Jobs actually succeeded, and that the data it should be producing is in fact arriving. Start with the object itself:

kubectl get cronjob reefer-hourly-rollup -n telemetry \
  -o custom-columns=NAME:.metadata.name,SCHEDULE:.spec.schedule,\
CONCURRENCY:.spec.concurrencyPolicy,SUSPEND:.spec.suspend,LAST:.status.lastScheduleTime

# Inspect the Jobs it has stamped and whether they completed.
kubectl get jobs -n telemetry -l job-name --sort-by=.metadata.creationTimestamp
kubectl logs -n telemetry job/reefer-hourly-rollup-28912345

Trigger an out-of-band run to test without waiting for 02:00, then confirm it completed exactly once:

kubectl create job --from=cronjob/reefer-hourly-rollup manual-test -n telemetry
kubectl wait --for=condition=complete job/manual-test -n telemetry --timeout=300s

Confirm the aggregate landed and, critically, that a deadman check will page if the CronJob stops producing data — a silently suspended CronJob or a wedged control plane is invisible until you assert on the absence of output. This Flux health check alerts when no hourly reefer point has arrived in the last three hours, catching a stalled schedule while there is still raw data to reprocess:

import "influxdata/influxdb/monitor"
import "experimental"

from(bucket: "reefer_hourly")
  |> range(start: -3h)
  |> filter(fn: (r) => r._measurement == "reefer_metrics_hourly")
  |> monitor.deadman(t: experimental.subDuration(from: now(), d: 2h))
  |> filter(fn: (r) => r.dead == true)

Under GitOps, verify Flux has reconciled the manifest and is not stuck on a drift or a decryption error, so what is running matches what is in Git:

flux get kustomization telemetry --watch
flux reconcile kustomization telemetry --with-source

If flux get reports the kustomization as Ready: False, the CronJob you are inspecting may be a stale version — reconcile before trusting kubectl. A green reconcile plus a passing deadman check together mean the schedule, the manifest, and the data are all in agreement.

# Integration points

A CronJob is one point on a spectrum, and knowing when to reach past it matters as much as configuring it. The decision of whether this middle ground is even the right layer — versus a native task or a full engine — is the whole subject of the parent External Orchestration & Workflow Engines section, and the head-to-head with staying in-database is drawn out in Kubernetes CronJob vs native InfluxDB tasks. The scheduling semantics themselves — how a cron expression maps to fire times and why a zone must be pinned — belong to cron & interval scheduling logic, and the broader question of what to schedule and in what order sits under automated task scheduling & orchestration. The Python inside the container follows the same connection, batching, and retry disciplines as any other client job in Python client orchestration patterns, and the token it mounts must be scoped and rotated per data ingestion security frameworks. InfluxDB’s own client-write API used by the job is documented in the InfluxDB write API reference.

# FAQ

# When should I use a Kubernetes CronJob instead of a native InfluxDB task?

Reach for a CronJob when the job needs something a Flux task cannot give it — arbitrary Python libraries, a call to an external API, a container image your platform already ships, or CPU isolation from the database. Stay with a native task when the work is pure Flux against InfluxDB, because a task runs in-process with no image to build or pod to schedule. The CronJob is the middle ground: more than a task, far less than a workflow engine.

# Why did my CronJob stop running entirely with no error?

Two common causes. Either it was suspend: true (check kubectl get cronjob), or the controller missed more than 100 scheduled starts — which happens when startingDeadlineSeconds is unset and the control plane was down long enough — after which it refuses to schedule and logs that it cannot determine if the job needs to start. Setting a sensible startingDeadlineSeconds prevents the second case.

# Does concurrencyPolicy: Forbid guarantee I never double-write?

It guarantees the CronJob controller will not start a second run while one is active, which removes the most common overlap. It does not protect against a manual kubectl create job --from fired during a run, or a Replace transition. Belt and braces: keep the job idempotent by pinning each written point’s timestamp to a fixed window bound so any repeat write is an in-place overwrite rather than a duplicate.

# How do I run the job on demand to test it without touching the schedule?

Use kubectl create job --from=cronjob/<name> <test-name> to stamp a one-off Job from the same template, then kubectl wait --for=condition=complete and read its logs. This exercises the exact image, env, and Secret mount the scheduled runs use, so a green manual run is strong evidence the 02:00 run will also succeed.

# Should the InfluxDB token be a Secret or can I use an env value for a dev cluster?

Always a Secret referenced via secretKeyRef, even in dev. A plaintext value: lands in kubectl describe, in the CronJob’s stored object, and — under GitOps — in Git history, all of which outlive the dev cluster. Encrypt the Secret with SOPS or Sealed Secrets so the committed form is safe and the plaintext token never leaves the moment of decryption inside the pod.

# Related

• Kubernetes CronJob vs native InfluxDB tasks — the decision between scheduling in-database and scheduling on the Kubernetes cluster, with the trade-offs spelled out.
• External Orchestration & Workflow Engines — the full spectrum from native tasks to CronJobs to Airflow, and how to choose a layer.
• Cron & Interval Scheduling Logic — how cron expressions map to fire times and why the zone must be pinned.
• Python Client Orchestration Patterns — the connection, batching, and retry disciplines the job container should follow.
• Data Ingestion Security Frameworks — scoping and rotating the token the CronJob mounts from a Secret.

Up one level: External Orchestration & Workflow Engines