Task Automation

Deadman checks for detecting silent IoT sensors

A remote pipeline pressure transducer that fails does not throw an error — it simply stops writing points, and every threshold rule you have goes quiet with it because there is no value left to breach. On a field of cellular-backhauled transducers scattered across a gas pipeline, a modem that browns out, a SIM that de-provisions, or a firmware watchdog that hangs all present identically: the last good reading sits at 812 psi, the dashboard shows a flat line, and nobody notices until a downstream integrity calculation runs on stale data hours later. The only signal a dead sensor emits is absence, and absence is exactly what a value-threshold check cannot see. This page builds a monitor.deadman check that alerts on silence per device group, derives the silence window from each group’s real reporting interval so it neither cries wolf nor sleeps through an outage, and suppresses the alarm during planned maintenance so a scheduled valve service does not page the on-call engineer. It is the absence-detection half of Anomaly Detection & Alerting; its value-breach counterpart is covered separately.

Prerequisites

How long to wait before declaring a sensor dead

The one parameter that makes or breaks a deadman check is the silence window t — the age past which a group with no fresh point is declared dead. Set it too tight and normal cellular jitter (a retransmit, a tower handover, a batched flush) trips a false alarm every few minutes; set it too loose and a genuinely dead transducer stays green for half an hour. Derive it from the group’s real cadence rather than guessing:

$$ t_{silence} = k \cdot I_{report} + m_{margin} $$

where $I_{report}$ is the nominal reporting interval (60 s here), $k$ is how many consecutive missed reports you tolerate before caring, and $m_{margin}$ absorbs backhaul jitter. For a 60 s cadence, tolerating three missed reports with a 60 s margin gives a 4-minute window — long enough to ride out a modem retransmit, short enough that a truly silent transducer is flagged within one task cycle. Groups on slower cadences (a daily-batched GPS asset, say) need a proportionally larger window, which is why the check keys off site_id/transducer_id groups rather than one fleet-wide threshold. The same “match the window to the cadence” discipline governs task timing itself; see cron & interval scheduling logic for aligning the task’s own every to the data it watches.

Deriving the deadman silence window from the reporting interval A transducer reports every 60 seconds until it goes silent. A grace band covering three missed reports plus a jitter margin ends at the deadman threshold at now minus four minutes. Absence beyond the threshold is declared dead; a shorter gap inside the band is tolerated. reporting every 60 s · healthy grace band = 3 × 60 s + 60 s margin missed reports tolerated as jitter deadman threshold t now − 4 min no report → DEAD crit status written last healthy report now silence begins

Solution walkthrough

Step 1 — Group the stream and mark dead groups with monitor.deadman

monitor.deadman(t) takes a stream that is already grouped by device identity and appends a boolean dead column: true for any group whose newest point is older than the threshold time t, false otherwise. Build t from now() with experimental.subDuration so the window slides with each task run rather than pinning to a fixed clock time.

flux
import "influxdata/influxdb/monitor"
import "experimental"

from(bucket: "pipeline-telemetry")
    |> range(start: -10m)
    |> filter(fn: (r) => r._measurement == "pressure" and r._field == "psi")
    |> group(columns: ["site_id", "transducer_id"])
    |> monitor.deadman(t: experimental.subDuration(d: 4m, from: now()))

Two parameters carry the design. The range(start: -10m) must be wider than the silence window t — it defines the set of groups the check can see at all, and it must include the last good point of a sensor that went quiet 4 minutes ago, otherwise the group vanishes from the stream before it can be flagged (the central gotcha below). The group(columns: ["site_id", "transducer_id"]) fixes the granularity of “a device”: each transducer is judged on its own last-report time, so a healthy neighbour on the same site cannot mask a dead unit. Note that monitor.deadman reads the newest timestamp per group, not per field — filtering to a single _field before grouping keeps the judgement unambiguous, because a group that carries several fields with different last-write times would otherwise be judged on whichever field happened to arrive most recently.

Step 2 — Turn dead flags into statuses with monitor.check

monitor.deadman only annotates rows; monitor.check is what evaluates them into crit/warn/ok levels and persists a status to _monitoring with a stable check identity, so downstream consumers and history queries have something durable to read. Wrap the whole thing in a task so it runs on a cadence just under the silence window.

flux
import "influxdata/influxdb/monitor"
import "experimental"

option task = {name: "deadman-pipeline-pressure", every: 2m, offset: 15s}

check = {
    _check_id: "0b1deadman0001",
    _check_name: "pipeline-pressure-deadman",
    _type: "deadman",
    tags: {fleet: "pipeline-pressure"},
}

messageFn = (r) =>
    "Transducer ${r.transducer_id} at site ${r.site_id} is silent (dead=${string(v: r.dead)})."

from(bucket: "pipeline-telemetry")
    |> range(start: -10m)
    |> filter(fn: (r) => r._measurement == "pressure" and r._field == "psi")
    |> group(columns: ["site_id", "transducer_id"])
    |> monitor.deadman(t: experimental.subDuration(d: 4m, from: now()))
    |> monitor.check(
        data: check,
        messageFn: messageFn,
        crit: (r) => r.dead == true,
        ok: (r) => r.dead == false,
    )

The task’s every: 2m is deliberately half the 4-minute window so the check re-evaluates at least twice within one silence period — a single missed task run therefore cannot let a dead sensor slip through undetected. The offset: 15s gives a straggling cellular batch a moment to land before the window is judged. monitor.check writes one status row per group per run into _monitoring; those crit rows are the durable event a notification layer consumes. Wiring those statuses to an outbound webhook — with idempotent de-duplication so one outage does not page ten times — is the job of a Python hook, exactly as in threshold-based alerting with Flux & Python hooks.

Step 3 — Suppress alarms during planned maintenance

A scheduled valve service takes a site offline for an hour; without suppression, every transducer there goes “dead” and floods the on-call channel with alarms that are entirely expected. Keep a small fleet-maintenance bucket holding an in_maintenance flag per site_id for active windows, then anti-join it against the check stream so maintenance sites never reach monitor.check.

flux
import "influxdata/influxdb/monitor"
import "experimental"
import "join"

option task = {name: "deadman-pipeline-pressure", every: 2m, offset: 15s}

// Sites currently under a planned service window.
maint =
    from(bucket: "fleet-maintenance")
        |> range(start: -1h)
        |> filter(fn: (r) => r._measurement == "maintenance" and r._value == true)
        |> keep(columns: ["site_id"])
        |> group()
        |> distinct(column: "site_id")

live =
    from(bucket: "pipeline-telemetry")
        |> range(start: -10m)
        |> filter(fn: (r) => r._measurement == "pressure" and r._field == "psi")

// Drop any point whose site is in maintenance, then run the deadman check.
join.left(
    left: live,
    right: maint,
    on: (l, r) => l.site_id == r._value,
    as: (l, r) => ({l with _under_maint: exists r._value}),
)
    |> filter(fn: (r) => not r._under_maint)
    |> group(columns: ["site_id", "transducer_id"])
    |> monitor.deadman(t: experimental.subDuration(d: 4m, from: now()))
    |> monitor.check(
        data: {_check_id: "0b1deadman0001", _check_name: "pipeline-pressure-deadman", _type: "deadman", tags: {fleet: "pipeline-pressure"}},
        messageFn: (r) => "Transducer ${r.transducer_id} at ${r.site_id} silent.",
        crit: (r) => r.dead == true,
        ok: (r) => r.dead == false,
    )

The maintenance window is read from a bucket rather than hard-coded so the operations team can open and close service windows without redeploying the task. When a window closes, its rows drop out of the -1h range and the affected transducers automatically resume being watched — no manual re-arm. Keep the maintenance flag’s own retention short; it is control state, not telemetry. One caution on the suppression range: if a service window can exceed the -1h lookback, widen the maintenance query to span the longest window you allow, or the flag will age out mid-service and the transducers will alarm before the crew is finished. Because suppression drops the whole site, the maintenance record should be scoped to the exact site_id under service — never a broad wildcard — so a bug in the maintenance feed cannot silently blind the entire fleet.

Gotchas and edge cases

A sensor that is silent longer than the query range disappears entirely. monitor.deadman can only flag groups that still have at least one point inside range(start: ...). If a transducer has been dead for 20 minutes but the range is only -10m, that group has zero rows in the window, so it is never in the stream and never flagged — the check goes quiet precisely when the outage is worst. Always keep the range start comfortably larger than the silence window t (here 10 m against a 4 m window), and for sensors that can vanish for long stretches, back the deadman check with a roster: periodically materialise the expected device list and alert on any transducer_id in the roster that produced no status at all this cycle.

Brand-new or intermittently commissioned devices trip a spurious “dead” on first sight. A transducer installed an hour ago, or one that legitimately reports only during a daily pump cycle, looks identical to a failed unit inside a fixed window. Gate the check on an “expected to be live” set — a tag, a commissioning bucket, or a schedule annotation — so a device is only judged dead while it is supposed to be reporting. Do not lower t globally to accommodate them; that reintroduces false alarms on the fast-cadence majority.

One fleet-wide silence window misjudges every device with a different cadence. A 4-minute window is correct for the 60 s transducers but declares a legitimately 30-minute-batched asset dead 26 minutes early. When device classes share a bucket, either run one deadman task per class with its own t, or carry the nominal interval as a tag and compute t per group. Mixing cadences under a single threshold guarantees either false positives on the slow devices or slow detection on the fast ones.

Verification

Read back the statuses the check wrote and confirm dead groups are being recorded as crit. Query the _monitoring bucket for the most recent level per device:

flux
import "influxdata/influxdb/schema"

from(bucket: "_monitoring")
    |> range(start: -15m)
    |> filter(fn: (r) => r._check_name == "pipeline-pressure-deadman")
    |> filter(fn: (r) => r._field == "_level")
    |> group(columns: ["site_id", "transducer_id"])
    |> last()
    |> filter(fn: (r) => r._value == "crit")
    |> yield(name: "currently_silent")

To prove the detection end to end without waiting for a real outage, stop writing for one test transducer and confirm it surfaces here within two task cycles; a healthy fleet returns an empty table, and any row is a device that has genuinely gone silent past its window. Cross-check the raw side too — a matching gap in pipeline-telemetry for that transducer_id confirms the alert is real absence, not a mislabelled group.

FAQ

Why not just alert when pressure crosses a threshold?

A threshold check needs a value to evaluate, and a dead sensor stops producing values entirely, so the rule simply never fires — the last reading stays frozen at whatever it was and looks superficially valid. Deadman detection watches the time of the newest point instead of its magnitude, which is the only signal an absent sensor leaves. Run both: thresholds catch bad readings, deadman catches missing ones.

How do I stop one outage from paging the team repeatedly?

monitor.check writes a fresh crit status every task run while a device stays dead, so the de-duplication belongs in the notification layer, not the check. Have the Python hook that consumes statuses key notifications on a stable identity such as site_id/transducer_id plus the check id, and only emit when the level transitions into crit (or after a cooldown), suppressing the repeats in between. The idempotent-hook pattern in the threshold-alerting page applies unchanged here.

Can one deadman task cover sensors with very different reporting rates?

Only if you give each cadence its own silence window. A single t that suits 60-second transducers will falsely flag a device that legitimately reports every 30 minutes. Either split into one task per device class with a window sized to that class, or tag each device with its nominal interval and compute t per group so every device is judged against its own expected cadence.

Up: Anomaly Detection & Alerting

Related pages