# Anomaly Detection & Alerting

A pressure sensor rarely fails loudly. It drifts a few kilopascals past its safe ceiling for ninety seconds, or it goes silent while the dashboard keeps showing its last green reading, and the first real signal reaches an operator only when a downstream valve trips. Detecting these conditions on time-series data is not a dashboard problem — a human is not watching at 03:00 — it is a scheduling problem: a task must query a fixed window on a cadence, decide whether the data crossed a level or stopped arriving, record that decision durably, and hand it to something that can page a human. This page sits under automated task scheduling & orchestration and covers how to build that loop on InfluxDB with three kinds of scheduled Flux check — threshold, deadman, and rate-of-change/z-score — each writing status rows to a monitoring bucket, and a Python notification hook that reads those statuses and fans out to a webhook and email without paging twice for the same event.

# The failure scenario this solves

A municipal water-treatment operator ran a fleet of 340 distribution pumps, each writing discharge pressure (pressure_kpa) and motor current at 5-second resolution into a bucket named plant_telemetry. Alerting was a single Grafana panel with a red threshold line at 620 kPa. It worked until it didn’t.

One night pump WP-118 developed a failing check valve. Its discharge pressure oscillated across the 620 kPa line every eight to twelve seconds — over the line, under it, over it again — for forty minutes. The Grafana alert, evaluated on each raw sample, fired and cleared 214 times, generating 214 SMS pages to the on-call engineer, who silenced the entire notification channel at page 30 to stop the noise. Two hours later, with the channel still muted, a different pump — WP-263 — lost its telemetry link entirely. It reported nothing at all. Because the dashboard only draws lines for data that exists, a pump that sends no data draws no line and crosses no threshold, so it raised no alert. WP-263 ran dry and burned out a seal before the morning shift noticed the flat panel. Two failures, one root cause: alerting was tied to a live dashboard and to raw samples, with no hysteresis to survive a flapping signal and no concept of absence to catch a silent sensor.

The rebuild moved evaluation off the dashboard and into scheduled tasks. A threshold check runs every minute against a one-minute mean with a dead band, so WP-118’s oscillation produces one crit status and one recovery instead of 214 pages. A deadman check runs every minute and flags any pump that has not reported inside a 3-minute window, so WP-263’s silence becomes an alert within three minutes rather than eight hours. Both write status rows to _monitoring, and a single idempotent Python hook turns distinct status transitions — not raw crossings — into pages. This page builds that system end to end.

# Prerequisites

• InfluxDB 2.7+ (or InfluxDB 3.x Cloud/Dedicated) with the native task engine enabled.
• Flux with the influxdata/influxdb/monitor package (bundled) for monitor.check and monitor.deadman.
• A source bucket (plant_telemetry) receiving sensor writes, and the system _monitoring bucket that ships with InfluxDB 2.x (statuses land in its statuses measurement).
• Python 3.9+ with influxdb-client 1.36+ and requests 2.31+ for the notification hook.
• An operator token for creating tasks, plus a read-only token scoped to _monitoring for the notification hook — never give the hook write access to telemetry.
• A webhook endpoint (chat, PagerDuty Events API, or an internal receiver) and an SMTP relay or transactional email API for the email fan-out.
• Timestamps written in RFC 3339 / UTC so check windows and reporting intervals are unambiguous; see the RFC 3339 timestamp specification.

# Core concept: a check is a scheduled query that writes a status

A dashboard evaluates a threshold in the browser, at read time, only while someone is looking. A check inverts that: it is a scheduled task that queries a bounded window, assigns each series a discrete level (ok, info, warn, crit), and writes that level as a row into the _monitoring bucket. The status becomes data — queryable, retained, and independent of any human being awake. Notification is then a second, decoupled stage that reads statuses and delivers them. Separating detection from delivery is what lets you page on a transition (ok → crit) rather than on every sample that happens to be over the line.

Three check shapes cover most operational needs. A threshold check compares an aggregated value to fixed levels — pressure over 620 kPa is crit. A deadman check flags the absence of data — a series that wrote nothing inside the window is dead, the only check that fires on silence. A statistical check compares each point to the recent distribution rather than a fixed line, catching drift that never trips an absolute threshold. For the statistical case, standardise each reading against the trailing mean and standard deviation as a z-score:

$$z = \frac{x - \mu}{\sigma}$$

and alert when a point falls outside a symmetric band around the trailing mean:

$$\mu - k\sigma ; \le ; x ; \le ; \mu + k\sigma$$

A point with $|z| > k$ is anomalous. Choosing $k = 3$ flags roughly the outer 0.3% of a normal distribution; $k = 4$ is quieter and better suited to noisy pump current. The band is adaptive — it widens when the fleet is genuinely noisy and tightens when it settles — so it catches a pump drifting away from its own baseline long before that drift reaches an absolute red line.

# Step-by-step implementation

# 1. A scheduled threshold check with a dead band

The threshold check runs every minute, reduces each pump’s raw 5-second stream to a one-minute mean, and assigns a level with monitor.check. The dead band is the whole point: the crit predicate fires above 620 kPa but the value must fall back below a lower recovery line (600 kPa) before the series returns to ok. That gap is hysteresis — it is what turns WP-118’s 214 crossings into one alert. The check data record is required metadata; monitor.check uses it to stamp each status row so downstream consumers can group by check. The Flux discipline behind this — retry safety and side-effect ordering — is developed across the platform’s task-scripting guidance.

import "influxdata/influxdb/monitor"

option task = {name: "check_pump_pressure_threshold", every: 1m, offset: 15s}

check = {
    _check_id: "pump-pressure-0001",
    _check_name: "Pump discharge pressure threshold",
    _type: "threshold",
    tags: {plant: "north", asset: "distribution-pump"},
}

// Levels. The warn/crit ceilings and the ok floor form a dead band:
// a pump must fall back under 600 kPa before it clears, not just under 620.
crit = (r) => r.pressure_kpa > 620.0
warn = (r) => r.pressure_kpa > 590.0 and r.pressure_kpa <= 620.0
ok = (r) => r.pressure_kpa <= 600.0

messageFn = (r) =>
    "${r._level}: pump ${r.pump_id} discharge ${string(v: r.pressure_kpa)} kPa"

from(bucket: "plant_telemetry")
    |> range(start: -task.every)
    |> filter(fn: (r) => r._measurement == "pump_pressure")
    |> filter(fn: (r) => r._field == "pressure_kpa")
    |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)
    |> group(columns: ["pump_id"])
    |> monitor.check(
        data: check,
        messageFn: messageFn,
        crit: crit,
        warn: warn,
        ok: ok,
    )

monitor.check writes one row per series into _monitoring’s statuses measurement, tagged with _check_id, _level, and the rendered _message. It evaluates predicates in priority order (crit, then warn, then info, then ok), so overlapping ranges resolve to the most severe matching level. The offset: 15s lets the last few seconds of late samples settle before the window is read — align this with your interval-scheduling configuration so the check cadence never drifts against the write cadence.

# 2. A deadman check for silent sensors

A threshold check is blind to WP-263, because absent data crosses no line. monitor.deadman is the only check that fires on silence: it marks a series dead: true when its most recent point is older than a cutoff. Here the cutoff is 3 minutes back from now(), comfortably longer than the 5-second reporting interval but short enough to catch a link failure fast. The result feeds monitor.check with a crit predicate on the dead column so a silent pump lands in the same statuses stream as a threshold breach. The narrow-focus walkthrough is in deadman checks for detecting silent IoT sensors.

import "influxdata/influxdb/monitor"
import "experimental"

option task = {name: "check_pump_deadman", every: 1m, offset: 15s}

check = {
    _check_id: "pump-deadman-0002",
    _check_name: "Pump telemetry deadman",
    _type: "deadman",
    tags: {plant: "north", asset: "distribution-pump"},
}

messageFn = (r) => "${r._level}: pump ${r.pump_id} has stopped reporting"

from(bucket: "plant_telemetry")
    |> range(start: -10m)
    |> filter(fn: (r) => r._measurement == "pump_pressure")
    |> filter(fn: (r) => r._field == "pressure_kpa")
    |> group(columns: ["pump_id"])
    // dead == true when the newest point is older than 3 minutes ago
    |> monitor.deadman(t: experimental.subDuration(d: 3m, from: now()))
    |> monitor.check(
        data: check,
        messageFn: messageFn,
        crit: (r) => r.dead,
        ok: (r) => not r.dead,
    )

Two parameters carry the correctness. The range(start: -10m) must be wider than the deadman cutoff t, otherwise a pump silent for longer than the range disappears from the pipe entirely and can never be evaluated as dead — a silent sensor with no rows is not the same as a row flagged dead. And group(columns: ["pump_id"]) ensures the deadman is evaluated per pump: without it, one still-reporting pump keeps the whole table alive and masks 339 silent ones.

# 3. A Python notification hook that fans out idempotently

Detection has written statuses; now deliver them. This hook — built with the same Python client orchestration approach used elsewhere on the platform — polls _monitoring for status rows at crit/warn since its last run, then posts each to a webhook and email. The critical detail is idempotency: it builds a stable dedup_key from the check id, series, level, and minute-bucketed time, and skips any key it has already delivered. Without that key a hook that overlaps its own previous run, or reprocesses a window, pages twice for one event.

import os
import time
import hashlib
import smtplib
from email.message import EmailMessage

import requests
from influxdb_client import InfluxDBClient

SEEN: set[str] = set()  # persist to Redis/SQLite in production

def dedup_key(check_id: str, pump: str, level: str, ts: str) -> str:
    minute = ts[:16]  # bucket to the minute so retries collapse
    raw = f"{check_id}|{pump}|{level}|{minute}"
    return hashlib.sha256(raw.encode()).hexdigest()[:16]

def read_new_statuses(query_api, since: str = "-2m"):
    flux = f'''
        from(bucket: "_monitoring")
          |> range(start: {since})
          |> filter(fn: (r) => r._measurement == "statuses")
          |> filter(fn: (r) => r._field == "_message")
          |> filter(fn: (r) => r._level == "crit" or r._level == "warn")
    '''
    for table in query_api.query(flux, org=os.environ["INFLUX_ORG"]):
        for rec in table.records:
            yield {
                "check_id": rec.values.get("_check_id", ""),
                "pump": rec.values.get("pump_id", "unknown"),
                "level": rec.values.get("_level", ""),
                "message": rec.get_value(),
                "time": rec.get_time().isoformat(),
            }

def fan_out(event: dict) -> None:
    key = dedup_key(event["check_id"], event["pump"], event["level"], event["time"])
    if key in SEEN:
        return  # already paged for this status in this minute
    SEEN.add(key)

    # Webhook, with the dedup key as an idempotency header the receiver can honour.
    requests.post(
        os.environ["ALERT_WEBHOOK_URL"],
        json={"text": event["message"], "severity": event["level"], "pump": event["pump"]},
        headers={"Idempotency-Key": key},
        timeout=5,
    )

    # Email for crit only; warn stays webhook-only to keep inboxes quiet.
    if event["level"] == "crit":
        msg = EmailMessage()
        msg["Subject"] = f"[CRIT] {event['pump']} pressure alarm"
        msg["From"] = os.environ["ALERT_FROM"]
        msg["To"] = os.environ["ALERT_ONCALL"]
        msg.set_content(event["message"])
        with smtplib.SMTP(os.environ["SMTP_HOST"], 587) as s:
            s.starttls()
            s.login(os.environ["SMTP_USER"], os.environ["SMTP_PASS"])
            s.send_message(msg)

def main() -> None:
    client = InfluxDBClient(
        url=os.environ["INFLUX_URL"],
        token=os.environ["INFLUX_MONITOR_TOKEN"],  # read-only on _monitoring
        org=os.environ["INFLUX_ORG"],
    )
    query_api = client.query_api()
    for event in read_new_statuses(query_api):
        fan_out(event)
    client.close()

if __name__ == "__main__":
    main()

Run this on the same one-minute cadence as the checks (cron, a systemd timer, or the platform scheduler). The since="-2m" window deliberately overlaps the previous run so no status slips through a scheduling gap; the dedup_key is what makes that overlap safe. In production, replace the in-memory SEEN set with a durable store (Redis with a TTL, or a small SQLite table) keyed on dedup_key, so a hook restart does not re-page everything in its lookback window.

# Configuration reference

Level predicates that leave gaps (no rule matches some value range) resolve to ok by default, which can silently clear a real condition — cover the whole numeric range across your predicates.

# Common failure modes and fixes

1. Alert storms from flapping thresholds. Symptom: a signal oscillating across the line pages dozens of times a minute; on-call mutes the channel and misses the next real event. Root cause: the check clears and re-fires on a single line with no hysteresis, and evaluation runs on raw samples rather than an aggregate. Fix: aggregate to a one-minute mean and add a dead band so recovery requires falling below a lower floor than the trip ceiling.

// Trip at 620, but require a drop below 600 to clear — the 20 kPa dead band
// is what stops a signal oscillating around 620 from re-paging every window.
crit = (r) => r.pressure_kpa > 620.0
ok = (r) => r.pressure_kpa <= 600.0

2. Deadman window shorter than the sensor reporting interval. Symptom: healthy sensors are flagged dead at random; alerts are ignored as noise. Root cause: the deadman cutoff t is close to or shorter than the interval at which the sensor actually reports, so normal gaps between writes look like silence. Fix: set t to at least two to three times the reporting interval, and keep the query range wider than t.

// 5s sensors: never set the cutoff near 5s. 3 minutes tolerates
// jitter, batching, and a missed write or two before flagging.
|> monitor.deadman(t: experimental.subDuration(d: 3m, from: now()))

3. Notification hook not idempotent, so it pages twice. Symptom: every alert arrives as two or three identical pages seconds apart. Root cause: the hook’s lookback window overlaps its previous run (by design, to avoid gaps) but it has no dedup key, so the same status row is delivered on consecutive runs. Fix: derive a stable dedup_key from check id, series, level, and minute-bucketed time, store delivered keys durably, and skip repeats — as the fan_out function above does.

4. Statistical band computed over a window contaminated by the anomaly. Symptom: a slow drift is never caught; the z-score stays near zero even as pressure climbs. Root cause: $\mu$ and $\sigma$ are computed over a window that includes the anomalous points, so the baseline drifts with the fault and the deviation vanishes. Fix: compute the baseline over a trailing window that excludes the most recent points under test, or over a known-good reference period, so the band reflects normal behaviour, not the fault.

5. Statuses written but nobody reads them. Symptom: checks run green in the task logs, _monitoring fills with crit rows, and no page is ever sent. Root cause: detection was deployed without the delivery stage, or the hook’s token cannot read _monitoring. Fix: deploy the notification hook on the same cadence as the checks and grant it a read-scoped token on _monitoring; verify with the query in the next section.

# Verification and testing

Confirm three things: that checks are writing statuses, that a threshold breach produces exactly one crit transition, and that the checks themselves are still running. Start by reading recent statuses straight from _monitoring:

from(bucket: "_monitoring")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "statuses")
  |> filter(fn: (r) => r._level == "crit" or r._level == "warn")
  |> keep(columns: ["_time", "_check_name", "pump_id", "_level", "_message"])
  |> sort(columns: ["_time"], desc: true)

To prove the dead band works, inject a value that oscillates across 620 kPa for several minutes and confirm the query above returns a single crit row followed by one ok, not a burst. Then guard the guards: a check task that silently stops scheduling leaves the plant unmonitored, so run a monitor.deadman over the statuses stream itself — if no status has been written by any check inside the window, the monitoring pipeline is the thing that has gone silent.

import "influxdata/influxdb/monitor"
import "experimental"

// Deadman on the monitoring pipeline: alert if the checks themselves stop writing.
from(bucket: "_monitoring")
  |> range(start: -15m)
  |> filter(fn: (r) => r._measurement == "statuses")
  |> group()
  |> monitor.deadman(t: experimental.subDuration(d: 5m, from: now()))
  |> filter(fn: (r) => r.dead == true)

Finally, confirm from the CLI that both check tasks are active and succeeding, so a paused or errored task is caught before it silently drops coverage:

influx task list --org iot-platform
influx task run list --task-id "$PRESSURE_CHECK_ID" --limit 5

# Integration points

Anomaly detection sits downstream of everything that produces reliable time-series and upstream of everything that acts on failure. The checks here are ordinary scheduled tasks, so their cadence and offset are governed by the same cron & interval scheduling logic that keeps any rollup aligned to its data — a check evaluated on a drifting schedule reads half-empty windows and flaps. The Flux inside each check follows the retry-safe, side-effect-aware discipline in Flux scripting for task automation, and the notification hook is a direct application of the Python client orchestration patterns — connection reuse, scoped tokens, and durable dedup state. The two focused techniques under this topic go deeper: threshold-based alerting with Flux and Python hooks develops the level-and-message design, while the companion deadman walkthrough covers reporting-interval math and per-series grouping. The task-management endpoints these checks are created and inspected through are documented in the InfluxDB Task API reference.

# FAQ

# Should I alert on raw samples or on an aggregate?

Almost always on an aggregate. Evaluating a threshold against raw 5-second samples makes every brief excursion a page and every oscillation a storm. Reduce the window to a one-minute mean (or median for noisier signals) inside the check, and pair it with a dead band so a signal hovering at the line clears only after it genuinely recovers. You lose sub-minute detection latency and gain a channel your on-call will not mute.

# How do threshold and deadman checks differ, and do I need both?

A threshold check evaluates the value of data that arrived; a deadman check evaluates the absence of data. They are complementary and you need both, because they fail to catch opposite problems: a threshold check is blind to a sensor that stops reporting, and a deadman check is blind to a sensor reporting a dangerous but present value. Run them as separate tasks writing to the same statuses stream so one notification hook handles both.

# Where do check statuses actually get stored?

In the system _monitoring bucket that ships with InfluxDB 2.x, in a measurement called statuses, one row per series per evaluation, tagged with _check_id, _level, and a rendered _message. Because statuses are ordinary time-series data, you can query, retain, and downsample them like any other measurement — including running a deadman over them to detect when the checks themselves stop running.

# How do I stop the notification hook from paging twice for one event?

Give each deliverable status a stable dedup key built from the check id, the series identity, the level, and a minute-bucketed timestamp, then record delivered keys in a durable store and skip any key you have already sent. This lets the hook’s lookback window safely overlap its previous run — which you want, so no status slips through a scheduling gap — without turning that overlap into duplicate pages.

# What value of k should I use for a z-score band?

It depends on how noisy the signal is and how much alert volume you can tolerate. $k = 3$ flags roughly the outer 0.3% of a normal distribution and suits relatively clean signals; $k = 4$ is quieter and better for jittery measurements like motor current. Critically, compute the mean and standard deviation over a trailing window that excludes the points under test, or the anomaly contaminates its own baseline and the deviation collapses toward zero.

# Related

• Threshold-based alerting with Flux and Python hooks — level design, dead bands, and message templating for monitor.check.
• Deadman checks for detecting silent IoT sensors — reporting-interval math, per-series grouping, and window sizing.
• Flux Scripting for Task Automation — the retry-safe Flux discipline every check task depends on.
• Python Client Orchestration Patterns — connection reuse, scoped tokens, and durable dedup state for the notification hook.
• Cron & Interval Scheduling Logic — keep check cadence aligned to the write cadence so windows are never half-empty.

Up one level: Automated Task Scheduling & Orchestration