Detection

Missing security telemetry: monitor the silence as well as the alerts

A quiet dashboard can mean no incidents, a broken producer, or a source that vanished from inventory.

3 min readPrometheusMonitoringSIEM

What should happen when a security sensor goes quiet? A failed target, a reachable collector receiving stale data and a source missing from discovery need different checks. Keep an independent inventory of expected sources so one disappearing from the dashboard still raises attention.

Distinguish three failure states

A known target can be down. A known target can be reachable but stale. A target can be missing from discovery entirely. One uptime graph does not cover all three.

Use an inventory of required sensors and log producers. Include ownership, expected event cadence, and maintenance rules. Compare observed sources against that inventory rather than deriving the entire expectation from whatever happened to report today.

Express freshness explicitly

Prometheus documents absent_over_time and time functions in its query reference. For an exporter that you implement to publish the most recent event time, illustrative expressions are:

(time() - security_last_event_timestamp_seconds{source="storage-audit"}) > 900
absent_over_time(security_last_event_timestamp_seconds{source="storage-audit"}[15m])

security_last_event_timestamp_seconds is an example metric, not a built-in metric supplied by a storage vendor or Prometheus. The first expression detects old values; the second detects no samples for the explicitly named source. Neither automatically enumerates every missing source in a changing fleet.

The 15-minute window is illustrative. Select the threshold from the system’s real event behavior and response requirement. A genuinely idle source needs a heartbeat or controlled validation event rather than an assumption that user activity is continuous.

Test the alert chain

In a lab, stop the event producer while leaving the exporter running. Confirm the stale-event alert. Then stop the exporter and confirm missing-sample or target-down behavior. Finally remove the source from discovery and verify that inventory reconciliation still identifies it.

These tests isolate different failure modes. If the third condition becomes invisible, the monitor still has an absence problem.

Carry enough context to act

An alert should name the producer, last event time, last successful collection, owning team, and a short diagnostic procedure. A generic “no data” notification encourages responders to guess whether the event source, network, parser, or query failed.

Preserve maintenance windows with expiry. A permanent silence created for a temporary outage can make the monitoring system look healthy for months.

Check timestamps and delivery

Clock skew can make a producer appear fresh or stale incorrectly. Compare event time with ingest time and monitor clock synchronization independently. Confirm that alert routing reaches a staffed destination using an approved test notification workflow.

Test the response to each failure: a missing source, stale data and an unavailable target. Keep the expected-source inventory independent of the systems it is checking. Apply the same approach to storage audit pipelines.

All articles