Production alerting requires keeping signal-to-noise high during planned maintenance, rapid autoscaling, and transient blips.
Alert Muting Rules suppress notifications (and optionally hide muted incidents in dashboards) for scheduled maintenance windows or specific dimensions (such as env:staging or cluster:blue-deploy) without disabling the underlying detector or losing evaluation history. Before saving a new detector, always run Detector Preview (backtesting) against historical data to see how many times it would have fired over the past hours or days.
To prevent alert flapping on noisy or ephemeral Kubernetes workloads, combine lasting durations, hysteresis (separate trigger and clear thresholds), EWMA smoothing, and Null extrapolation so disappearing containers do not leave stale alerts stuck open.
mem = data('memory.utilization').mean(by='pod.name')
# Fires above 90% for 5m; clears only after dropping below 80% for 5m
detect(when(mem > 90, lasting='5m'), off=when(mem < 80, lasting='5m')).publish('Pod Mem High')