A notification policy is evaluated by one pure function: given the object's notification state, the policy, the kind of notification the state machine asked for and the current time, it returns either the contacts to notify or a reason not to. This page specifies that function. It has two independent clocks, the re-notification interval and the escalation threshold, and one gate, the time windows, and the three are easy to conflate. Nothing here performs I/O; the decision is a value the engine appends to the event log.
1. Levels
levels: [
{contacts: ["alice"], threshold: 3, renotify_interval: "30m"},
{
contacts: ["oncall_hook", "bob", "carol", "dave"]
renotify_interval: "10m"
disabled_kinds: ["FLAPPING_START", "FLAPPING_STOP"]
},
]
levels is a non-empty list; a policy without one is a config error. A
problem starts at the first level and only ever moves down the list while
it persists. The level never decreases during a problem and returns to
the first level on recovery.
| Field | Type | Default | Meaning |
|---|---|---|---|
contacts |
non-empty list of contact names | -- | who this level notifies |
threshold |
int, >= 0 |
0 |
how many notifications this level sends before the next level takes over; 0 means the ladder never escalates past this level |
renotify_interval |
duration | 1h |
the minimum gap between two PROBLEM notifications from this level |
disabled_kinds |
list of kinds | [] |
kinds this level does not send |
The kinds are PROBLEM, RECOVERY, STATUS_CHANGE, FLAPPING_START
and FLAPPING_STOP. A level that should only page can turn off the
flapping meta-notifications with disabled_kinds rather than needing a
second policy.
2. Re-notify versus escalate
These are two independent counters, not one. renotify_interval throttles
the current level: a PROBLEM evaluation that arrives before the interval
has elapsed since the level's last send is suppressed with reason
TOO_SOON. threshold counts the level's sends: once the current level
has sent threshold notifications, the next evaluation escalates, the
count resets, and the next level's contacts and interval apply from then
on.
With the example above, whose first level has threshold: 3 and
renotify_interval: 30m, a problem that keeps being evaluated runs like
this:
t+0 level 0 alice PROBLEM count 1/3
t+30m level 0 alice PROBLEM count 2/3 (30m interval elapsed)
t+60m level 0 alice PROBLEM count 3/3 threshold reached
t+90m level 1 oncall_hook PROBLEM count 1 escalated; 10m interval from here
t+100m level 1 oncall_hook PROBLEM count 2
...
recovery level 1 oncall_hook RECOVERY ladder resets to level 0
Two details follow from the counters being separate. The interval that
throttles is the one on the level that sent last, so the ladder slows
down or speeds up as it moves. And the last level's threshold is
irrelevant: with no level after it there is nowhere to escalate to, and
the count keeps growing without effect.
3. Recovery and flapping
A RECOVERY always notifies the level the problem had reached, then
resets the ladder: level and count return to the base, the re-notify
throttle is cleared so the next problem's first notification is never
swallowed by the closed one, and a stale acknowledgement is dropped. A
problem that recovers and immediately returns starts again at level 0.
FLAPPING_START and FLAPPING_STOP notify the current level without
touching the ladder: they neither count toward threshold nor reset the
re-notify throttle. They are meta-notifications about the object's
stability, not part of the problem's cadence.
4. Windows
period: [
{weekday: "monday", start: "09:00", end: "17:00"},
{weekday: "tuesday", start: "09:00", end: "17:00"},
]
A window is one weekday plus a start and end in 24-hour HH:MM.
end may be 24:00; start may not. A window does not wrap past
midnight, so an overnight shift is two windows: 22:00--24:00 on one
day and 00:00--06:00 on the next. The policy is open whenever the
current time falls inside any window in period. An empty period, the
default, is always open.
Outside every window a notification of any kind is suppressed with
reason OUTSIDE_PERIOD. It is not queued for delivery when the window
opens: the condition is dropped, and the next notifiable transition is
evaluated on its own terms. A policy that wants a recovery announced at
09:00 for a problem that happened at 03:00 does not get one.
5. Order of suppression
Evaluate checks its gates in a fixed order and stops at the first that
fires. Each reason is logged by name and counted in
jaque_notify_suppressed_total, whose reason label carries the same
name in lower case (downtime, too_soon, ...).
| Order | Condition | Reason | Silences |
|---|---|---|---|
| 1 | the object is in a downtime | DOWNTIME |
every kind |
| 2 | the object is acknowledged | ACKED |
PROBLEM and STATUS_CHANGE only; RECOVERY still goes out |
| 3 | now is outside every window | OUTSIDE_PERIOD |
every kind |
| 4 | the current level does not exist | NO_LEVEL |
every kind |
| 5 | the kind is in the level's disabled_kinds |
KIND_DISABLED |
that kind |
| 6 | a PROBLEM arrives before renotify_interval elapsed |
TOO_SOON |
PROBLEM only |
An acknowledged problem therefore still announces its recovery, and a downtime silences even that. Acknowledgements and downtimes are suppression signals, not states; the state model keeps transitioning underneath them and the event log records every transition whether or not anyone was told.