kill -9 a jaque engine mid-check and restart it: it comes back with the
state it had a moment before, history included. That is not a snapshot
written on the way out; it is a consequence of how the system is built.
Every fact is an event appended to one log, current state is a projection
folded from that log, and every consumer -- the engine's own state, the
dashboard, the notifier, the perfdata sinks -- is a fold at its own read
position. This page states that rule, shows one event fanning out, and
names what the shape buys beyond an audit trail.
Status: the durable transports are file:// and nats://; the default
memory transport keeps nothing across a restart, so the kill -9
property holds only under one of the other two. The reload-rejection event
is named ConfigReloadRejected in the code; the name ConfigRejected
from an earlier version of this page is documented, not verified. See
Status.
1. Nothing mutates state directly
A check result arrived, a state transitioned, an ack was set, a downtime was scheduled, a notification was sent: each is an event appended to the log, and no component reaches in and flips a status field. "Current state" is a projection: what you get by replaying the log from the start and folding each event into a running view. The engine that decides transitions, the UI, the notifier and the sinks are different folds over the same log. None of them talk to each other.
state machine
|
v
event log 41 CheckExecuted | 42 StateChanged | 43 NotificationRequested
| | |
v v v
sink (perfdata) projection Table notifier Consumer
(at 41) ui, api, livestatus (at 43) -> adapter
^
|
replay from 1 rebuilds it
state machine
|
v
event log 41 CheckExecuted | 42 StateChanged | 43 NotificationRequested
| | |
v v v
sink (perfdata) projection Table notifier Consumer
(at 41) ui, api, livestatus (at 43) -> adapter
^
|
replay from 1 rebuilds it
The names in the figure are the real ones. CheckExecuted carries the
result and its perfdata; StateChanged is the transition the state
machine decided; NotificationRequested is appended by the owning engine
after the notification policy has run. The projection Table folds the
first two into the per-object read model the dashboard, the query API and
Livestatus read. The notifier's Consumer folds the third and delivers
through an adapter. A sink reads the perfdata and never looks at the rest.
Nothing downstream writes back into the path above it.
2. Why this buys more than an audit trail
Crash recovery is free. There is no "save state on exit" path to get wrong: a process that dies mid-check loses nothing beyond the in-flight check, because state was never anywhere but the log.
History is the same mechanism as current state, not a table bolted on
beside it. ListHistory in the query API and the dashboard's timeline
read the same log a fresh replica would replay from zero.
Hot config reload is an event too. ConfigReloaded, and the rejection
event when a new config fails validation, land in the log like anything
else, so "what changed and when" is answerable after the fact rather than
observable only in a log line that scrolled past.
Consumers scale independently. A perfdata sink, a Livestatus reader and the engine's own projection each follow the log at their own pace. A slow one does not block a fast one; it falls behind, and the lag is a metric you can alert on. See Metrics.
3. The log itself
The log has three transports, selected with -eventlog. memory is the
default: no durability, replay only from process start, right for a
first run and wrong for anything you would miss. file:// is the
embedded, on-disk event log: durable, no external service, the
transport a single-process deployment should run on; see
Overview. nats:// is an external,
shared event log server, for clustered deployments where more than one
process reads the same log; see
Cluster and coordination.
The events are identical on all three; only where they rest differs.
4. What this is not
The event log is not a metrics time series database and does not try to
be one. Perfdata, the numeric measurements a check produces, is a separate
concern that flows out to a Sink -- ClickHouse, remote_write and the
others -- for whoever already owns that discipline. What the log
guarantees is that every decision the engine ever made is reconstructable,
not that every number a check ever printed lives there forever.
Archive is the log's own long-term
retention, opt-in and separate from perfdata sinks.
5. Security considerations
The log is the system of record, and it holds everything a check printed:
plugin output, acknowledgement comments, the addresses and names in your
config. A file:// log is a directory with the permissions you gave it; a
nats:// log is reachable by anything that can reach the log server.
Protect both the way you would protect the config that produced
them, and read Security before exposing
either outside the host.