jaque exposes its own vital signs in the Prometheus exposition format on
/metrics, so a jaque process is scraped the way every other target is.
This page is the reference for every jaque_* series: type, labels and
what a change in it means. The metrics are emitted by internal
instrumentation covering the engine's scheduling loop and the notifier's
delivery path, plus a collector that reports how far each event log
follower is behind. This page is exhaustive; a series not listed here
does not exist.
1. Where
/metrics is served on the same -listen address as the dashboard and
the command API; there is no separate flag or port. An empty -listen
turns all three off. -target sink and -target notifier processes
serve /metrics on their own -listen with the series that apply to
them.
curl -s 127.0.0.1:8080/metrics | grep jaque_
2. Event log followers
Every consumer of the event log -- the engine, the UI projection, the
Livestatus server, the notifier, each sink, the -events-out dump -- is
registered as a named follower. Lag is the number of entries it has not
yet applied.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
jaque_eventlog_last_seq |
Gauge | -- | Sequence of the most recently appended event log entry. |
jaque_eventlog_consumer_lag |
Gauge | follower |
Entries the named follower has not yet applied. Follower names are engine, ui, livestatus, notifier, eventdump and sink/<name>. |
jaque_build_info |
Gauge | -- | Always 1; proves the metrics pipeline end to end. |
3. Engine and scheduling
| Metric | Type | Labels | Meaning |
|---|---|---|---|
jaque_scheduler_drift_seconds |
Histogram | -- | Difference between a check's scheduled and actual run time. |
jaque_scheduler_queue_depth |
Gauge | -- | Checks currently queued by the scheduler. |
jaque_engine_not_owned_total |
Counter | -- | Inputs dropped because this engine does not own the object (ADR-015 D5). |
4. Checks and work dispatch
| Metric | Type | Labels | Meaning |
|---|---|---|---|
jaque_check_duration_seconds |
Histogram | check_type |
Duration of an executed check, by check type. |
jaque_check_timeouts_total |
Counter | check_type |
Checks that timed out, by check type. |
jaque_work_dispatched_total |
Counter | zone |
Checks dispatched to a remote worker, by zone. |
jaque_work_timeouts_total |
Counter | zone |
Dispatched checks that timed out waiting for a result, by zone. |
jaque_work_rejected_total |
Counter | zone |
Checks that failed to dispatch to the work queue, by zone. |
jaque_work_late_results_total |
Counter | -- | Results received for a dispatch no longer pending. |
5. Cluster and membership
| Metric | Type | Labels | Meaning |
|---|---|---|---|
jaque_membership_size |
Gauge | -- | Members currently in the coordination membership view. |
jaque_membership_epoch |
Gauge | -- | Epoch of the last membership view observed by this engine. |
jaque_engine_owned_objects |
Gauge | -- | Objects this engine currently owns. |
jaque_membership_heartbeat_failures_total |
Counter | -- | Membership heartbeats that failed. |
jaque_engine_stale_epoch_total |
Counter | -- | Tail events dropped for carrying a stale membership epoch. |
jaque_engine_epoch_behind_total |
Counter | -- | Tail events observed with an epoch ahead of this engine's own. |
jaque_engine_commands_forwarded_total |
Counter | outcome |
Commands forwarded to another engine for an object this engine does not own, by outcome (sent, error). |
6. Notifications
| Metric | Type | Labels | Meaning |
|---|---|---|---|
jaque_notify_suppressed_total |
Counter | reason |
Notifications suppressed, by reason: downtime, acked, outside_period, kind_disabled, too_soon, no_level. The order they are checked in is in Escalations and windows. |
jaque_notify_delivery_failures_total |
Counter | reason |
Notification deliveries that failed. Today reason always carries the literal value adapter: every delivery failure lands in one bucket, not one per adapter (webhook, email, command, telegram). |
jaque_notifier_pending |
Gauge | -- | Notifications currently pending delivery across a notifier's shards. |
jaque_notifier_blocked_total |
Counter | -- | Times a notifier's log follower blocked because a delivery shard's queue was full (ADR-015 D5: block, never drop). |
7. Sinks
| Metric | Type | Labels | Meaning |
|---|---|---|---|
jaque_sink_dropped_total |
Counter | sink |
CheckExecuted events skipped by a perfdata sink's ownership gate because this process does not own the object (ADR-015 D5), by sink name. With several sinks one event counts once per sink. |
jaque_sink_write_errors_total |
Counter | sink |
Batches a perfdata sink failed to write, by sink name. |
8. UI and API
| Metric | Type | Labels | Meaning |
|---|---|---|---|
jaque_ui_commands_submitted_total |
Counter | outcome |
Commands a -target ui process submitted to the engine fleet over jaque.work.cmd.ingress, by outcome (sent, error). |
The Prometheus client library's default collectors are exposed as well:
the go_* and process_* metric families, covering scheduler activity,
GC, memory and file descriptors.
9. What to watch
jaque_eventlog_consumer_lag climbing on any follower is the earliest
warning this page has. Every downstream of the log -- projection, sink,
notifier, Livestatus -- is a follower, and a follower that is falling
behind is the first visible symptom of a slow sink endpoint, a blocked
delivery shard or a starved process, before a dashboard goes stale or a
page is late. Alert on it per follower; the follower label says which
subsystem to look at.
jaque_notifier_blocked_total and jaque_sink_write_errors_total are
the two counters that name a receiver outside jaque as the problem.
10. Security considerations
/metrics shares -listen with the dashboard and the command API and is
not protected by -api-token. Series carry subsystem labels and names of
sinks, zones and followers, never object output or secrets. Expose the
listener to the scraper and to nothing else; see
Security.