> Section: [8. Observing it](https://jaque.sh/docs/ui/dashboard.md)
> Next: observability/archive
> Index: https://jaque.sh/llms.txt


jaque is not a time series database and does not store perfdata. A sink
is a follower of the event log that forwards every `CheckExecuted`
event's perfdata, or the whole log, to a backend the operator already
runs: ClickHouse, anything that speaks Prometheus remote write, an
OpenTelemetry Collector, an HTTP endpoint, a process, a file, or a
zstd-compressed archive. Sinks are declared under `sinks:` in the CUE
config, selected per process with `-sinks`, and reconciled on reload. Each
perfdata point is normalized once, at the forwarder, to a canonical unit,
so every backend sees the same value for the same plugin output.

A sink only writes. To read perfdata back and graph it in the dashboard,
declare a `datasources:` entry pointed at the same backend -- see
[Datasources and panels](https://jaque.sh/docs/ui/panels.md).

Status: `-target sink` does not hot reload. See the
[status page](https://jaque.sh/docs/project/status.md).

## 1. The `sinks:` block

Each sink is a named entry under `sinks:`, discriminated by `type` the
same way `#Checks` and `#Contacts` are.

```cue
sinks: {
	perf: {
		type:  "clickhouse"
		url:   "clickhouse://ch:9000/jaque"
		table: "perfdata"
	}
	prom: {
		type: "remote_write"
		url:  "http://victoriametrics:8428/api/v1/write"
	}
	log: {
		type:          "archive"
		url:           "file:///var/lib/jaque/archive"
		segment_bytes: 67108864
	}
	webhook: {
		type:     "http"
		url:      "https://ingest.example.com/perfdata"
		headers: {"Authorization": "Bearer ..."}
		selector: "env=prod"
	}
	pipe: {
		type:    "exec"
		command: ["/usr/local/bin/jaque-sink.sh"]
	}
	disk: {
		type:         "file"
		path:         "/var/log/jaque/perfdata.jsonl"
		rotate_bytes: 67108864
	}
	otel: {
		type: "otlp"
		url:  "http://otel-collector:4318"
	}
}
```

Several sinks of the same type coexist: two `archive` entries pointing at
different destinations are both valid.

### 1.1 Common fields (`#SinkCommon`)

| Field | Type | Default | Meaning |
|---|---|---|---|
| `input` | `"metrics"` or `"events"` | `"metrics"` | `metrics` sees perfdata points; `events` sees the whole event log. Only `archive`, `http`, `exec` and `file` accept `events`, and `archive` requires it. |
| `selector` | label selector string | `""` | Objects whose events this sink forwards; empty means no filter beyond ownership. |
| `batch_size` | int `> 0` | `1000` | Rows buffered before a write. |
| `flush_interval` | duration | `5s` | Longest a partial batch waits before a write. |

### 1.2 Per-type fields

| Type | Fields | Notes |
|---|---|---|
| `clickhouse` | `url`, `table`, `cluster` | `table` defaults to `perfdata`; `cluster` defaults to none |
| `remote_write` | `url`, `cursor` | Prometheus remote write; series shape in section 4 |
| `archive` | `url`, `segment_bytes` | the whole log as zstd JSON lines; see [Archive](https://jaque.sh/docs/observability/archive.md) |
| `http` | `url`, `headers`, `cursor` | JSON batches to an HTTP endpoint |
| `exec` | `command`, `cursor` | JSON lines on a process's stdin |
| `file` | `path`, `rotate_bytes`, `cursor` | JSON lines on disk, rotated by size |
| `otlp` | `url`, `headers`, `cursor` | OTLP over HTTP to an OpenTelemetry Collector |

`cursor` is a path where the sink remembers the last sequence it sent.
Without one, `LastSeq` is always 0 and a persistent log resends its
retained window on every restart.

### 1.3 The clickhouse sink's schema and migrations

jaque creates and migrates the `clickhouse` sink's table itself on
startup; there is no schema to apply by hand. Migrations are idempotent
and safe when several replicas start at the same time: each one is
recorded in `<table>_migrations` as it completes, and a migration
already recorded there is skipped on the next run. Declaring `cluster`
runs every migration's DDL `ON CLUSTER` and switches the table to a
replicated engine instead of a local one.

## 2. Delivery: retries and what a 4xx means

Failures are handled by kind. A `429` or any `5xx` retries the same
batch. Any other `4xx`, a malformed label for instance, discards the batch
with a logged warning and advances the cursor anyway: retrying forever
against an endpoint that will never accept that batch is not a retry, it
is a stuck sink. A retried batch counts in
`jaque_sink_write_errors_total` by sink name; a discarded one is only in
the log, because from the sink's point of view it was handled.

## 3. Unit normalization

Every perfdata point is normalized once, at the forwarder boundary, to a
canonical UCUM unit plus a scale factor and a kind (`gauge` or
`counter`). The raw value and UOM are kept as metadata, never discarded.
The [SNMP check](https://jaque.sh/docs/checks/native-checks.md) is the native counter
source: Counter32/Counter64 values reach this table already tagged `c`.

| Raw UOM | Canonical unit | Factor | Kind |
|---|---|---|---|
| (empty) | `1` | 1 | gauge |
| `s` | `s` | 1 | gauge |
| `ms` | `s` | 1e-3 | gauge |
| `us` | `s` | 1e-6 | gauge |
| `ns` | `s` | 1e-9 | gauge |
| `B`, `b` | `By` | 1 | gauge |
| `KB`, `kb`, `KiB` | `By` | 1024 | gauge |
| `MB`, `mb`, `MiB` | `By` | 2^20 | gauge |
| `GB`, `gb`, `GiB` | `By` | 2^30 | gauge |
| `TB`, `tb`, `TiB` | `By` | 2^40 | gauge |
| `PB` | `By` | 2^50 | gauge |
| `%` | `%` | 1 | gauge |
| `c` | `1` | 1 | counter |

Lowercase `b`, `kb`, `mb`, `gb` and `tb` are aliases of their uppercase
forms. There is no lowercase alias for `PB` and none for the `*iB`
variants, which already normalize to the same factor as their base form.
An unrecognized UOM passes through untouched, factor 1, kind gauge: the
table never rejects a point, it stops normalizing it.

## 4. The series shape

For a `remote_write` sink, `__name__` is derived from the kind and the
normalized unit, not the raw UOM.

| Kind and unit | `__name__` | `uom` label |
|---|---|---|
| counter | `jaque_perfdata_total` | absent |
| gauge in `s` | `jaque_perfdata_seconds` | absent |
| gauge in `By` | `jaque_perfdata_bytes` | absent |
| gauge in `%` | `jaque_perfdata_percent` | absent |
| gauge in `1` | `jaque_perfdata` | absent |
| gauge in any other unit | `jaque_perfdata` | the raw UOM |

The labels `host`, `service` (service-level results only), `label` (the
perfdata point's name) and `origin` are always present. One `Sample` per
series. `warn`, `crit`, `min` and `max` are not emitted; thresholds live
in the check config, not in the series.

## 5. Choosing sinks per process: `-sinks`

Which sinks a process runs is a flag, not part of the config graph.
`-sinks perf,prom` selects by name; empty runs every declared sink. A
name absent from the merged `sinks:` block is a startup error that lists
the valid names.

## 6. Which process runs a sink

Which process runs a sink is a property of the sink's `input` class, not
a per-process flag list: metrics-input sinks (`clickhouse`,
`remote_write`, and any other sink with `input: "metrics"`) run on
`-target sink` and `-target all` only; events-input sinks (`archive`, and
any `http`/`exec`/`file` sink with `input: "events"`) run on `-target
engine` and `-target all` only. `-target engine` never runs a metrics
sink; `-target sink` never runs an events sink.

`-sinks` (the chart's `sink.select`) still narrows which of the declared
sinks a process runs, on top of the role rule above. A sink the role rule
drops while it is explicitly named in `-sinks` is a startup error naming
it; dropped implicitly -- an empty `-sinks` running every sink the role
allows -- it is only logged, once, listing every sink skipped.

## 7. Hot reload, with one exception

SIGHUP on `-target all|engine` reconciles the set of sinks by name: a sink
new to the config starts, one that disappeared
stops, and one whose spec changed (URL, type, selector, ...) is rebuilt.
Sinks that did not change are not touched. `-target sink` does not handle
SIGHUP at all, the same as `-target ui`, so a change to which sinks run
there means a restart.

A sink's `selector` is evaluated live against the current config on
every event. An object a reload added that matches a still-running
sink's selector is picked up without that sink restarting.

`-target sink` requires `-config` to resolve the `sinks:` block, though it
runs no engine and no projection of its own. Why events sinks stay off it
is the role rule in section 6: `-target sink` shards the log by object
ownership, and an events sink looks at the whole log rather than at
objects, so it only makes sense under `-target all|engine`, which runs one
unsharded engine.

## 8. Extensibility: nothing in-process

There is no plugin interface for sinks: no WASM (WASM support is scoped
to checks), no directory of modules. A backend not listed above is
reached either through `otlp` to an OpenTelemetry Collector, whose
exporters cover most of the field, or through an `exec` sink that reads
JSON lines on stdin. Never a fork, never a plugin directory.

## 9. Ownership in a clustered deployment

With `-target engine|all` and `replicas > 1` sharing a log, or with
several `-target sink` processes, each `CheckExecuted` event is written by
exactly one owner; skipped writes are counted in
`jaque_sink_dropped_total`. After a membership change, a sink picking the
log back up may re-evaluate events it already saw under the new
membership view. That is a transient duplicate, and ClickHouse's
`ReplacingMergeTree` collapses it on its own.

## 10. A complete setup

A `sinks:` block declaring one `remote_write` entry pointed at a
Prometheus-compatible backend, or one `otlp` entry pointed at an
OpenTelemetry Collector, is a complete, working setup. The examples in
section 1 are the production mechanism with placeholder endpoints, not a
simplification of it.

## 11. Security considerations

A sink's `url` and `headers` can carry credentials, as the `http` example
above does with a bearer token, and so can a ClickHouse or S3 URL's
userinfo. The CUE config is then a secret and must be treated as one.
`exec` sinks run a process on the sink host with the event stream on
stdin; the `command` is trusted configuration. Sink traffic is outbound
only; a sink process opens no listener beyond `/metrics` on `-listen`.