Disk alarm

A full disk takes the database down with it, and nothing else on the box says so first. Since v0.3.3 the kernel measures the disk it runs on every minute and warns when it is filling: once in the log, and on every page of the admin for as long as it stays that way. It is on by default.

What it measures

The used share of one filesystem, as df prints Use%: used ÷ (used + available to non-root). Blocks reserved for root count as neither.

The filesystem is the one holding FORGE_DISK_WATCH_PATH inside the kernel container (default /). With Docker’s default overlay storage a container’s / sits on the host disk that also holds Docker’s volumes — the caches, the edge’s log — so on a single-disk box the default is right and there is nothing to configure.

If a cache or other data lives on a separate disk, mount that path into the kernel (read-only is enough) and name it:

# compose override, kernel service
volumes:
  - /mnt/data:/watch/data:ro
environment:
  FORGE_DISK_WATCH_PATH: /watch/data

The kernel watches one path. A box with two data disks watches the one that fills first.

The threshold

FORGE_DISK_ALARM_PERCENT, default 85. At or above it the state is high. A value outside 1–100 is refused at boot rather than silently never alarming. 85 % leaves room to act while there are still gigabytes to free.

Three states

state	Means	Log	Admin
ok	Measured, below the threshold	Nothing (only the line when it comes back)	Nothing
high	Measured, at or above the threshold	One line, on the crossing	A red band on every page
unknown	Not measured — the measurement failed, has not run yet, or is stale	One line, with the error	A yellow band, with the reason

“I could not look” is never shown as ok: a last measurement older than three minutes answers unknown with reason: stale.

Reading it

In the log — one line per transition, never per minute. The stable keys come first; a monitoring rule can match on disk_alarm state=high:

[forge-api] disk_alarm state=high path=/ threshold_percent=85 used_percent=91.4 available_bytes=8600000000 total_bytes=100000000000 — …

docker compose logs kernel | grep 'disk_alarm state='

Through the port — read.internal.disk_usage, on the internal face, with an operator credential that has tenant.settings.read. The result’s shape is on that page.

curl -s -H "authorization: Bearer $TOKEN" https://<your-domain>/v1/read/internal/disk_usage

It is deliberately not in /health: /health is anonymous and served on the store’s own hostname, and how close a merchant’s box is to falling over is not public.

In the admin — the band sits above every authenticated page while the state is not ok. An operator whose role lacks tenant.settings.read sees no band; the log still says it.

What it does not do

It calls nobody — no e-mail, no phone — and frees nothing. When it fires, find what grew (docker system df -v) and free it before the database runs out. The usual suspects and what is safe to delete are on Volumes & backups; the edge’s access log is bounded and cannot be the cause (see Edge access log).