Observability Stack

Breeze includes an optional observability stack as a separate Docker Compose overlay (docker-compose.monitoring.yml). Enable it alongside the core stack to get full metrics, dashboards, log aggregation, and infrastructure alerting.

docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d

Components

Component	Port	Purpose
Prometheus	9090 (localhost)	Time-series metrics collection and alerting rules
Grafana	3000 (localhost)	Dashboards and visualization
Alertmanager	9093 (localhost)	Alert routing and notifications
Loki	3100 (localhost)	Log aggregation and querying
Promtail	9080 (localhost)	Log shipping from Docker containers to Loki
Redis Exporter	9121 (internal)	Exports Redis metrics for Prometheus
Postgres Exporter	9187 (internal)	Exports PostgreSQL metrics for Prometheus

Note

All monitoring ports bind to 127.0.0.1 by default. Access Grafana and Prometheus via SSH tunnel or VPN – they should not be exposed to the public internet.

Accessing Grafana

# Via SSH tunnel
ssh -L 3000:127.0.0.1:3000 user@your-server

# Then open http://localhost:3000
# Username: admin
# Password: (your GRAFANA_ADMIN_PASSWORD from .env)

Tip

You can override the Grafana port with the GRAFANA_PORT environment variable. Similarly, PROMETHEUS_PORT, ALERTMANAGER_PORT, and LOKI_PORT let you change the other monitoring ports.

Pre-Built Dashboards

Breeze ships with a Grafana dashboard (monitoring/grafana/dashboards/breeze-overview.json) that is automatically provisioned. It includes these panels:

Panel	What It Shows
Service Status	Up/down status of API, Redis, PostgreSQL, and other services
Request Rate	HTTP requests per second with breakdown by method
Response Times	P50, P95, and P99 latency over time
Error Rate	4xx and 5xx response rates as percentages
HTTP Status Distribution	Breakdown of responses by status code
Top Endpoints	Most-used API endpoints by request volume
Active Devices	Count of agents with recent heartbeats
Organizations	Number of active tenants
Redis Memory	Memory usage, evictions, and hit rate
PostgreSQL Connections	Active connection count vs. max pool size

Four panel groups shipped after the table above was originally written and aren’t reflected in it:

Panel	What It Shows
Requests by Response Class	Request rate broken down by status class over time, alongside the status-code pie chart
Redis Operations	Redis commands processed per second
Active Alerts	Count of currently open in-app alerts
Agent Heartbeats	Agent heartbeat rate over time
Script Executions	Script execution rate over time
Backup Operations — Start Failures (24h)	Backup, restore, or verification start failures in the last 24 hours
Backup Operations — Restore Timeouts (24h)	Restore command timeouts in the last 24 hours
Backup Operations — Verification Skips (24h)	Scheduled backup verification runs skipped in the last 24 hours

Adding Custom Dashboards

To add your own dashboards:

1. Create or import a dashboard in the Grafana UI.
2. Export it as JSON from the Grafana dashboard settings.
3. Save the JSON file to monitoring/grafana/dashboards/.
4. The dashboard provisioner (monitoring/grafana/dashboards.yml) rescans that directory every 30 seconds and picks up new files automatically — no Grafana restart needed.

Data Sources

Configured automatically via monitoring/grafana/datasources.yml:

Source	Type	URL
Prometheus	Time-series	http://prometheus:9090
Loki	Logs	http://loki:3100
PostgreSQL	SQL	postgres:5432
Redis	Key-value	redis://redis:6379

Prometheus Configuration

Located at monitoring/prometheus.yml. Key settings:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s
  external_labels:
    monitor: 'breeze-rmm'
    environment: 'production'

Scrape Targets

Job	Target	Interval	Auth
prometheus	localhost:9090	15s (default)	None
breeze-api	api:3001 at /metrics/scrape	10s	Bearer token
breeze-worker	worker:3001 at /metrics	10s	Bearer token
redis	redis-exporter:9121	15s (default)	None
postgres	postgres-exporter:9187	15s (default)	None
node	node-exporter:9100 (optional)	15s (default)	None

Tip

The breeze-worker target is only present when the worker-split Compose profile is running (docker compose --profile worker-split up -d worker). Outside a split deployment the target is simply reported DOWN, same as the optional Redis/Postgres/node exporters — this is expected. Unlike breeze-api, the worker exposes /metrics directly from a slim raw-node:http server rather than the Hono route graph, and it publishes only the role-agnostic runtime series (event-loop lag, DB pool health) — not the HTTP/business/fleet series breeze-api exposes. Filter dashboards and alerts on job="breeze-worker" vs job="breeze-api" rather than assuming a single source, or the two roles’ series can be double-counted once both report.

The API metrics endpoint is protected by a bearer token. Store the token in monitoring/secrets/metrics_scrape_token and reference it in prometheus.yml:

- job_name: 'breeze-api'
  metrics_path: /metrics/scrape
  authorization:
    type: Bearer
    credentials_file: /run/secrets/metrics_scrape_token

Rule Files

Prometheus loads all .yml files from monitoring/rules/. Breeze ships with breeze-rules.yml containing API, infrastructure, and business alert rules plus recording rules for common aggregations. See Infrastructure Alerts for the full list.

Rules are evaluated every 30 seconds (API and infrastructure groups) or every 60 seconds (business alerts). Recording rules pre-compute expensive queries:

Recording Rule	Description
breeze:http_requests:rate5m	Request rate by status, method, and route
breeze:http_error_rate:ratio5m	5xx error rate as a ratio
breeze:http_request_duration:avg5m	Average request duration by route
breeze:http_request_duration:p95_5m	95th percentile request duration
breeze:http_request_duration:p99_5m	99th percentile request duration
breeze:devices:active_count	Total active device count
breeze:redis_ops:rate5m	Redis operations per second
breeze:postgres_query_duration:avg5m	Average PostgreSQL query duration
breeze:backup_dispatch_failures:increase15m	Backup/restore/verification start failures over the last 15 minutes, by operation and reason
breeze:restore_timeouts:increase15m	Restore command timeouts over the last 15 minutes, by command type
breeze:backup_verification_skips:increase1h	Scheduled backup verification skips over the last hour, by verification type and reason
breeze:failed_logins:increase5m	Failed logins over the last 5 minutes, by reason and tenant
breeze:agent_enrollments:increase10m	Agent enrollments over the last 10 minutes, by result and tenant
breeze:commands_dispatched:increase5m	Commands dispatched over the last 5 minutes, by type, actor, and tenant

Key Metrics

HTTP Metrics (from the API)

Metric	Type	Description
http_requests_total	Counter	Total HTTP requests by method, path, status
http_request_duration_seconds	Histogram	Request latency distribution
http_requests_in_flight	Gauge	Currently processing requests
agent_heartbeat_total	Counter	Agent heartbeats received

Caution

These series require Breeze v0.107.0 or newer. On earlier releases the request-timing instrumentation was never wired into the API, so http_requests_total, http_request_duration_seconds, and agent_heartbeat_total were always absent – the /metrics endpoint answered normally, the HTTP panels of the pre-built Grafana dashboard simply stayed empty, and the NoAgentHeartbeats alert could never fire. If those panels are blank, check the API version before troubleshooting Prometheus.

Business Metrics

Metric	Type	Description
breeze_active_devices	Gauge	Devices with a recent heartbeat
breeze_active_organizations	Gauge	Organizations with active devices
breeze_commands_total	Counter	Commands executed, labeled by type
breeze_alerts_total	Counter	Alerts fired, labeled by severity

Infrastructure Metrics

Metric	Type	Description
redis_memory_used_bytes	Gauge	Redis memory consumption
redis_commands_processed_total	Counter	Total Redis commands processed
pg_stat_activity_count	Gauge	PostgreSQL active connections
pg_database_size_bytes	Gauge	Database size in bytes
pg_settings_max_connections	Gauge	PostgreSQL max allowed connections

Since v0.117.0 the API also publishes its own view of Redis memory — breeze_redis_used_memory_bytes, breeze_redis_maxmemory_bytes and breeze_redis_memory_used_ratio — and warns once usage crosses 80% of maxmemory. Redis runs noeviction, so a full instance fails writes; alert on the ratio well before 1. See Redis memory watchdog for the full series list and tuning variables.

Event-Loop Metrics

The API exports its own event-loop delay so a stalled main thread is visible as such, rather than only as the downstream errors it causes (see Event-loop lag monitor for the tuning variables). All five series are published from process start, so an alert rule referencing them never queries a metric that does not exist yet.

Metric	Type	Description
breeze_nodejs_eventloop_monitored	Gauge	1 when the instrumentation is running, 0 when it is disabled or not yet started
breeze_nodejs_eventloop_lag_max_seconds	Gauge	Delay over the last completed sampling interval, including any stall still in flight (instantaneous)
breeze_nodejs_eventloop_lag_window_max_seconds	Gauge	Worst delay retained across the whole sampling window (high-water mark)
breeze_nodejs_eventloop_lag_window_mean_seconds	Gauge	Mean delay across the retained sampling window
breeze_nodejs_eventloop_starved	Gauge	1 while the current lag is at or above EVENT_LOOP_STARVATION_WARN_MS, else 0

Two things to know before writing alert rules against these:

• Alert on breeze_nodejs_eventloop_monitored == 0. When the monitor is off, the other four gauges sit at 0 — which is indistinguishable from a perfectly healthy loop. This series is what separates the two cases.
• _lag_max_seconds is instantaneous, _lag_window_max_seconds is not. Use the instantaneous series (or breeze_nodejs_eventloop_starved) for “is it stalled right now”; the window max stays elevated for the retention window, so a max_over_time on it counts one stall many times.

Admission Metrics

Agent admission control (see Agent admission control) publishes its state and every refusal.

Metric	Type	Description
breeze_admission_mode{mode}	Gauge	1 for the active mode (off, observe, enforce)
breeze_admission_level	Gauge	0 normal, 1 inventory-type uploads refused, 2 status uploads refused too, 3 critical
breeze_admission_decisions_total{class,route,outcome,reason}	Counter	Requests refused (outcome="shed"), or that would have been in observe mode (would_shed). reason is level, bulk_in_flight, ws_bucket or ws_setups
breeze_admission_transitions_total{from,to,trigger}	Counter	Level changes. trigger is lag, db, db_timeout or emergency for escalations; recovery steps carry the last escalation’s trigger, and none once back at level 0. A fail-open reset after repeated tick failures reports tick_failure
breeze_admission_signal_lag_seconds	Gauge	Median event-loop lag the controller used (-1 when the lag monitor is off)
breeze_admission_signal_db_wait_seconds	Gauge	How long requests wait for a database connection (the pool’s own series are breeze_db_pool_admission_*)
breeze_admission_db_saturation_responses_total{error}	Counter	Requests answered 503 because the database pool (pool_acquire_timeout) or the RLS prologue (prologue_timeout) timed out
breeze_admission_ws_bucket_tokens / breeze_admission_ws_setups_in_flight	Gauge	WebSocket pacing state
breeze_admission_bulk_in_flight	Gauge	Inventory-type uploads being processed
breeze_admission_tick_failures_total	Counter	Admission controller ticks that threw. Five in a row fail open: the level is forced to 0 so shedding cannot persist on a stuck controller
breeze_admission_offline_deferred_total{event}	Counter	WebSocket disconnects whose offline flip was left to the offline detector while under load
breeze_nodejs_eventloop_utilization	Gauge	Event-loop utilization over the last second

Alert on breeze_admission_level == 3 for 5 minutes: the instance is down to protected traffic and needs capacity. In observe mode, compare would_shed against incidents before switching to enforce.

Log Aggregation with Loki

Promtail scrapes Docker container logs and ships them to Loki. Loki stores logs for 14 days by default (configurable via retention_period in monitoring/loki-config.yml).

Querying Logs in Grafana

Open the Explore page in Grafana, select the Loki data source, and enter LogQL queries.

Basic Queries

# All API logs
{container="breeze-api"}

# API errors only
{container="breeze-api"} |= "error"

# Structured JSON logs — filter by level
{container="breeze-api"} | json | level = "error"

# Logs from a specific container
{container="breeze-web"}

Filtering and Searching

# Search for a specific device ID
{container="breeze-api"} |= "device_id=abc123"

# Exclude health check noise
{container="breeze-api"} != "/health"

# Filter by HTTP status code in structured logs
{container="breeze-api"} | json | status >= 500

# Rate of errors over time (useful for dashboards)
rate({container="breeze-api"} |= "error" [5m])

Time-Based Queries

# Logs from the last hour containing "timeout"
{container="breeze-api"} |= "timeout"

# Count log lines per minute
sum(rate({container="breeze-api"} [1m])) by (container)

Tip

Use the Live toggle in Grafana Explore to tail logs in real time. This is useful when debugging agent connectivity or watching alert processing.

Adding Custom Prometheus Alert Rules

1. Create a new YAML file in monitoring/rules/ (e.g., monitoring/rules/custom-rules.yml).

2. Define your alert rules following the Prometheus format:

   groups:
     - name: custom-alerts
       rules:
         - alert: HighAgentChurn
           expr: rate(breeze_device_enrollments_total[1h]) > 10
           for: 30m
           labels:
             severity: warning
           annotations:
             summary: "High agent enrollment rate"
             description: "More than 10 new enrollments per hour for 30 minutes"

3. Reload the Prometheus configuration (no restart required):

   curl -X POST http://localhost:9090/-/reload

4. Verify the rule loaded successfully by checking http://localhost:9090/rules in the Prometheus UI.

Data Retention

Component	Default Retention	Configuration
Prometheus	15 days	--storage.tsdb.retention.time=15d in compose file
Loki	14 days (336h)	retention_period in monitoring/loki-config.yml
Grafana	Unlimited (dashboards only)	N/A
Alertmanager	Silences and notification log only	--storage.path in compose file

To change retention, edit the relevant configuration and restart the container.

Troubleshooting

Prometheus Is Not Scraping the API

Symptom: The breeze-api target shows as DOWN in http://localhost:9090/targets.

1. Verify the API is running and healthy: curl http://localhost:3001/health
2. Check the scrape token is correct. Compare monitoring/secrets/metrics_scrape_token with the METRICS_SCRAPE_TOKEN environment variable on the API container.
3. Verify network connectivity. Both Prometheus and the API must be on the same Docker network (breeze).
4. Check Prometheus logs: docker compose -f docker-compose.yml -f docker-compose.monitoring.yml logs prometheus --tail 50

Grafana Shows “No Data”

Symptom: Dashboard panels show “No data” instead of charts.

1. Confirm Prometheus is running and scraping: visit http://localhost:9090/targets and verify all targets are UP.
2. In Grafana, go to Configuration > Data Sources > Prometheus and click Test. It should say “Data source is working.”
3. Check the time range selector in Grafana. If metrics collection just started, narrow the range to “Last 15 minutes.”
4. If using a custom dashboard, verify the metric names match what Prometheus is collecting. Test a simple query like up in Grafana Explore.

Loki Queries Are Slow

Symptom: Log queries in Grafana take more than 10 seconds or time out.

1. Narrow the time range. Loki performs best with shorter ranges (last 1 hour vs. last 7 days).
2. Add label matchers. {container="breeze-api"} |= "error" is much faster than {job="varlogs"} |= "error" because the label narrows the search before the text filter runs.
3. Check Loki’s compactor. If it has fallen behind, compaction can slow queries: docker compose -f docker-compose.yml -f docker-compose.monitoring.yml logs loki --tail 50
4. Increase Loki resources if needed. In the compose file, add memory limits and CPU limits that match your server capacity.

Alertmanager Is Not Sending Notifications

Symptom: Alerts fire in Prometheus but no notifications arrive.

1. Confirm Alertmanager is receiving alerts: visit http://localhost:9093/#/alerts and check for active alerts.
2. If no alerts appear, verify Prometheus is configured to send to Alertmanager. Check alerting.alertmanagers in monitoring/prometheus.yml.
3. If alerts appear but notifications are not sent, check the receiver configuration in monitoring/alertmanager.yml. Look for commented-out sections that need to be enabled.
4. Check Alertmanager logs for delivery errors: docker compose -f docker-compose.yml -f docker-compose.monitoring.yml logs alertmanager --tail 50
5. Verify webhook URLs, API keys, and SMTP credentials are correct. Test Slack webhooks with curl to rule out network issues.

Containers Not Starting

Symptom: One or more monitoring containers fail to start or keep restarting.

1. Check which containers are failing: docker compose -f docker-compose.yml -f docker-compose.monitoring.yml ps
2. Read the logs: docker compose -f docker-compose.yml -f docker-compose.monitoring.yml logs <container> --tail 100
3. Common causes:
   • Grafana: GRAFANA_ADMIN_PASSWORD not set in .env. The compose file requires this variable. (This is separate from .env.prod, which only scripts/prod/deploy.sh’s digest-pinned production flow reads.)
   • Postgres Exporter: POSTGRES_PASSWORD not set or incorrect. The exporter needs the same credentials as the database.
   • Prometheus: Invalid YAML in prometheus.yml or rule files. Validate with promtool check config monitoring/prometheus.yml.
   • Loki: Permissions issue on the data volume. Loki runs as a non-root user and needs write access to /loki.

Disk Space Growing

Monitoring data can accumulate over time, especially on busy systems.

1. Check volume sizes: docker system df -v | grep -E 'prometheus|grafana|loki'
2. Reduce Prometheus retention: lower --storage.tsdb.retention.time from 15d to 7d in the compose file.
3. Reduce Loki retention: lower retention_period in monitoring/loki-config.yml (e.g., from 336h to 168h).
4. Prune old Docker volumes if containers were previously removed without cleaning up: docker volume prune
5. Restart the affected containers after configuration changes.

Hot-path cache

The agent heartbeat and UniFi collector poll serve a few per-org and global values from small in-process caches (agent update config, topology flags, the helper flag, the policy probe, the PAM fallback, the manifest trust keyset and delegations, and UniFi collector absence). When one API instance writes a change, it announces the change over Redis and every other instance drops its entry within about a second. The message carries the cache name and an id (or none, for the whole cache), never a value.

Metric	Type	Description
breeze_hot_path_cache_lookups_total{cache,result}	Counter	Lookups by result: hit, miss, or bypass (a read inside a database transaction, which never uses the cache)
breeze_hot_path_cache_entries{cache}	Gauge	Entries currently held, read at scrape time
breeze_cluster_cache_invalidations_published_total{cache,outcome}	Counter	Invalidations this instance announced. outcome is published, unavailable (no Redis) or failed (error, or no reply within 1 s)
breeze_cluster_cache_invalidations_received_total{cache,result}	Counter	Messages handled: applied, self (this instance’s own echo), unknown_cache, malformed, or error (the cache failed to drop the entry; it keeps it until the TTL)
breeze_cluster_cache_invalidation_lag_seconds{cache}	Histogram	Time from publish to apply for applied messages
breeze_cluster_cache_subscriber_up	Gauge	1 while this instance is subscribed
breeze_cluster_cache_resyncs_total	Counter	Full local cache clears after the subscriber reconnected (messages sent during the gap are lost, so every cache is dropped once)

A healthy instance shows breeze_cluster_cache_subscriber_up 1, a lag p99 well under one second, and a flat published_total{outcome="failed"}. Without Redis the caches still work: each entry simply lives until its TTL (60 seconds, or 5 minutes for collector absence), which was the behaviour before cross-instance invalidation existed. The subscriber waits at most one second at boot and keeps retrying in the background.

Agent capacity KPIs

The breeze-capacity-kpis rule group in monitoring/rules/breeze-rules.yml and the Breeze RMM - Agent Capacity dashboard (monitoring/grafana/dashboards/breeze-agent-capacity.json, uid breeze-agent-capacity) report what an agent costs the API, per region and per API instance. The number of online agents is the count of agent WebSockets owned by each API process.

KPI	Definition	Recording rule	M1 / M2 / M3 target
Agent cost	API process CPU-ms per agent-minute	breeze:agent_cost_cpu_ms_per_agent_minute:region and :instance	60 / 25 / 15
Heartbeat latency	p50, p95, p99 of POST /agents/:id/heartbeat, server histogram	breeze:heartbeat_latency_seconds:p50_region, :p95_region, :p99_region, :p95_instance	none set
DB intensity	Committed transactions per agent-minute	breeze:db_tx_per_agent_minute:region	8 at M1, 4 at M3
Request mix	Agent requests per agent-minute	breeze:agent_requests_per_agent_minute:region	none set
Online agents	Agent sockets owned by API processes	breeze:agent_ws_connections:instance and :region	n/a

Event-loop utilization and lag, pool wait, prologue-deadline expiries and the admission level are on the dashboard’s Saturation row and have no targets.

New series

Metric	Type	Labels	Description
breeze_process_cpu_seconds_total	Counter	mode = user or system	CPU time of the process, read from process.cpuUsage() at scrape time
breeze_agent_ws_connections	Gauge	none	Agent WebSockets owned by this process. The worker role publishes 0

Requirements

• region label. The rules aggregate by (region). Every scrape job on the Prometheus that evaluates them needs a region label, including postgres-exporter: the DB intensity rule joins the exporter’s pg_stat_database_xact_commit to the API socket count on region. deploy/prometheus-remote-scrape.example.yml shows the API jobs; the placeholder values are eu and us.
• Single-host stacks. The dashboard’s region and instance variables default to All (allValue: ".*"), and that matcher also matches an absent label, so a stack without a region label still renders. The per-instance rules also work without it; the region DB rule does not.
• Database name. The DB rule filters datname="breeze", the Compose default. Edit it in breeze-rules.yml if you renamed the database.
• Empty fleet. The rules divide by (... > 0), so zero online agents gives “no data”, not +Inf.

Caveats

• Per-instance cost is approximate with more than one API instance. HTTP requests spread across instances; sockets do not. The region figure is the one to trust.
• Event-loop “lag p99” is approximated. breeze_nodejs_eventloop_lag_max_seconds is a gauge, not a histogram. The Event-loop lag panel plots the raw gauge, max_over_time over 5 minutes and quantile_over_time(0.99) over 15 minutes.
• Two heartbeat p95 figures exist. The dashboard uses the server histogram, which starts at the handler. The perf gate uses the simulator’s client-side p95, which also includes queueing before the handler, so it is higher under load.
• The lab baseline overstates production cost. The perf gate runs one role: all, dev-mode API container that also runs the BullMQ workers, so its CPU-ms per agent-minute is above what the production api role costs. It is a regression reference for that profile, not a production figure.

Perf regression gate

A nightly or manually dispatched workflow measures these KPIs on fixed hardware and fails on a regression of more than 15% against a committed baseline. See load-tests/perf/README.md for what it gates, how to run it and how to read a failure.