Upgrade Guide

Docker Compose Upgrade

1. Back up the database

   ./scripts/backup.sh --db

   Database migrations are forward-only. Rolling the container tag back does not roll the schema back. See Backup & Restore.

2. Diff your Compose files against the tagged release and merge the required changes before pulling images; follow Compose file changes, preserving your local configuration. Back up .env, the active Compose files, and overrides alongside the database backup.

3. Check how your images are pinned

   grep '^BREEZE_API_IMAGE_REF=' .env

   • Digest-pinned — the value ends in @sha256:…. Every install made with guided-setup.sh for v0.112.0 or later looks like this. Follow step 4a.
   • Tag-form — the value ends in :${BREEZE_VERSION} (older installs, and manual installs that never switched). Follow step 4b.

   On a digest-pinned install, `docker compose pull` does not upgrade

   The image refs name exact digests, so docker compose pull fetches the release you already run, and editing BREEZE_VERSION alone leaves the old images running. BREEZE_VERSION also selects the agent release your fleet is offered, so a hand-edited value can move agents ahead of the server.

4. Move to the new release

   4a. Digest-pinned: run guided-setup.sh --upgrade

   # Fetch the current installer. --upgrade is new; an older copy does not have it.
   curl -fsSLO https://raw.githubusercontent.com/lanternops/breeze/main/scripts/guided-setup.sh
   
   # Run from the directory that holds docker-compose.yml and .env, or pass --work-dir.
   bash guided-setup.sh --upgrade 0.118.2   # omit the version to use the latest release

   The script:

   • downloads the target release’s signed image inventory and verifies it against RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS, the same fail-closed check as a fresh install;
   • backs up .env, then rewrites BREEZE_VERSION and the four BREEZE_*_IMAGE_REF digests together. If verification fails, .env is left unchanged and nothing is pulled;
   • prints links to the target release’s docker-compose.yml and .env.example. It never edits your Compose file, so complete step 2 for the target release;
   • pulls the verified images, runs docker compose up -d, and waits for the API to report healthy.

   Add -y to run it without prompts, for example from a script. Add --no-up to update .env only and restart later. The script refuses a target older than your current BREEZE_VERSION. Migrations are forward-only, so restore the matching database backup before you pass --allow-downgrade. Targets older than 0.112.0 are always refused, because release signatures do not cover those images; --allow-unverified-release does not apply to --upgrade. It also re-pins the digests when BREEZE_VERSION already names the target, which repairs an install where only BREEZE_VERSION was edited.

   If you run the manual verifier instead, follow Signed, digest-pinned image references and update BREEZE_VERSION and all four refs together.

   4b. Tag-form: set the version, then pull and restart

   # In .env: set to the version you want (e.g. 0.118.2)
   BREEZE_VERSION=0.118.2

   docker compose pull
   docker compose up -d

   BREEZE_VERSION also selects the agent release your fleet is offered, so set it to a version that is published on GitHub, not one that has only been tagged. See Binary Distribution.

   Either path pulls the new API, web, and binaries images and restarts all services. Database migrations run automatically on API startup when AUTO_MIGRATE=true (the default).

   Expected downtime: Typically 5–15 seconds while containers restart. If the new version includes database migrations, the API container will take additional time to apply them before accepting traffic – usually under 60 seconds for routine schema changes, but potentially several minutes for large data migrations. Agents continue running independently during this window and reconnect automatically.

5. Verify

   # Check API health
   curl -s https://breeze.yourdomain.com/health | jq .version
   
   # Check all running image versions
   docker compose images
   
   # Check agent version (on the device)
   breeze-agent version

   /health reports the version baked into the running API image. If it does not match BREEZE_VERSION in .env, the image was not upgraded. The API logs [version] BREEZE_VERSION=… but this API image is … at startup, and Administration → System → Connections shows the same warning. This requires the Compose change described in APP_VERSION comes from the image.

   Also sign in through your public URL and confirm agents resume checking in. /health bypasses the HTTPS redirect and can stay healthy while a stale proxy address sends other requests into a redirect loop.

Tip

The binaries-init container uses pull_policy: always, so docker compose up -d automatically pulls the latest binaries image and refreshes the volume — no manual steps needed.

Check what an upgrade retires

Every API image carries a list of the API fields and endpoints Breeze has retired, with the version each one was deprecated and removed in and what replaces it. The list is cumulative, so an image knows every retirement you cross even when you skip versions. The API also records each version your deployment boots, so it can tell which retirements are new to you.

At every boot, before migrations run, the API logs a report headed [upgrade-preflight]. It lists each retirement the upgrade crosses, the endpoints and fields affected, and the replacement. Read it after upgrading:

docker compose logs api | grep upgrade-preflight

Before upgrading, you can run the same check with the new image against your current database, after docker compose pull and before docker compose up -d:

docker compose run --rm --no-deps api node dist/scripts/upgrade-preflight.cjs

This only reads. It does not run migrations or record a version. It exits 0. Add --strict (or set BREEZE_UPGRADE_PREFLIGHT_STRICT=true) to exit 1 when a removal is crossed, for use in an upgrade script.

The report never blocks the API from starting. If the deployment has no recorded version history (the first upgrade to a version with this check, or a database the check could not read), the report lists every retirement in effect for the new image and marks it as possibly crossed. It does not report “no issues” in that case.

In the web UI

Platform administrators can see the same information at Administration → System (/admin/system, v0.117.0+). The Deprecations tab lists every retirement the running image knows about with its status for this deployment (crossed, possibly crossed, in effect, or upcoming), along with the running version, the last version recorded, the applied and pending migration counts, and the deployment’s version history. The Connections tab shows each integration the API container is configured for, grouped by area, as enabled, disabled, misconfigured, or required but missing. It reads the API container’s environment only, never shows secret values, and does not contact the services themselves, so “enabled” means configured, not reachable. Both tabs are read-only.

Compose file changes

docker compose pull updates images; it does not download a new docker-compose.yml, override file, or .env.example. Current images can therefore run with old network settings or missing environment mappings. Review the configuration shipped with every target release, including when skipping versions; the entries below highlight changes that need operator action, not a complete Compose changelog.

Fetch a separate copy of the target release’s Compose file and compare it with the one you actually deploy. Replace the placeholder with the published release tag matching your intended BREEZE_VERSION:

release_tag="v<target-version>"
release_compose=$(mktemp)
curl --fail --show-error --location \
  "https://raw.githubusercontent.com/LanternOps/breeze/${release_tag}/docker-compose.yml" \
  --output "$release_compose" && diff -u docker-compose.yml "$release_compose"

diff exits with status 1 when differences exist; a failed download must be resolved before continuing. If you deploy deploy/docker-compose.prod.yml or additional override files, compare those same paths at the same tag too. Review the tagged .env.example and any configuration files referenced by changed services.

Merge the required changes into your backed-up local files, preserving custom services, domains, mounts, secrets, and proxy topology. Do not overwrite .env with .env.example. Add required variables to both .env and the appropriate service’s environment: mapping; setting .env alone does not pass a variable to the container. For digest-pinned deployments, update the image references to the target release as well as BREEZE_VERSION. guided-setup.sh --upgrade does this for you (step 4a).

Validate the merged configuration before the pull/restart step:

docker compose config --quiet

Use the same -f options and profiles you normally deploy for validation, pull, and restart so your overrides are included. Validation checks the Compose configuration; it does not prove that a trusted proxy address matches the running proxy.

v0.121.0 — new optional AI and enrollment settings; production Compose catches up

The v0.121.0 Compose files change the api environment:

Change	Required action
AGENT_ENROLL_RATE_LIMIT, AGENT_ENROLL_RATE_WINDOW_SECONDS, AI_PLATFORM_INFERENCE_GEO and AI_GATEWAY_HEADERS_TIMEOUT_SECONDS are mapped in both docker-compose.yml and deploy/docker-compose.prod.yml. All are optional; empty keeps the defaults.	Copy the lines into your own Compose file if you want to set any of them. See Environment Variables.
deploy/docker-compose.prod.yml passes 82 more variables to the API, the same lines the root docker-compose.yml already carries (for example the MCP OAuth, Mailgun, Twilio, SMTP, M365, Cloudflare and Redis memory monitor settings, SESSION_SECRET, JWT_SIGNING_KEYRING and SENTRY_TRACES_SAMPLE_RATE).	If you deploy that file, a value in your .env that it used to ignore now takes effect. Check .env for stale entries before upgrading. Installs on the root docker-compose.yml (including guided-setup.sh) are unaffected.
The stock Compose file does not pass MCP_LLM_* to the API.	If you use MCP_LLM_PROVIDER=openai-compatible, keep your own mapping and read the v0.121.0 notes first.

v0.120.0 — start fence default and newly mapped API settings

The v0.120.0 Compose files change the api environment in both docker-compose.yml and deploy/docker-compose.prod.yml:

Change	Required action
REMOTE_DESKTOP_FENCE_REQUIRED defaults to true (was false).	See v0.120.0 — remote desktop start fence. Until you take this change, an unset variable keeps the fence off.
AUDIT_ANCHOR_SIGNING_KEY, AUDIT_CHAIN_ANCHOR_ENABLED, AUDIT_CHAIN_VERIFY_ENABLED, AUDIT_CHAIN_VERIFY_MODE, AUDIT_CHAIN_VERIFY_RESCAN_SLICES and AUDIT_ADMIN_DATABASE_URL are now passed to the API. Before this release Compose installs ignored them even when they were set in .env.	Copy the entries into your own Compose file if you set any of them. All are optional; leaving them empty keeps today’s behaviour. See Environment Variables.
AI_INVOCATIONS_RETENTION_DAYS, AI_INVOCATIONS_RETENTION_BATCH_SIZE and AI_INVOCATIONS_RETENTION_MAX_BATCHES are mapped (optional; retention defaults to 400 days). deploy/docker-compose.prod.yml also gains the three DEVICE_COMMAND_QUEUE_*_TTL_HOURS deadlines.	Copy them if you maintain your own Compose file and want to tune them.

v0.119.0 — APP_VERSION comes from the image

Earlier Compose files set APP_VERSION: ${BREEZE_VERSION...} in the API environment. That value replaced the version baked into the image, so /health, the upgrade preflight, and the recorded version history reported whatever .env said, even when the running image was older. The shipped docker-compose.yml and deploy/docker-compose.prod.yml no longer set it.

Change	Required action
The APP_VERSION: line is removed from the shared API environment block (x-api-env in docker-compose.yml, the API environment in deploy/docker-compose.prod.yml).	Delete the APP_VERSION: line from your own Compose file. Keep BREEZE_VERSION:, which still selects the agent release. Until you remove the line, /health keeps reporting BREEZE_VERSION and the version-mismatch warning cannot fire.

v0.119.0 — coturn relays again

Change	Required action
The v0.118.0 coturn configuration listed denied-peer-ip=::. coturn reads an all-zero address as “no bound”, so that one entry refused every peer and relayed (TURN) connections could not be set up. The entry is removed from docker/turnserver.conf, docker-compose.yml and deploy/docker-compose.prod.yml; coturn already refuses the unspecified address on its own.	Delete the denied-peer-ip=:: line ('--denied-peer-ip=::' in a Compose file) from your coturn configuration and restart coturn, even if you skip this release. Do not replace it with 0.0.0.0 or a range starting at ::.

v0.119.0 — coturn refuses IPv4-mapped and NAT64 peers

Change	Required action
The two denied-peer-ip lines that v0.118.0 shipped commented out are now active in docker/turnserver.conf, docker-compose.yml and deploy/docker-compose.prod.yml: ::ffff:0:0-::ffff:ffff:ffff (IPv4-mapped IPv6) and 64:ff9b::-64:ff9b::ffff:ffff (NAT64 well-known prefix). Relay-to-relay sessions are unaffected.	Nothing if you use the bundled coturn with the shipped files. If you run your own coturn, add both lines from TURN Server and restart it. Never deny the relay’s own public IP.

v0.118.0 — coturn peer restrictions and API stop grace

The v0.118.0 Compose files and docker/turnserver.conf change two things:

Change	Required action
coturn relays only to public addresses: denied-peer-ip rules for loopback, private, CGNAT, link-local, documentation and reserved ranges (IPv4 and IPv6), plus no-tcp-relay. verbose is turned off.	Nothing if you use the bundled coturn with the shipped files. If you run your own coturn, apply the same rules from TURN Server. Never deny the relay’s own public IP: relay-to-relay traffic targets it. The IPv4-mapped and NAT64 lines ship commented out; enable them only after checking relay-to-relay on your deployment.
The api service (and worker) has stop_grace_period: 90s, so background work in the process can finish during a restart.	Copy it into your own Compose file if you maintain one.

v0.95.0 — stable Caddy address and proxy trust

The v0.95.0 Compose file introduced these coupled settings. If your deployed file predates them, apply all three even if your images are already newer:

Compose setting	Required action
networks.breeze.ipam.config.subnet uses BREEZE_DOCKER_SUBNET	Merge the IPAM configuration. If the shipped subnet overlaps another Docker network, host route, or VPN, choose a non-overlapping subnet in .env.
services.caddy.networks.breeze.ipv4_address uses BREEZE_CADDY_IP	Merge the static address pin. If you customize the subnet, choose an unused Caddy address inside it and set BREEZE_CADDY_IP alongside BREEZE_DOCKER_SUBNET.
services.api.environment.TRUSTED_PROXY_CIDRS defaults to BREEZE_CADDY_IP with a /32 mask	Merge the derived default. An explicit TRUSTED_PROXY_CIDRS in .env or an override takes precedence: remove a stale override when using bundled Caddy, or set it to the actual immediate proxy’s stable address for a custom topology.

Keep TRUST_PROXY_HEADERS=true when using the bundled proxy. Do not widen trust to an entire private network to work around an address mismatch. A proxy’s dynamically assigned address can move after a container recreation or host reboot.

Changing an existing network’s IPAM/subnet requires recreating that network during a maintenance window; docker compose restart cannot apply it. For a standalone stack whose network is used only by that stack, stop/remove its containers and network with docker compose down, then recreate them with docker compose up -d after pulling the target images. Do not use down -v, which deletes named data volumes. If other containers share the network, coordinate their migration before recreating it. This network migration takes longer than the routine image-only restart described above.

Before upgrading a proxy-fronted deployment to v0.103.0 or later, verify that TRUSTED_PROXY_CIDRS matches the proxy that directly connects to the API. That release’s TRANSPORT-001 enforcement only honors forwarded HTTPS headers from trusted peers; a stale address can cause repeated 308 redirects for the UI and agents while /health still returns 200. After restarting, check public sign-in and agent check-ins as well as API logs for [proxy-trust] or [force-https] warnings (available from v0.104.0).

Version-specific notes

v0.122.0 — legacy AI model settings removed

The AI model registry (AI Providers & Models) has been the only source of AI model decisions since v0.121.0. This release deletes what it replaced.

• /api/v1/ai/provider is gone. Use /api/v1/ai/models: GET /ai/models, POST /ai/models/connections, POST /ai/models/connections/:id/key, POST /ai/models/connections/:id/endpoint, DELETE /ai/models/connections/:id, and PUT /ai/models/assignments for per-feature defaults. Requests to the old paths return 404.
• Retired request fields are rejected with HTTP 400 that names the field and its replacement: reviewerModel on both script-policy endpoints, allowedModels on the AI for Office organization policy, and model on AI agent create/update/preview (send offeringId). They were already ignored; a client that kept sending them now gets an error instead of a silent no-op. The GET responses no longer include them.
• BREEZE_AI_SCRIPT_REVIEWER_MODEL and WORKSPACE_CONTENT_LLM_MODEL are no longer read. Remove them from your .env. New partners start every feature on the platform default model (ANTHROPIC_MODEL, else the default set on Admin → AI models); change any feature under AI Providers & Models.
• Skipping releases: the one-time conversion of legacy AI settings into the registry ran only in v0.121.0. If you upgrade straight from an older release, each partner is set up from the platform default instead; a partner that had its own Anthropic key keeps using that key, but its legacy per-feature choices (script reviewer, AI for Office, AI agent allowlists) are not carried over. Set them again under AI Providers & Models. The next release archives the old values before it drops them (see the next section when it ships).
• Database: no schema change in this release. The legacy table and columns are dropped in the following release, after their values are archived.

v0.121.0 — AI model registry, BYO OpenAI-compatible models, encrypted backup credentials

Roll forward only

Backup destination credentials are encrypted at rest on the first start of v0.121.0. After the API logs [startup] Backup destination credentials sealed: <sealed>/<scanned> config(s) …, do not roll back to an earlier release: older code would hand the encrypted values to agents as storage credentials and S3 backups would fail. Take your usual database backup before upgrading, and fix forward if something goes wrong.

Security. v0.121.0 fixes GHSA-vjqq-fmr4-m576 (High: a user who can configure a webhook, alert channel or image-from-URL field, or whoever runs the URL it points at, could repeatedly stop the API process) and GHSA-7xq8-mcq6-j438 (Low: a technician limited to selected organizations could read, edit or delete ticket comments in the partner’s other organizations). Upgrade from 0.120.0 or earlier.

Don’t skip this release on the way to v0.122.0. The one-time move of legacy AI settings (script reviewer, AI for Office and AI agent model choices) into the AI model registry runs in v0.121.0. An install that jumps straight to v0.122.0 starts each partner from the platform default and loses those choices.

Upgrade order.

1. Back up the database.
2. If you use MCP_LLM_PROVIDER=openai-compatible, prepare the changes below (explicit IS_HOSTED=false, no localhost URL, a tool-capable model).
3. Upgrade the server: bash guided-setup.sh --upgrade 0.121.0 on a digest-pinned install, or set BREEZE_VERSION=0.121.0 and run docker compose pull api web portal && docker compose up -d.
4. Merge the Compose file changes if you maintain your own Compose file.
5. Let agents and Breeze Assist update to 0.121.0.

MCP_LLM_PROVIDER=openai-compatible installs

The instance-wide OpenAI-compatible chat path becomes a connection in the AI model registry. See Bring Your Own LLM Key.

• Set IS_HOSTED=false explicitly. If IS_HOSTED is unset, empty or not a recognised self-host value, the API refuses to start with MCP_LLM_PROVIDER=openai-compatible is for self-hosted Breeze (set IS_HOSTED explicitly to false)…. The stock docker-compose.yml passes IS_HOSTED: ${IS_HOSTED:-false}, so a Compose install gets false unless it overrides it; systemd, Kubernetes and bare Node installs must set it.
• At boot, each partner gets one read-only, env-managed connection and a priced, enabled model (prices from MCP_LLM_PRICE_INPUT_PER_M_USD / MCP_LLM_PRICE_OUTPUT_PER_M_USD), verified once. Chat is pointed at it once, only where chat used the platform default. Restarts and multiple replicas don’t create duplicates. Partners created later get their connection within 10 minutes.
• The model must support tool calling to serve chat. A model that fails tool-use verification can’t serve chat; chat says to choose a tool-capable model.
• Loopback, link-local and cloud-metadata URLs are refused, so http://localhost:11434 (the usual Ollama URL) no longer works. Use the Docker service name or the host’s LAN address; private-network addresses over http:// are allowed on self-host. The API still starts, bootstraps nothing, logs [envOpenAiBootstrap] MCP_LLM_BASE_URL refused: … and retries at 1, 5, 15 and 60 minutes, then every 10 minutes.
• MCP_LLM_API_KEY is optional but must be at least 8 characters when set.
• Removing the variables keeps the connection active and read-only; you can disconnect it in the UI. Delete the MCP_LLM_BASE_URL line rather than blanking it: an empty value fails config validation.
• Model server sizing. Allow at least 70k tokens of context. The gateway waits up to 120 seconds for a self-hosted model server to start responding (30 seconds on hosted); raise AI_GATEWAY_HEADERS_TIMEOUT_SECONDS (5–600) for slow or CPU-only servers. When it runs out, the request returns 504 naming the setting and the API logs [modelGateway] upstream sent no response headers within the <N> ms response-header deadline.

AI model settings move to AI Providers & Models

• Model choices from the Office policy editor, the script-reviewer model fields and the partner AI provider default are migrated once, on first start, into Partner Settings → AI Providers & Models. The API logs [startup] AI model registry cutover sweep: … N partner(s) cut over, 0 failed.
• BREEZE_AI_SCRIPT_REVIEWER_MODEL and WORKSPACE_CONTENT_LLM_MODEL are no longer read at runtime; their values are migrated at that first start. ANTHROPIC_MODEL remains the bootstrap default.
• Deprecated, removed in v0.122.0: /api/v1/ai/provider and the reviewerModel, allowedModels and AI agent model request fields. In v0.121.0 they still work and the retired fields are ignored. See the API reference for the replacements.
• Residency switch. Turning on the residency requirement disables every AI feature whose model can’t guarantee it; BYO OpenAI-compatible models are never residency-eligible. AI_PLATFORM_INFERENCE_GEO (new, optional) accepts us or global; any other value takes platform models offline, and the API warns at boot.
• AI chargeback is off until you configure it. Every billing profile starts as not billed, and usage from before this release is never billable. A daily job (05:28 UTC) closes the previous UTC month per organization. Chargeable usage is kept at least 156 days regardless of AI_INVOCATIONS_RETENTION_DAYS. Installs without a billing service never mark AI usage for credit debit. See AI Usage Chargeback.

Other behaviour changes

• API keys and sign-out. A sign-out, role change, email change or organization merge no longer stops a key minted by a person; a password change or reset, invite acceptance, an admin status change or an MFA factor change still does. Keys that stopped working after a sign-out on v0.120.0 are not revived: mint a replacement. See API Keys.
• Duplicate built-in alert rules are merged once at upgrade. The oldest copy is kept; if any copy was switched off, the merged rule is off. Open alerts from removed copies move to the kept rule, and where two were open for the same device the extra one is resolved as source_retired. The API logs #7650: … warnings with the counts.
• Metric history. On the first daily metric-rollup maintenance run, rows in the catch-all metric_rollups_default partition move into monthly partitions (rows past your 5-minute retention are discarded, as before). Until it finishes, those older points are missing from charts. Look for [MetricRollupMaintenance] … drained metric_rollups_default: moved=N discarded=M.
• Quick Support always shows the viewing indicator, whatever the device’s remote access indicator setting, and AI screen tools are refused on Quick Support devices. See Remote Access.
• Agent enrollment rate limit is configurable with AGENT_ENROLL_RATE_LIMIT (default 10) per AGENT_ENROLL_RATE_WINDOW_SECONDS (default 60), per source IP. Raise it for mass deployments behind one NAT address. A non-integer or non-positive value stops the API from starting.
• Breeze Viewer on Windows. Viewers on 0.118–0.120 whose in-app update kept failing pick up 0.121.0 on their next launch; no reinstall is needed. A failed update now names the stage and the log path (%LOCALAPPDATA%\com.breeze.viewer\logs\updater.log). As a fallback, download breeze-viewer-windows.msi from the release and run it once.

Environment variables

No new required variables, except IS_HOSTED=false on MCP_LLM_PROVIDER=openai-compatible installs (above). New optional: AI_PLATFORM_INFERENCE_GEO, AI_GATEWAY_HEADERS_TIMEOUT_SECONDS, AGENT_ENROLL_RATE_LIMIT and AGENT_ENROLL_RATE_WINDOW_SECONDS, all mapped in both Compose files. Deprecated (no longer read): BREEZE_AI_SCRIPT_REVIEWER_MODEL, WORKSPACE_CONTENT_LLM_MODEL.

Migrations

45 idempotent migrations apply on boot ([auto-migrate] Applied 45 migration(s)). Some are named earlier than v0.120.0’s newest; they still apply, in order. Most are additive: tables for the AI registry cutover and AI chargeback, autopay foundation tables (off for every partner), new columns on AI, billing profile, user and API key tables, a unique index on built-in alert rules, the Partner API ticket groundwork (ticket_external_refs, and tickets:read / tickets:write as allowed service-principal scopes that have no effect yet), and a topology index.

• Three indexes are built with CREATE INDEX CONCURRENTLY, outside a transaction, so writes continue. If the ai_budget_reservations build is interrupted, the next start refuses with an “INVALID index” message: run DROP INDEX CONCURRENTLY ai_budget_reservations_session_turn_idx; and restart. The other two have no such guard, so check them after the upgrade:

  SELECT c.relname, i.indisvalid FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
   WHERE c.relname IN ('ai_invocations_chargeable_idx', 'idx_device_commands_install_patches_device_created');

  If either is false, drop it with DROP INDEX CONCURRENTLY <name>; and re-run the CREATE INDEX CONCURRENTLY from its migration (2026-11-26-100150-ai-invocations-chargeable-idx.sql or 2026-11-19-101100-device-commands-install-patches-index.sql).
• Brief locks. A few constraint and index steps on AI session, budget and egress tables scan while holding a write lock (well under a second at 300,000 rows per table in testing; expect a few seconds of paused AI chat writes on very large tables). No migration sets a lock timeout, so a long idle-in-transaction session on these tables delays boot.
• Device mTLS certificate repair. One migration records every device’s issued client certificate in the certificate history. It holds only a read lock on devices, so agents keep checking in, but it runs inside the boot and can add seconds to minutes on a large fleet. It always logs WARNING device mTLS history reconcile: demoted <D> stale active rows, imported <I> issued certificates, skipped <S> devices…. Demoted certificates are revoked at Cloudflare in the background, 200 every 5 minutes.

v0.119.0 — network topology, time sync, Xero, S3 restores through storage sessions

Before you upgrade: coturn and HTTPS for S3 restores

The v0.118.x Compose files pass --denied-peer-ip=:: to coturn, which refuses every relayed (TURN) connection. Remove that line on every install, even if you skip v0.119.0; see Compose file changes. S3 restores, test restores and verification now go only through a short-lived storage session, with no fallback: they need HTTPS end to end (MinIO endpoint and agent API), PUBLIC_API_URL set and matching, and agents on v0.118.0 or later. See HTTPS for Restores.

Security. v0.119.0 fixes GHSA-9qpw-p6rx-5f2h (High, agent: restoring a Windows backup as a Hyper-V VM could run code from that backup on the host), GHSA-c445-q84m-rph4 (Medium, server on a non-UTC host time zone) and GHSA-vjq5-x2g3-pjhq (Low, AI file tools). Upgrade the server and let agents and Breeze Assist update to 0.119.0.

Upgrade command. On a digest-pinned install (every guided-setup.sh install from v0.112.0), fetch the current guided-setup.sh from main and run bash guided-setup.sh --upgrade 0.119.0; it verifies the signed image inventory and rewrites BREEZE_VERSION and all four image digests together. Do not upgrade by only editing BREEZE_VERSION on a digest-pinned install: that pulls the images you already run. Delete the APP_VERSION: line from your own Compose file (details). Coming from v0.117.x or earlier, read the v0.118.0 notes first.

Behaviour changes

• Live device inspection needs devices:execute. Process, service, scheduled-task, event-log, file and registry reads, and the live session list, need it (/devices/:id/sessions/live also accepts remote:access). Org Viewer, Partner Viewer and the two seeded approver roles lose live inspection; custom roles that should keep it need devices:execute. API keys with only ai:read can no longer trigger live reads.
• Site scoping. Fleet security posture, threats, the executive summary and OS distribution respect site-restricted access.
• Remote consent. On devices whose policy is Require consent, VNC is refused (409 CONSENT_REQUIRED_TRANSPORT_UNAVAILABLE) and desktop starts need a consent-capable agent (409 CONSENT_UPGRADE_REQUIRED); update those agents before the server. AI screen tools and Diagnose are refused on consent-mode devices. Audit actions are renamed: session_consent_denied → session_consent_blocked_unanswered, session_consent_bypassed → session_consent_blocked_unavailable. Update saved audit filters and SIEM rules; older rows keep the old names.
• Approver assurance becomes the default for high- and critical-risk approvals from 2026-11-05 for partners that never saved an Approval Security choice. Move the date with APPROVER_ASSURANCE_DEFAULT_ENFORCE_FROM.
• Settings secrets are encrypted and masked. Notification secrets in partner, organization and site settings are encrypted by a background sweep after first boot (using APP_ENCRYPTION_KEY) and returned as ********; integrations that GET then PUT settings must send the marker back unchanged. If you run a separate worker container, roll it together with api.
• Billing contact. Invoices and quotes go to the organization contact with the Billing role, with no fallback to the primary contact. Fix affected organizations in Contacts.
• Compliance alerts stay open until the device is compliant. Duplicate rule set names within one policy are refused.
• Built-in time-sync monitors are provisioned for every partner but not attached, so nothing alerts until you attach them. Opt out with BREEZE_BUILTIN_MONITORS_AUTOSEED=false.
• AI not configured. Without a usable model provider, chat and the script builder answer 503 ai_not_configured.

Environment variables

No new required variables. New optional: XERO_CLIENT_ID, XERO_CLIENT_SECRET, XERO_REDIRECT_URI, XERO_WEBHOOK_KEY, XERO_DAILY_CALL_LIMIT (Xero stays hidden until configured), AI_TOOL_SEARCH (auto, on, off), APPROVER_ASSURANCE_DEFAULT_ENFORCE_FROM, PRE_ASSIGNMENT_ENROLLMENT_ENABLED (default false) and METRIC_ANOMALY_EPISODE_MIN_BUCKETS. METRIC_ROLLUP_MAX_DELETE_BATCHES is no longer read. Never set BREEZE_BINARIES_VERSION; it is baked into the image. See Environment Variables.

Migrations

50 idempotent migrations apply on boot, all before the API serves. Take a database backup first.

• Backup history backfill. One migration writes a reservation row per existing backup snapshot (about 13 seconds per 300,000 snapshots in testing). Check SELECT count(*) FROM backup_snapshots; beforehand if your backup history is large.
• notification_channels.config is dropped. On a v0.118.x image, alert delivery keeps working but creating or editing a notification channel fails. Do not roll back below v0.118.0.
• Metric history storage. Older months are rewritten by the nightly maintenance job once their 5-minute data has expired. Set METRIC_ROLLUP_5M_RETENTION_DAYS=30 (in .env and the api service’s environment: block) to reclaim space sooner. If the migration role can’t create the metric_rollups_staging schema, create it by hand.

v0.120.0 — S3 backups through storage sessions, restore integrity, consent prompt protocol 2

Before you upgrade: agents and HTTPS for S3 backups

Backups to S3 and S3-compatible storage are now written only through a short-lived, write-scoped storage session, with no fallback. Before upgrading, update every device that backs up to S3 to agent v0.119.0 or later, switch MinIO and other S3-compatible endpoints to https:// (same host and port), and make sure agents reach the API over HTTPS with PUBLIC_API_URL set. Local and NAS destinations are unchanged. See HTTPS for Backups.

Upgrade order.

1. Update every device that backs up to S3 to agent v0.119.0 or later, and move storage endpoints and the agent API to HTTPS (above).
2. Back up the database.
3. Upgrade the server: bash guided-setup.sh --upgrade 0.120.0 on a digest-pinned install, or set BREEZE_VERSION=0.120.0 and run docker compose pull api web portal && docker compose up -d.
4. Merge the Compose file changes if you maintain your own Compose file.
5. Let agents and Breeze Assist update to 0.120.0. Server first: a server older than this release refuses consent- and notify-mode desktop starts on updated agents.
6. Open each Backup policy’s Destination section and work through the storage key card: replace each listed key with your storage provider, disable the old one, then use Check old key or I disabled this key.

Backups and restores

• Who is refused for S3 backups. A device whose backup component has reported it cannot write through a storage session (409 helper_update_required on the on-demand SQL Server and Hyper-V routes; scheduled jobs fail with “Update the Breeze agent on this device, then try again.”), an http:// storage endpoint, a plain-http:// agent API or a missing PUBLIC_API_URL, and providers other than S3 and local. Devices that have not reported yet are held until their first heartbeat. The manage_backup_configs AI tool accepts only the s3 and local providers.
• Restores are checked against the snapshot’s integrity record. Every restore, verify, test restore, VM restore and bare-metal recovery carries the snapshot’s integrity expectations, and the updated backup helper checks the manifest and each file before installing it. A snapshot that failed its integrity check is refused; a snapshot without an attestation still restores and is labelled unattested. See Restoring from Backup.
• Windows restores are stricter. Reserved Windows names (CON, NUL, COM1 and the like), and path components containing : or ending in . or a space, are refused. A restored security descriptor naming accounts the machine doesn’t recognise is replaced with a restrictive one (owner Administrators; SYSTEM and Administrators only), and the result lists those paths. bmr_recover on Windows restores files only; system state is applied by a bare-metal rebuild.
• DR plans. A Bare metal rebuild step now waits up to 24 hours by default (was 4 hours).

Remote desktop start fence required by default

• REMOTE_DESKTOP_FENCE_REQUIRED now defaults to true. Remote desktop (WebRTC and the WebSocket fallback) starts only on agents that report the session start fence, which agents v0.114.0 and later do. On an older agent the technician sees “Remote desktop needs an agent update on this device (session revocation lease and start fence support)…” (HTTP 503 agent_upgrade_required, or the WebSocket error AGENT_UPGRADE_REQUIRED) and no session starts. Terminal, Files and every other remote tool are unaffected. Agents update themselves; if you pin agent versions below v0.114.0, move the pin, or set REMOTE_DESKTOP_FENCE_REQUIRED=false until they have updated. If your .env still carries REMOTE_DESKTOP_FENCE_REQUIRED=false copied from an older .env.example, that line keeps the fence off; remove it. A Compose file from an earlier release also passes false when the variable is unset, so an existing install keeps the fence off until you either set REMOTE_DESKTOP_FENCE_REQUIRED=true in .env or take the v0.120.0 Compose change. New installs get it on. A value that is not a recognized boolean (true/false, 1/0, yes/no, on/off) now refuses boot.
• The WebSocket desktop fallback now behaves like WebRTC. The technician’s view stays on connecting until the agent reports the stream started, so a consent prompt is answered before anything is shown; a denial ends the session as denied with its reason. End stops the stream immediately. Updated agents also show the notify notice and on-screen indicator on this transport, keep the session lease (a Redis or API outage longer than about 90 seconds ends the stream, as it already does on WebRTC), and refuse a start that arrives after the session was stopped.

Consent prompts report whether they were shown

Only devices whose remote access policy uses the consent prompt are affected.

• A signed-in user who cannot be shown the prompt is always refused, whatever “If no one can respond” says (for example, Breeze Assist not running or a native dialog failing). That setting still applies when nobody is signed in, and when a prompt the user could see went unanswered.
• On macOS and Linux an unanswered prompt is refused, because only Windows reports the lock state reliably.
• Update Breeze Assist with the agent. An older Assist still handles Allow and Deny, but its unanswered prompts are refused.
• The technician sees a specific message for each outcome, and consent audit events gain consentOutcome, consentOccupancy and consentProtocol, plus the reasons no_user_session and helper_unreachable. See Remote Access.

Breeze Assist on Windows

• Console session only. The agent delivers Assist’s credential only over its local channel to the active console session and no longer stores it in agent.yaml. Assist in an RDP or other non-console session shows that it has not received its credentials until a later release adds support.
• On first start the agent moves any credential it finds in agent.yaml into its protected secrets file and rotates it once. macOS and Linux are unaffected.

AI

• The default model is claude-sonnet-5-5 with adaptive thinking. Existing chats keep their stored model, and BREEZE_AI_SCRIPT_REVIEWER_MODEL follows the new default. If AI runs through a LiteLLM or other gateway alias, add an alias for claude-sonnet-5-5 and let the gateway accept or drop the thinking parameters (LiteLLM drop_params), or set ANTHROPIC_MODEL to your backend’s model id. See Environment Variables.
• POST /ai/sessions rejects a model that isn’t offered (400 invalid_model).
• AI model registry. Platform admins get Admin → AI models (/admin/ai-models). A daily discovery job lists new Anthropic models for review; it never enables, prices or removes anything, and it is skipped when ANTHROPIC_BASE_URL points at a gateway. Routing, pricing and billing still come from your existing settings in this release.

Other behaviour changes

• API error codes. Five codes from device and agent routes now use the standard uppercase form: SCHEME_NOT_ALLOWED, INVALID_CUSTOM_FIELD_VALUE (was invalid-custom-field-value), RACE_LOST, RE_ENROLLMENT_REQUIRED and SOFTWARE_INVENTORY_LOCK_TIMEOUT. HTTP statuses and messages are unchanged; update integrations that match on the old codes.
• S3-backed binary hosting. Backup, watchdog, user-helper and recovery-media downloads are served from the agent/ prefix, and the Breeze Assist installer is synced to helper/ at boot. The backup/, watchdog/, user-helper/ and recovery-iso/ prefixes are no longer read and can be deleted. An unset HELPER_BINARY_DIR now means the agent binaries directory. See Binary Distribution.
• Device restores count against the device limit, like enrollment.

Environment variables

No new required variables. REMOTE_DESKTOP_FENCE_REQUIRED now defaults to true (see above). New optional: AI_INVOCATIONS_RETENTION_DAYS (default 400), AI_INVOCATIONS_RETENTION_BATCH_SIZE and AI_INVOCATIONS_RETENTION_MAX_BATCHES. The audit-chain settings are now passed to the API by Compose; see Compose file changes.

Migrations

Twelve idempotent migrations apply on boot. They add tables for portal session revocations, backup storage key history and the AI model registry, seed the platform model catalog, and copy each partner’s AI provider setup into the registry. No large rewrites.

• ai_sessions index. Built with CREATE INDEX CONCURRENTLY outside a transaction. If the build is interrupted, the next start refuses with “ai_sessions offering index build left INVALID index”. Run DROP INDEX CONCURRENTLY public.ai_sessions_offering_idx; and restart the API.
• First start. The API records the storage key each S3 destination uses and builds the AI registry from your current AI settings, logging one [startup] line for each.

v0.118.0 — legacy alerting retired, network checks become monitors

Before you upgrade: convert everything on v0.117.0

While you are still on v0.117.0, open Alerts → Monitors → Needs conversion, run Convert everything, and review every refusal. After this upgrade, Undo in conversion history is no longer available for legacy alert rules, templates, inline policy rules or service/process watches, so fix anything you want to keep while you can still roll a conversion back. Then take a database backup.

Upgrade order.

1. On v0.117.0, run Convert everything and review the refusals (above).
2. Back up the database.
3. Set BREEZE_VERSION to the new release in .env, then run docker compose pull api web portal && docker compose up -d.
4. Merge the Compose file changes if you maintain your own Compose file or coturn configuration.
5. Let every agent and Breeze Helper update to 0.118.0, then rebuild any recovery media you keep.

Update every agent and Helper. Many changes in this release take effect on a device only once it runs the 0.118 agent and Helper. Until then, the device keeps its previous behaviour. This covers installer secret handling, Windows local file and pipe checks, PAM file-hash pinning, the new restore path, the Hyper-V, restore-as-VM and instant-boot fixes, consent-prompt reporting, memory module inventory and the macOS binary move. A few paths refuse older agents rather than fall back; they are listed below.

Alerting and network checks

• Legacy alerting is retired. The policy Alerts and Service & Process Monitoring tabs are gone, and /alerts/rules redirects to Alerts → Monitors. Writes under /alerts/rules* and /alert-templates*, and writes to the retired alert_rule and monitoring policy feature types, return 410 Gone. Reads still work, so history stays available.
• First boot converts what is left. The API converts any remaining legacy sources for each partner and retires the ones it cannot convert, with a reason for each. A dismissible banner under Alerts → Monitors lists them. Recreate what you need as monitors before you dismiss it. Open alerts from retired sources stay open until someone resolves them. Set BREEZE_LEGACY_ALERTING_SWEEP=false to skip the conversion at boot; the API still counts what is left.
• Leftovers are logged on every boot. If any legacy policy alert rules or service/process watches are still unretired after the sweep, the API logs the counts at error level on every start. Nothing evaluates them any more. Restart the API to run the sweep again (after removing BREEZE_LEGACY_ALERTING_SWEEP=false if you set it), or run Convert everything.
• Review alert routing before you upgrade. Queued legacy alerts and unconverted standalone rules no longer use their old per-rule delivery settings. Delivery follows the monitor’s settings, your routing rows and the explicit Everything else row. An alert that matches no destination sends no notification.
• Network checks move to Monitors. You now create and edit network checks as network_check monitors under Alerts → Monitors. Network → Results is read-only and offers conversion of existing checks. Checks you have not converted keep polling and alerting. Creating or editing a check through the old endpoints returns 410. An asset cannot be deleted while it still has active or retained checks (409). The boot sweep never converts or retires network checks. See Network Monitors.
• Maintenance windows suppress monitor alerts again. On v0.117.0, Suppress alerts in a maintenance window did not stop monitor alerts, including alerts from rules converted to monitors. Expect fewer alerts during maintenance than on v0.117.0.
• Some monitor thresholds now fire. Bandwidth, disk I/O, network-error and patch-compliance monitors now evaluate in the units they display. Bandwidth thresholds had been compared about 8× too high, so a bandwidth monitor that never fired may fire now.

Dynamic groups as execution targets

Dynamic groups whose rules match on a field the device reports about itself no longer act as execution targets. These fields are hostname, display name, tags, device role, last user, agent version, Helper version, installed software (installed or not installed), and any custom field.

• Affected: scripts, automations, software policies and update rings. The automation, software, remote-access, PAM, maintenance, OneDrive and Assist policy features also skip such groups.
• Not affected: patch and backup assignments still resolve through these groups. So does every non-execution use of a dynamic group, such as views, filters and reports.
• Visible in the audit log. Each skipped group is logged with the fields that caused it. Where the request is interactive, the response also carries a warning.

To keep targeting these devices, rebuild the group on fields Breeze accepts for targeting, such as site, organization, OS, hardware, network facts or metrics. Or use a static group. See Device Groups.

Backups, restores and DR

• DR plan steps cannot store credentials. A step whose restore settings include a recovery token or storage credentials fails when it runs. The upgrade removes those keys from saved steps, and saving a step that includes them is rejected. Open each affected step and save it again without credentials. Breeze resolves the storage destination from the snapshot when the step runs. See Disaster Recovery.
• Commands queued across an org merge or device move. A backup, restore or verify command that is still queued when its device changes organization is not delivered. This includes devices moved by an organization merge. The command is expired with a reason. Run it again after the move.
• Restores no longer receive storage credentials. A restore, verify or test-restore command no longer carries the storage destination. With the 0.118 backup helper, the device fetches each file it needs through a short-lived, read-only link that Breeze issues for that command and device only. Devices whose backup helper predates 0.118 keep using the previous path until they update. See Restoring from Backup.
• Restore as VM and instant boot report failures. A restore as VM that cannot place every file now fails before any VM is created, instead of reporting success. Instant boot runs its background sync inside the command, and an incomplete sync reports the VM as degraded. Hyper-V restore refuses a VM name that already exists.
• New run state: completed with errors. A backup run that produced a snapshot but could not read or upload some files (up to 10%) is now Completed with errors, not Completed. API consumers see the new status value completed_with_errors. See Monitoring Backups.
• Legal hold blocks erasure. Erasing an organization’s data is refused (409) while any of its snapshots is under legal hold, and a legal hold cannot be placed on an organization that is being erased. Permanently deleting a device is refused while one of its snapshots is under legal hold or inside its immutability window.
• Storage cannot be shared across partners. A bucket, endpoint or local path already used by an organization under another partner is refused. Organizations under the same partner can still share one.
• Cove name matches are no longer mapped automatically. Only a Cove customer whose external code is set to a Breeze organization ID is mapped on sync. Map name-only matches yourself. Mappings made before this release are not changed. See Third-Party Backup Providers.
• Recovery bundles verify their helper. A recovery bundle’s launch script checks the included helper binary’s signature before running it. Bundles need recovery signing configured on the server when they are built, and the machine that runs them needs minisign (plus curl on Linux and macOS). Rebuild recovery bundles after upgrading. The recovery console also no longer relaxes its HTTPS requirement for breeze.insecure=1 in interactive mode.

Agent

• Windows: linked data and log directories are replaced. If C:\ProgramData\Breeze\data or C:\ProgramData\Breeze\logs is a junction or symbolic link, the installer and the agent replace it with a real directory that only SYSTEM and Administrators can write to. Files in the old link target are not moved or copied, so copy anything you need first. If the link cannot be removed, the agent refuses to start and writes the reason to C:\ProgramData\Breeze\logs\agent-start-failed.txt. Remove the link by hand, then start the service.
• macOS: agent binaries move only when /usr/local/bin is unsafe. Agents 0.118.0 through 0.118.2 moved the agent and watchdog binaries from /usr/local/bin to /Library/Breeze/bin on every Mac, on first start, which broke Full Disk Access for anyone on that path (macOS ties FDA to a bare binary’s file path, not its identity). Upgrade straight to 0.118.3 or later — do not promote 0.118.0-0.118.2 to Macs you cannot touch. From 0.118.3, the agent moves only when /usr/local/bin (or a directory above it) is not owned by root, or is group- or world-writable — the known case is Homebrew on Intel Macs, which hands that directory to an admin user. A stock Mac (root-owned, non-writable /usr/local/bin) never moves and keeps all of its grants. Only a Mac that does move needs Full Disk Access re-granted for /Library/Breeze/bin/breeze-agent; the agent logs a warning when it moves and reports a macos_fda_relocation self-health warning until the grant lands. A Mac already moved by 0.118.0-0.118.2 stays at /Library/Breeze/bin on 0.118.3+ — it is not moved back, so a re-grant already done keeps holding. See macOS Permissions.
• Helper token file. On Linux and macOS, the Helper token moves to helper_token.yaml.

Access, MFA and limits

• Account changes reach delegated access. A password change or reset, or an MFA factor reset, ends that user’s live remote sessions and invalidates the API keys they created before the change. A role change revokes the user’s refresh tokens. Service-principal keys are not affected, and the upgrade itself revokes nothing.
• More actions need a fresh MFA check:
  • revealing a disk-encryption key;
  • report email recipients;
  • partner billing profiles and work types;
  • org billing-profile assignment and org billing settings;
  • linking a backup-provider device;
  • topology template library changes;
  • software-policy executable rules, which also need pam:manage_policy;
  • AI script policy changes;
  • deactivating a policy.
• Permissions. System Tools file-content and registry reads need devices:execute, and so do sensitive event-log channels (Security, PowerShell, Sysmon). Reads of the agent’s own configuration directory and the SAM and SECURITY registry hives are always refused. Enrollment short codes are shown only to users who can create or rotate enrollment keys. Role, MFA-reset and member actions require the caller to hold every permission of the target user’s role.
• PAM and AI approvals. An approval for a specific executable without its file hash is denied with a reason. AI elevation requests always wait for a human approver, and a requester cannot approve their own elevation. AI ticket comments are private unless approved as customer-visible. Newly discovered MCP tools must be reviewed before they are enabled.
• Enrollment and installers. Rotating an enrollment link issues a new short code, so the old /s/ and /i/ links return 404. A custom-field value used in installer arguments must not contain spaces, quotes or backslashes; otherwise that device’s install fails as an unresolved field.
• Stored credentials stay tied to their destination. Changing a tool source, backup S3 endpoint, log-forwarding URL or monitoring integration URL to a new origin requires re-entering its credential. Changing an SNMP discovery profile’s subnets, exclusions or site requires re-entering or clearing its SNMP secrets.
• Remote sessions. Where the remote-access policy requires consent or an on-screen notice, desktop sessions over the WebSocket fallback to agents older than 0.118 are refused until the agent updates.
• New limits. Most can be tuned; see Environment Variables → Agent traffic and ingest budgets.
  • public downloads: 600 requests per 5 minutes per route per IP;
  • TURN credentials: 30 per 10 minutes per user;
  • password-reset emails: one per account per 5 minutes;
  • agent WebSocket messages, plus daily agent log, event-log and change-log ingest (429 when exceeded);
  • Helper screenshots;
  • pending log-forwarding jobs per organization;
  • the partner’s organization limit when creating organizations.

Other behaviour changes

• Public invoice and quote links stop working while the owning partner is not active.
• Adopting a sending domain that already exists at the email provider needs a DNS TXT record at _breeze-verify.<domain>, or the domain listed in EMAIL_DOMAINS_ADOPT_EXISTING_ALLOWLIST. Otherwise provisioning ends as provider_conflict.
• Organization-scoped users cannot set audit-log retention below 90 days.
• Revoking a topology template is final.
• Inviting an address that already belongs to another tenant returns the normal success response.
• Public download routes, including install.sh and uninstall.sh, return 503 while Redis is unavailable.
• With BINARY_SOURCE=local, the Breeze Helper download is checked against the signed release manifest and returns 503 on a mismatch. Keep the manifest files next to the Helper binaries (HELPER_BINARY_DIR is respected).
• Notification channel settings (webhook URLs, bot tokens, routing keys) move to their own table, which only the channel’s owner can read. The old column is kept for rollback and is dropped in the next release.

Environment variables

No new required variables. New optional ones all have defaults; see Environment Variables. The four OFFLINE_DETECTOR_REEVAL_* variables no longer do anything: the job that read them was retired with legacy alerting. You can remove them.

Migrations

All migrations are idempotent and apply on boot. Migrations that change rows report their counts in the API log. Read these before upgrading:

• One accounting connection per partner. The API does not start if any partner has more than one accounting connection. Disconnect the extra provider first.
• Backup snapshot index. A new unique index on backup snapshots is built without blocking writes. If duplicate snapshot rows exist, the API logs a warning with the count and skips the index, and tries again on every start until the duplicates are resolved. If an interrupted build left an invalid index, the migration stops with an error. Run DROP INDEX CONCURRENTLY backup_snapshots_storage_identity_snapshot_id_uq and restart.
• Elevation audit index. Same pattern. If an interrupted build left the index invalid, run DROP INDEX CONCURRENTLY public.elevation_audit_org_created_id_idx and restart.
• Stored storage destinations are cleared. Storage destinations are removed from finished device commands, and credential keys from DR plan steps. On an install with a long command history, the first boot takes longer. Queued commands are left alone and cleared when they finish.
• Patch job results index. Built without blocking writes, but it can take a while on a large table.
• Network checks. A new asset/organization constraint on network checks stops the migration if existing rows already break that pairing. It converts and retires nothing.

Rolling back to v0.117.0 means restoring the database backup you took before upgrading, together with the v0.117.0 images.

v0.117.0 — legacy labour-pricing columns dropped

The six legacy labour-pricing columns are dropped. The migration removes default_billable, default_hourly_rate and rate_currency from both ticket_categories and org_ticket_settings. Nothing in Breeze has read them since v0.115.0. Any report, BI query or integration that still selects them directly from the database fails after this upgrade. Read pricing from the billing-profile tables (billing_profiles, billing_profile_rules, org_billing_profile_assignments) or from the values stamped on each time entry (time_entries.hourly_rate, coverage, work_type_id, billable_minutes).

Nothing is lost: the old values are archived first. Before it drops the columns, the migration copies every row that still holds a legacy value into a new table, legacy_labour_pricing_archive. A few values never made it into a billing profile during the v0.115.0 conversion. For those rows, skip_reason says why:

skip_reason	What the conversion did
org_currency_off_list	The organization’s currency is not on the supported list, so its rate and billable default were skipped entirely.
org_rate_currency_mismatch	The organization’s rate was entered in a different currency from the organization’s own, so the rate was not carried over.
non_billable_org_rate	The organization was non-billable but also had a rate. A non-billable card carries no rate, so the rate was dropped.
non_billable_category_rate	The category was non-billable but also had a rate. The rate was dropped.
category_rate_currency_unsupported	The category rate had no currency, or an unsupported one, so it was not carried over.
(empty)	Converted into a billing profile. The row is kept for reference.

To see what still needs re-entering on Settings → Billing → Rates:

BEGIN;
SELECT set_config('breeze.scope', 'system', true);  -- the table is row-level secured
SELECT source_table, source_name, default_billable, default_hourly_rate,
       rate_currency, owner_currency_code, skip_reason
  FROM legacy_labour_pricing_archive
 WHERE skip_reason IS NOT NULL
 ORDER BY partner_id, source_table, source_name;
COMMIT;

The migration counts what it archived in the API log (billing profiles W04b: lines). It refuses to run, and the API does not start (refusing to drop legacy labour-pricing columns), in two cases. The first is when any partner was never converted. A normal upgrade path cannot produce that, because the conversion runs earlier in the same migration sequence. The second is when manual DDL left only some of a table’s three legacy columns in place, which the migration will not drop without archiving them.

Rolling back to v0.116 after this upgrade needs a database restore. The v0.116 image’s tenant-export policy still lists the three org_ticket_settings columns, and tenant export rejects any mismatch between that list and the live schema, so organization data export fails on a v0.116 image running against an upgraded database. Back up your database before upgrading. To roll back, restore that backup together with the v0.116 image.

Billables CSV: two new columns at the end. The billables export (GET /tickets/export/billables.csv) appends work_type and included_minutes (worked minutes of contract-included time, 0 on other time rows, empty on parts). Existing columns keep their positions. An importer that rejects unknown trailing columns needs updating.

v0.116.0 — retired labour-pricing fields are rejected

Breaking API change: defaultHourlyRate, defaultBillable and rateCurrency are rejected. In v0.115.0 labour pricing moved from per-organization ticket settings and per-category defaults to partner-level billing profiles (rate cards) with per-work-type rows. The conversion migration builds the profiles from your existing rates and assigns one to every organization, so nothing needs re-entering. From v0.116.0 PATCH /organizations/:id/ticket-settings, POST /ticket-categories and PATCH /ticket-categories/:id return 400 when the request carries any of the three retired fields, and the whole request is rejected — a rename sent alongside a retired field does not land either. The error names the field and its replacement. Update any integration or script that writes these fields: assign a profile with PUT /organizations/:id/billing-profile, edit rates under Settings → Billing → Rates, and point a category at a work type via defaultWorkTypeId. GET responses no longer return the retired fields. Breeze’s own web, mobile and add-in clients never send them.

Note

Why there is no grace period: the cut-over already stopped honouring these writes, so a request that “succeeded” with 200 was silently discarding a rate change — an integration raising a rate would have kept billing at the old one. A loud 400 is the safer failure. Background and the resulting deprecation policy: LanternOps/breeze#6472.

The six underlying database columns are still present in v0.116.0, so v0.116.0 can still be rolled back to v0.115.0. Nothing reads or writes them any more, and they will be dropped in v0.117, which will carry its own rollback note. Repoint any report, BI query or integration that reads ticket_categories.default_billable, default_hourly_rate or rate_currency, or the same three columns on org_ticket_settings, directly from the database before upgrading to v0.117. Read pricing from the billing-profile tables (billing_profiles, billing_profile_rules, org_billing_profile_assignments) or from the values stamped on each time entry (time_entries.hourly_rate, coverage, work_type_id, billable_minutes). The legacy columns hold only the pre-conversion values. Back up your database before upgrading all the same.

v0.115.0 — billing profiles replace per-organization labour rates

Labour pricing moved to billing profiles, and the old fields are silently ignored in this release. The conversion migration builds partner-level billing profiles from your existing organization and category rates and assigns one to every organization; nothing needs re-entering. However, v0.115.0 still accepts defaultHourlyRate, defaultBillable and rateCurrency on PATCH /organizations/:id/ticket-settings, POST /ticket-categories and PATCH /ticket-categories/:id, returns 200, and discards the value. The only signal is a deprecationWarnings array in the response body. An integration that changes a rate through these fields on v0.115.0 keeps billing at the old rate. Move rate changes to billing profiles (PUT /organizations/:id/billing-profile, or Settings → Billing → Rates) before or during this upgrade. v0.116.0 turns these writes into a 400 (above).

v0.114.0 — evidence reports, network device page, account board

Navigation moved; old URLs removed. The Fleet Management entry Monitoring is now Network Monitor and holds only network assets, checks and SNMP templates. Device monitors (CPU, memory, disk, service, process, custom) moved to Alerts → Monitors (/alerts/monitors). The pre-v0.114 URLs /monitoring/monitors, /monitoring/rules, /monitoring/delivery, and /monitoring/network were removed and now return 404. Update bookmarks to /alerts/monitors, /alerts/rules, /alerts/channels, and /monitoring, respectively. If a technician reports that “Monitors disappeared”, this is where they went.

catalog_items.unit_price is dropped. Catalog sell prices have lived in catalog_item_prices (per-organization price book) since the multi-currency work landed; unit_price on catalog_items was a write-only leftover column that nothing read. The API and catalog UI are unaffected — the unitPrice field on create/update requests is still accepted as an input alias mapped into the price book. If you have custom SQL, a BI tool, or a script reading catalog_items.unit_price directly, it will break; point it at catalog_item_prices (via catalog_item_org_pricing) instead.

Caution

Rolling back to a pre-v0.114.0 image after upgrading is not clean — the dropped column does not come back on its own. Back up your database before upgrading if you might need to roll back.

v0.113.0 — remote desktop stability, AI-authored scripts, service deliverables

No breaking API changes. Two settings deserve a decision before you pull: APP_ENCRYPTION_KEY_ID (now a boot refusal for split-role deployments and a 503 on integration credential saves everywhere else) and BREEZE_AI_SCRIPT_AUTHORING_ENABLED, which is now on by default. Read Action required before you pull.

If remote desktop has been dropping every 30–90 seconds since early August, take this release

Since late July the orphan-session sweeper was ending healthy peer-to-peer viewer sessions because it only recognised a WebSocket-owned session as alive — and the default P2P transport never opens that socket. The fix is entirely server-side; no agent update is needed for it. The separate secondary-monitor fix (input and cursor landing on the wrong monitor on Windows hosts with mixed DPI scaling) does need the updated agent and helper.

Pull api, web and portal together, in one pass (plus worker if you run the worker split). The customer portal gains two new sections and the web dashboard gains new navigation, so a mixed set of images shows menu entries that 404. Afterwards assert version parity on every ghcr.io/lanternops/breeze/* container rather than trusting the service list.

Action required in v0.113.0

1. Set APP_ENCRYPTION_KEY_ID if you have not already. The API refuses to boot without it when BREEZE_ROLE is api or worker or when M365_GRAPH_ACTIONS_TOOLS_ENABLED=true. Every other deployment boots with a warning, but saving integration credentials returns 503 — Discord, Teams, Grafana, OpsGenie, PagerDuty, monitoring webhooks, ticketing and PSA settings — until the key id is set, because those credentials are now sealed at rest with key-bound ciphertext and no longer fall back to unbound ciphertext. Add any [A-Za-z0-9._-]+ identifier to .env and map it in the api (and worker) service’s environment: block; installs whose guided-setup.sh run predates the key never had it generated. Setting it does not rotate APP_ENCRYPTION_KEY. See Environment Variables.
2. AI script authoring is on by default. The chat assistant and background AI agents can now write a script as a reviewed proposal and, once a human approves it on a card, run it. Set BREEZE_AI_SCRIPT_AUTHORING_ENABLED=false to keep the feature dark. Two related permission changes: approving a proposal that matched a Strict danger pattern requires the approver to hold scripts:write in the proposal’s organization and to have completed MFA (the same bar the script library already applies), and four-eyes requests for such proposals fan out only to approvers who hold scripts:write — grant it to whoever decides AI script runs. The unattended lane ships off: it needs a partner to open the ceiling and each organization to opt in, and no environment variable controls it. See AI Script Authoring.
3. Two new permissions, documents:read and documents:write, gate the new organization document library and the Documents tab. The migration grants both to the built-in Partner Technician, Org Admin and Org Technician roles; custom roles get nothing until you add them under Settings → Roles. Service deliverables themselves use the existing contracts:read / contracts:write.
4. If you take online payments through Stripe, resetting a payment link, recording a manual payment, voiding an invoice or replacing the Stripe key now refuses with 503 STRIPE_REVOCATION_PENDING when Breeze cannot prove the customer’s open Checkout session is dead. A 60-second sweep retries, so the refusal is brief; if it ever blocks invoice work during an incident, set STRIPE_SESSION_REVOCATION_MODE=observe rather than rolling back.
5. Customers with Microsoft 365 Graph-read consent see a re-consent prompt. The customer-graph-read manifest moves to v3 with four additional Graph permissions for the upcoming posture program. The prompt is non-interrupting — existing connections keep working until the customer administrator re-consents.

Database migrations in v0.113.0

22 migrations (2026-10-15-170300 through 2026-10-16-170700). Take a database backup first. All are idempotent, apply automatically on API startup unless AUTO_MIGRATE=false, and have been dry-run as a non-superuser role. None rewrites a large table. The ones with operator-visible effects:

Migration	Effect
2026-10-15-170400-documents-permissions.sql	Seeds documents:read / documents:write and grants them to the built-in Partner Technician, Org Admin and Org Technician roles (counts logged as Postgres WARNINGs).
2026-10-16-100000-script-versions-immutable.sql	Creates one head-version row per existing script and makes script versions append-only — every save now cuts a new immutable version.
2026-10-16-100300-scripts-origin.sql	Adds origin (human / ai / system) to scripts and backfills system on the built-in ones.
2026-10-16-160200-checkout-session-revocation.sql	Marks aged-out pending Stripe Checkout session mappings revoked and pre-existing ones legacy_unbounded; adds the revocation-intent bookkeeping.
2026-10-16-170100-mfa-enrollment-grace.sql	Adds the enrolment-deadline columns on users. No rows written.
2026-10-16-110000-portal-branding-service-documents-flags.sql	Adds the portal Service and Documents visibility flags, both false.

The rest — organization documents, script proposals and policies, monitor definitions, the remote-session start generation, M365 tenant-sync foundation, Fleet Designer and device-function tables — are DDL-only additions with no backfill.

New settings in v0.113.0

Only APP_ENCRYPTION_KEY_ID (item 1 above) is required, and only for the configurations listed there. Full descriptions are in Environment Variables.

Variable	Default	Why you might set it
BREEZE_AI_SCRIPT_AUTHORING_ENABLED	true	Set false to keep AI script authoring dark.
BREEZE_AI_SCRIPT_REVIEWER_MODEL	—	Platform-default model for the independent script reviewer; a partner or organization policy can override it.
STRIPE_SESSION_REVOCATION_MODE	enforce	observe is the incident de-escalation lever for item 4 above.
M365_SYNC_CONTINUATION_KEY, M365_SYNC_MAX_IN_FLIGHT, M365_MAX_IN_FLIGHT	— / 4 / 32	Read by the separate m365-graph-read-executor sidecar only. Without a continuation key the executor mints one per process and a sync resumes from page 1 after a restart — self-healing, so existing executors need nothing.
BACKUP_GC_ORPHAN_MANIFEST_MAX_AGE_MS, BACKUP_BASE_LEASE_MS, BACKUP_RESTORE_PIN_LINGER_MS, BACKUP_PUBLISH_MARGIN_MS	9 d / 7 d / 7 d / 1 h	Lab/test knobs for the backup storage sweep. Production enforces floors and logs when a lower value is ignored. Leave unset.

New infrastructure: one new queue, script-verify, consumed by the API replica that owns the agent socket. No new service, no new port.

Rolling back v0.113.0

Prefer roll-forward. Every schema change is additive, so rolling the images back needs no schema rollback; scripts saved on v0.113.0 keep their version rows. Do not revert the Checkout-session migration to escape a revocation refusal — flip STRIPE_SESSION_REVOCATION_MODE=observe instead, otherwise the older code ignores the pending intents and re-opens the duplicate-charge window.

Behaviour changes in v0.113.0

• Navigation. Automations now live under a top-level Jobs entry (bookmarked /automations/* links redirect); the sidebar’s Network Monitor entry is now Monitoring, a hub with Monitors, Network, Delivery and Legacy rules tabs; Devices is now Devices & Assets, with your last five devices under it and a Cmd+K palette that opens on recent devices and pages. See Automations and Monitors.
• Monitors are the new single object a technician authors — condition, severity, delivery and responses in one place — compiled into the alert templates, alert rules and automations that already run. Existing alert rules keep working; Legacy rules offers one-click conversion.
• Service deliverables and the organization document library ship with two new customer-portal sections, Service and Documents, both off per organization until you turn them on. A daily sweep opens tickets for deliverables that come due and for key-date reminders. See Service deliverables and Customer Portal.
• AI Fleet Designer is a fourth AI-agent kind, designer, that reads the fleet and writes a Fleet Design report. It is inert until an administrator creates one for an organization, and nothing in its design is applied automatically. It also assesses a Function for each device (domain controller, kiosk, …), shown next to the billable Device Role. See AI Agents.
• Consecutive supervised Tier-3 approvals need one step-up ceremony — a 120-second grant bound to the approver’s session — instead of one per decision. Four-eyes approvals still take a fresh ceremony each time.
• MFA enrolment grace window. A user who lands in a Force MFA role and has never held a factor gets a nonrenewable 14-day window (banner plus email notices) before the 428 enrolment wall applies. Inert unless MFA_FORCE_FOR_PARTNER_ADMIN=true, whose default is unchanged (false). See Users & Roles.
• Backup storage usage starts to drop. The reclamation sweep now deletes retired restore points’ exclusive data and orphaned upload prefixes, on top of the loose-object grace rules. See Storage.
• Integration credentials are sealed and masked. Discord, Teams, Grafana, OpsGenie, PagerDuty and monitoring-webhook secrets are encrypted at rest and never echoed back to the settings form; re-enter a secret to change it.
• Alert-correlation views honour the site ceiling and alerts:read. A site-restricted or permission-stripped reader no longer sees the whole organization’s correlation graph.
• Approver devices now distinguish Hardware-attested from Not attested platform-bound mobile devices. See Approval Security.
• M365 tenant-sync foundation lands schema and executor plumbing only — nothing new in the dashboard this release.
• Remote desktop start and stop commands are now strictly ordered per session on both server and agent. No operator action; a late start can no longer win against a terminal decision.

v0.112.0 — Windows MSI fix, security remediations, bare-metal recovery

No breaking API changes and no new required environment variables, but this release removes implicit access and pauses some scheduled work the moment the migrations run. Read Action required before you pull.

If you enroll Windows devices with the MSI, take this release now

The agent MSI published with v0.110.0 through v0.111.1 refused every fresh install on Windows 10, 11 and Server 2016+ with “Breeze Agent requires Windows 10 or Windows Server 2016 or later”. Already-enrolled agents were unaffected (self-update never runs the MSI); only new MSI enrollments were blocked. v0.112.0 reads the Windows version from the registry instead of the value Windows Installer reports. Replace any installer links or MSIs you handed out from those releases. See Supported Windows versions.

Pull api, web and portal together, in one pass (plus worker if you run the worker split). Two changes in this release have no flag-level rollback: session-binding enforcement (below) and the remote desktop lease, which expects every replica on the new protocol. Sequence the API restart as stop-then-start rather than rolling: stop the old API replicas, let the migrations apply, then start only the new ones — an older API must not resume serving after the installer-token cutover (item 6 below). Afterwards assert version parity on every ghcr.io/lanternops/breeze/* container rather than trusting the service list.

Action required in v0.112.0

1. PAM auto-approve rules are suspended. Every rule with verdict Auto-approve is switched to Require approval and badged “Auto-approve suspended”. Matching elevations wait for a human until an admin holding the new pam:manage_policy permission clicks Re-approve on the rule (MFA). See PAM → Rules.
2. New permissions replace implicit access. pam:approve, pam:manage_policy, accounting:read, accounting:manage, workspace:read|write|credentials|execute and connected_apps:read|manage are granted to the built-in Org Admin only (Partner Admin holds *:*). Every other built-in role and every custom role loses QuickBooks, Workspace, connected-apps and PAM-approval access until you grant the keys in Settings → Roles. A Partner Technician or Partner Billing user who could browse or sync QuickBooks yesterday gets a 403 today. See Built-in permissions.
3. Recurring sensitive-data scan policies are switched off and any queued or running scheduled scan is failed. There is no badge: each affected policy simply shows as inactive on the Scan Policies tab. A user who holds devices:write and devices:execute, with MFA satisfied, re-approves it by turning Active back on (or re-saving it), which binds the schedule to that user’s authority. Manual scans are unaffected. See Sensitive Data.
4. Recurring network-baseline scans are paused with the banner “Recurring scan paused: re-approval required”. Re-saving the schedule arms it under the saving user; Scan Now still works. Newly auto-created baselines now start with recurrence off. See Network Baselines.
5. Office add-in technician bindings are revoked. Every technician who uses the Outlook/Office add-in technician pane repeats the one-time sign-in; the account must now have an authenticator-app (TOTP) factor. See AI for Office.
6. Outstanding installer bootstrap tokens are invalidated. Unused bootstrap-derived enrollment keys are deleted. Regenerate any installer links or copies you handed out before the upgrade.
7. Remote desktop needs an updated agent and is capped at 12 hours. Only an agent that reports lease support can start a desktop session; older agents show “Remote desktop needs an agent update on this device” (HTTP 503 agent_upgrade_required). Expect up to one heartbeat interval (about a minute) per device while agents self-update; if you pin agent versions, desktop sessions are unavailable until you move the pin. Max session duration is clamped to 1–12 hours (0 no longer means unlimited), a membership/role/site-scope/MFA change ends a live session within about 25 s, and a Redis outage longer than 90 s ends live desktop sessions. Terminal, Files and VNC are unaffected. See Session lease and revocation.
8. AUTH_BROWSER_TRANSITIONS_ENFORCED is removed. Session binding is enforced on every issuance path (password, SSO, Cloudflare Access, MFA, passkey, refresh, registration, invitation) with no opt-out. Delete the line from .env; the safety net is an image rollback, not a flag. Existing browser sessions self-heal on their next refresh. Mobile app builds older than v0.109.0 cannot sign in until updated.
9. Legacy policy execution routes are gone. POST /policies/:id/evaluate, /remediate and /activate are retired; use POST /automations/:id/trigger. /deactivate is unchanged.
10. Site-restricted technicians can no longer create or edit organization-wide governance objects (webhooks, notification channels, software and peripheral policies, configuration policies, PAM config, backup destinations and profiles) or manage PSA and DNS-security connections. They can still assign an existing policy to their own site.
11. Check TRUSTED_PROXY_CIDRS. With proxy trust unset, client-IP attribution now records the raw socket peer instead of unknown. Behind a real proxy that peer is the proxy, so make sure the variable is set on every deployment that fronts Breeze with one.

Database migrations in v0.112.0

20 migrations. Take a database backup first. All are idempotent, none abort on pre-existing data, and all apply automatically on API startup unless you have set AUTO_MIGRATE=false. The ones with operator-visible effects:

Migration	Effect
2026-10-15-150200-pam-dedicated-permissions.sql	Suspends every auto_approve PAM rule (original verdict kept for re-approval); seeds pam:approve / pam:manage_policy and grants them to each built-in Org Admin.
2026-10-15-150051-office-addin-binding-mfa-epoch.sql	Revokes every existing Office add-in technician binding.
2026-10-15-140001 / 140002 / 140003-sensitive-data-*.sql	Deactivate every recurring sensitive-data policy and fail queued/running scheduled scans.
2026-10-15-141001-installer-bootstrap-credential-generation.sql	Deletes unused bootstrap-derived enrollment keys and bumps the generation on their parents.
2026-10-15-150030 / 150031 / 150032 and 150500-accounting-dedicated-permissions.sql	Seed workspace:*, connected_apps:*, accounting:read, accounting:manage; grant them to the built-in Org Admin only.
2026-10-15-140004-backup-snapshot-lineage-fk-set-null.sql	Makes restore_jobs.snapshot_id and recovery_tokens.snapshot_id nullable and moves five foreign keys onto backup_snapshots to ON DELETE SET NULL, so retention can finally delete a snapshot that was ever restored, verified or tokened.
2026-10-15-150600-network-baseline-recurring-authority.sql	Pauses every enabled recurring baseline scan (reapproval_required) until an authorized user re-saves the schedule.
2026-10-15-150001-bound-webhook-retries.sql	Normalizes legacy webhook retryCount values into 0–2; the count is logged as a Postgres WARNING.
2026-10-15-150050-mobile-refresh-family-binding.sql	Adds a partial index on refresh_token_families in-transaction (not CONCURRENTLY). Instant at typical sizes; run off-peak if yours is very large.

The rest (150010, 150020, 150040, 150100, 150300, 160010) are additive column adds with no data change. A non-superuser migrating role is sufficient, as before.

New optional settings in v0.112.0

None are required. Full descriptions are in Environment Variables.

Variable	Why you might set it
LENOVO_WARRANTY_ENABLED=true and/or LENOVO_API_KEY	Lenovo warranty lookup works for the first time — credential-free via Lenovo’s public lookup endpoint, or through the official Warranty API. DELL_CLIENT_ID / DELL_CLIENT_SECRET and both Lenovo knobs are now mapped through every shipped Compose file, so a value in .env is enough. See Warranty lookups.
RECOVERY_MEDIA_WORK_DIR	Already defaulted to /data/recovery-work by the shipped Compose files; moves bare-metal recovery media builds off the container’s 64 MB /tmp. Override only to relocate it.
BACKUP_GC_GRACE_MS	Lab/test only. Production enforces a one-hour floor; the 48 h default is unchanged.

Removed: AUTH_BROWSER_TRANSITIONS_ENFORCED (see item 8 above).

Fresh installs and image pinning in v0.112.0

.env.example and deploy/docker-compose.prod.yml now expect signed, digest-pinned image references (ghcr.io/lanternops/breeze/api@sha256:…) resolved from the release’s signed image inventory. The guided installer resolves and verifies them itself for 0.112.0 and later (fail-closed). The current installer refuses releases older than 0.112.0 unless you pass --allow-unverified-release; with that flag it warns and uses tag-tracking refs. Existing installs whose .env still carries …/api:${BREEZE_VERSION} refs keep working with the upgrade steps at the top of this page. If you have already switched to digest pins, update the four BREEZE_*_IMAGE_REF lines from the new release’s inventory on every upgrade — bumping BREEZE_VERSION alone changes nothing. guided-setup.sh --upgrade does this in one command (step 4a above). See Signed, digest-pinned image references.

Rotating RELEASE_ARTIFACT_MANIFEST_PUBLIC_KEYS is a fleet re-enrollment: agents pin the key they enrolled under and there is no remote revocation. See Binary Distribution.

Rolling back v0.112.0

Prefer roll-forward. If you must roll back, downgrade agents before the API: a v0.112.0 agent talking to an older API gets no lease answer and ends its desktop session after about 150 s. The added columns are additive and need no schema rollback. Rolling back the network-baseline change resumes every paused recurring baseline under no authority — clear each affected baseline’s schedule first.

Other behaviour changes in v0.112.0

• Bare-metal recovery now restores system state on Linux, every system_image backup records a disk-layout manifest with a restorability verdict, and a breeze-backup rebuild command can provision a disk from a snapshot. See Bare Metal Recovery.
• Backup exclude patterns with a leading / are root-anchored (gitignore semantics) instead of matching at any depth. File backups now capture symlinks, directories, ownership and setuid/setgid/sticky bits; manifests carrying them use a new format that older agents cannot read, so restore with a v0.112.0 agent. See Backup Policies.
• Retention GC no longer stops on the first snapshot that was ever restored, verified or tokened. Bucket usage still does not shrink — storage reclamation lands in a later release.
• Webhooks make one HTTP attempt per invocation; the queue owns retries, retryCount is capped at 2, and permanent failures (bad configuration, SSRF-blocked URL, ordinary 4xx) dead-letter immediately. /test and /retry on a paused webhook return 409.
• Org-scoped sessions are now evaluated against the partner IP allowlist.
• MCP tool get_fleet_status is renamed get_invite_funnel. The old name dispatches for one more release but no longer appears in tools/list.
• HP warranty lookup is de-registered — HP’s API is not available to RMM vendors, so HP devices stay at unknown.

v0.111.1 — hotfix for a v0.111.0 upgrade-blocking migration

Install v0.111.1, not v0.111.0

If you have not yet upgraded to v0.111.0, install v0.111.1 instead — it contains everything in v0.111.0 plus this fix, and skips straight past the problem below. If you are already on v0.111.0 and it started up normally, your database was not affected and you can upgrade to v0.111.1 at your leisure. Everything else in this guide’s v0.111.0 section still applies.

One of the migrations that shipped in v0.111.0, 2026-10-14-100100-discovered-assets-manual-source.sql, backfills a new provenance column on discovered network assets. On a database where those discovered_assets rows span more than one organization or partner, the backfill’s two update statements can request the partner-export lock hierarchy out of order, which Postgres rejects. The migration transaction rolls back, autoMigrate retries it on the next boot, and the API never finishes starting — a crash loop, not a degraded mode. A single-partner or empty install is not affected.

v0.111.1 adds a new migration, 2026-10-14-100050-discovered-assets-source-backfill-prelock.sql, which sorts immediately before the failing one. It acquires every partner and organization lock the backfill will need, up front and in the correct order, so that by the time the original migration replays, it has nothing left to do and completes normally.

If your API is already crash-looping on v0.111.0, you have two ways out:

1. Upgrade to v0.111.1 (recommended). The failed migration’s transaction rolled back cleanly, so nothing is half-applied — pulling v0.111.1 and restarting is sufficient on its own.

2. Apply the fix-forward SQL below by hand against your database, using the same role your migrations run as, then restart the API on v0.111.0 before upgrading further:

   BEGIN;
   
   -- The failed migration rolled back its own CREATE TYPE / ADD COLUMN, so in the
   -- crash-loop state the `source` column does NOT exist yet: create it first
   -- (guarded, so this is also safe where it already exists).
   DO $$ BEGIN
     CREATE TYPE public.discovered_asset_source AS ENUM ('scan', 'unifi', 'manual');
   EXCEPTION WHEN duplicate_object THEN NULL;
   END $$;
   ALTER TABLE public.discovered_assets ADD COLUMN IF NOT EXISTS source public.discovered_asset_source;
   ALTER TABLE public.discovered_assets ADD COLUMN IF NOT EXISTS url text;
   
   -- Required: the lock helpers read organizations, and discovered_assets is
   -- FORCE ROW LEVEL SECURITY. Without this the discovery SELECTs return no rows
   -- and the UPDATEs match nothing, silently.
   SELECT set_config('breeze.scope', 'system', true);
   
   -- Acquire every affected partner (shared) and organization (exclusive) export
   -- lock in ascending order BEFORE the backfill, so the per-row triggers find
   -- them already held.
   SELECT public.breeze_partner_export_lock_partners_shared(ARRAY(
     SELECT DISTINCT o.partner_id
       FROM public.discovered_assets d
       JOIN public.organizations o ON o.id = d.org_id
      WHERE d.source IS NULL AND o.partner_id IS NOT NULL
      ORDER BY 1));
   
   SELECT public.breeze_partner_export_lock_orgs_exclusive(ARRAY(
     SELECT DISTINCT d.org_id
       FROM public.discovered_assets d
      WHERE d.source IS NULL AND d.org_id IS NOT NULL
      ORDER BY 1));
   
   UPDATE public.discovered_assets a
      SET source = 'unifi'
    WHERE a.source IS NULL
      AND (a.detected_type_source = 'unifi_controller'
           OR EXISTS (SELECT 1 FROM public.unifi_devices u
                       WHERE u.discovered_asset_id = a.id));
   
   UPDATE public.discovered_assets SET source = 'scan' WHERE source IS NULL;
   
   COMMIT;

   This is idempotent: on a database where the migration already succeeded, every statement is a no-op.

Recovery for a deployment that is already stuck in the crash loop:

1. Set BREEZE_VERSION=0.111.1 in .env (or apply the fix-forward SQL above against v0.111.0 first, if you cannot upgrade immediately).
2. docker compose pull api web portal && docker compose up -d binaries-init api.
3. Wait for the API to report healthy (curl -sf https://yourdomain/health).
4. Once the API is healthy, bring the rest of the stack back up: docker compose up -d web portal.

No schema shape changes beyond v0.111.0: the same enum, the same two columns, the same constraints. The new migration only moves the backfill earlier and wraps it in the correct lock order.

v0.111.0 — offline work queue, script cancellation, organization record page

No breaking API changes and no new required environment variables. But five defaults change on upgrade, and two of them will alter what your customers and your technicians see the moment the API restarts. Read Behaviour changes before you pull.

Pull the portal container too, alongside api and web — and worker if you have opted into the worker split.

Do these two things before you pull

1. If you have QuickBooks connected and are not ready for Breeze to write payments into a customer’s books, turn Payment push off first (Integrations → QuickBooks). It ships on. See Payment push.
2. If you run network monitors with alert rules, review those rules, your notification channels and your escalation policies. Monitor alerts now notify and fire automations where before they only sat in the inbox. See Network Monitors.

Database migrations in v0.111.0

35 migrations. Take a database backup first. All are idempotent and apply automatically on API startup unless you have set AUTO_MIGRATE=false.

Two migrations deliberately abort the deploy on bad pre-existing data

Both concern custom fields, and both are intentional — the alternative is silently picking a winner among rows that are supposed to be unique. Each ships with a preflight query so you can find out before you pull rather than during:

Migration	Aborts on	Preflight
2026-10-10-100300-custom-field-definition-integrity.sql	Duplicate or ownerless custom-field definitions	apps/api/migrations/preflight/2026-10-10-100300-custom-field-definition-integrity-preflight.sql
2026-10-11-141000-custom-field-no-cross-axis-shadowing.sql	A custom-field key defined at both the organization and the partner level for the same device	apps/api/migrations/preflight/2026-10-11-141000-custom-field-shadowing-preflight.sql

Run both against your database before you upgrade and reconcile anything they report. See Custom Fields → Uniqueness and shadowing.

Six indexes are built concurrently, so writes are not blocked while they run — on a large audit_logs or process-sample history they are the long pole and the API does not serve until they finish. An interrupted build leaves an invalid index that the next boot reports loudly; recover with DROP INDEX CONCURRENTLY <name> and restart.

Three migrations touch existing rows and record what they did as a Postgres WARNING. Read these in the API log after the first boot:

Log line	What it means
device_custom_field_values backfill: copied N value(s), minted N definition(s) for previously undefined keys, N stored key(s) do not match the enforced key pattern	Custom-field values move into a table of their own. Sized by devices × custom-field keys. Keys written before the naming rules were enforced are deliberately kept in the old projection only, so nothing is lost or renamed under you.
backfilled N discovered_assets rows to source=unifi / source=scan	Every discovered asset gains a provenance column. Sized by your discovery inventory, not your device count — on a large scanned estate this is the longest single statement in the set.
The Partner Admin forced-MFA stamp	Records the reconcile described under Behaviour changes. Enforcement is off this release, so nothing changes for your admins at upgrade time.

One hot-table lock worth planning around

The offline-work-queue migration adds a foreign key and a partial index to the agent command table without the concurrent/deferred options, so it holds an exclusive lock on the agent command path for one sequential scan. At ordinary sizes that is seconds. If you retain millions of device commands, build both by hand before you pull — add the constraint NOT VALID then VALIDATE, and create the index CONCURRENTLY under the name the migration uses (idx_device_commands_deliver_by) — and the migration’s guards will no-op at boot.

A non-superuser migrating role is sufficient, as in every previous release: the range creates no roles, extensions, or schema-level grants.

New optional settings in v0.111.0

None are required and none fail boot when missing — except PLAY_INTEGRITY_SERVICE_ACCOUNT, which is validated when set. Full descriptions are in Environment Variables.

Variable	Why you might set it
Offline work queue	Queueing is always enabled for queueable actions. The former opt-out has been removed; use action-specific offline policies where available.
DEVICE_COMMAND_QUEUE_TTL_HOURS, DEVICE_COMMAND_QUEUE_SHORT_TTL_HOURS, DEVICE_COMMAND_QUEUE_POWER_STATE_TTL_HOURS	Shorten or lengthen how long queued work waits for an offline device (7 days / 24 hours / 24 hours).
MFA_FORCE_FOR_PARTNER_ADMIN	Now defaults to false. Set true to enforce forced MFA for Partner Admins immediately.
AI_OPERATOR_TASKS_ENABLED, AI_OPERATOR_RECIPE_SERVICE_RECOVERY_ENABLED	Turn on the opt-in AI Operator preview. Both ship off.
READINESS_CACHE_TTL_MS, READINESS_PROBE_TIMEOUT_MS	Tune the new readiness contract. Defaults suit every deployment we know of.
OFFLINE_DETECTOR_WORKER_CONCURRENCY	Raise offline-sweep concurrency on a very large fleet.
PLAY_INTEGRITY_SERVICE_ACCOUNT	Verify Android approver devices with Play Integrity as well as hardware Key Attestation. A malformed value stops the boot rather than silently reporting “not configured” forever.

Readiness versus liveness

/health is unchanged and remains liveness — it answers 200 before the instance can actually serve. /ready and /health/ready now answer a real aggregate readiness question, and API admission is gated on the consumers the process role requires. If your load balancer or Compose healthcheck should hold traffic until Postgres, Redis and the background consumers are genuinely up, point it at /ready. See Health Monitoring.

Behaviour changes in v0.111.0

• ⚠️ QuickBooks starts pushing payments into every connected realm. The Payment push switch ships on, so a connected realm in automatic push mode begins creating QuickBooks Payments as soon as the API restarts — no operator action. Only payments recorded after the switch became active are pushed, and a payment deletion always propagates regardless of the switches. Turn it off first if you are not ready. See Payment push.
• ⚠️ Network-monitor alerts now notify, escalate and automate. ICMP/TCP/HTTP/DNS monitor alerts previously landed in the Alerts inbox only. Expect a step change in notification and automation volume if you run monitors with alert rules. The five-minute cooldown, the deduplication and the auto-resolve on recovery all still apply. See Network Monitors.
• Work aimed at an offline device queues instead of failing. Patch jobs, automation Run script / Run command actions and scan/rollback commands are persisted with a delivery deadline and claimed on the device’s next heartbeat. A waiting step reads “Queued — device offline” instead of Failed. The other side of that coin: work you previously saw fail fast now sits pending for up to a week. Existing automations have no stored offline preference and therefore queue — set a step to Skip if it is only meaningful against a live device. See Devices → Queued actions.
• ⚠️ On Linux and macOS, breeze-agent service stop is now temporary while the watchdog is installed. The watchdog restarts the agent after its standby grace period. For a durable stop, stop or uninstall the watchdog too. See Agent Watchdog.
• service install now restarts the service on an already-enrolled host instead of leaving it stopped — the old behaviour stranded a host Offline until someone started it by hand. See Agent Installation. The Windows equivalent is fixed too, including installing over an existing service, which previously failed while reporting the service as running when it was not.
• “Remove device” now uninstalls the agent by default. The removal flow asks about the agent and defaults to uninstalling it; previously the uninstall was effectively never sent and agents were left running on removed devices. A permanent delete now refuses until that uninstall has been collected.
• Forced MFA for Partner Admins is reconciled but NOT enforced. A migration marks every existing system Partner Admin role as requiring MFA — reconciling installs that predate the rule — but MFA_FORCE_FOR_PARTNER_ADMIN now defaults to false, because acting on it at upgrade time would lock existing Partner Admins into enrolment with no warning. Set it to true to enforce now. A notification-period rollout — grace window, banner, deadline — ships next release, at which point enforcement returns to on by default. Separately, an MFA policy change that would lock out already-enrolled users is now refused outright with the number of users affected.
• Maintenance windows suppress every script path. A device inside a window with Suppress scripts is now skipped by the AI assistant, automation Run script / Run command, fleet-fix remediation and edition auto-migration — not just a manual Run Script. Each is recorded as a skip, not a failure, so an automation run stays green and its trailing actions still run. A window that cannot be evaluated deliberately keeps the ordinary failure treatment, so a maintenance-config outage can never render as a fleet of green runs. See Maintenance Windows.
• Script cancellation kills the whole process tree. Cancel and timeout now terminate descendants on Windows and Unix. A script that intentionally detaches a child will now have that child killed too — worth a canary group if you rely on that pattern. See Stopping a running script.
• Configuration-policy inheritance changes what a “linked” policy delivers. A policy marked as inheriting could previously deliver nothing to its devices; every reader now resolves the real inherited settings. If you have such a policy, devices under it will start receiving configuration they were not receiving before. See Configuration Policies.
• Service Management is a partner-level module. native is the default and existing partners are unchanged. Turning it off hides the Service Desk and Billing sections and refuses new ticket creation everywhere. See Service Management mode.
• The unified device list is on by default. PUBLIC_ENABLE_NETWORK_DEVICES_IN_LIST now defaults to true. It is a build-time variable baked into the web image, so the published image ships with the All / Agent / Network / Manual segments on and there is nothing to change in .env or Compose. Build the web image yourself and pass false to keep the agent-only list.
• Metric-anomaly detection looks back 15 minutes instead of 30, with a statement timeout, so overlapping runs stop piling up on the rollup tables. The trade-off: a skipped tick no longer self-heals and a gap needs a manual backfill.
• Cross-organization device moves may now drop custom-field values that are not visible in the target organization. Previously they were kept silently; now they are dropped and logged.
• Still off by default: AI agents and the AI Operator preview, reboot deferral (per policy), and the removed-device purge retention policy.

Known wrinkle on Linux

systemctl stop breeze-agent runs its full 30-second stop timeout and ends in a kill, so each service install takes roughly 32 seconds and the unit is briefly reported as failed before it comes back. This is pre-existing and cosmetic — the agent ends up running.

v0.110.0 — reboot deferral, device-set billing, QuickBooks payments, portal visibility

No breaking changes and no new required environment variables. Everything new is either off by default or optional. A number of new optional variables were added, though — see New optional settings below and the full Environment Variables reference.

Pull the portal container too. The customer portal runs in its own container and gained several new pages in this release. A bare docker compose pull covers it; if you pull services by name, make sure portal is on the list alongside api and web. If you have opted into the worker split, include worker as well.

Refresh your Compose file from the release

docker compose pull updates images — it does not update your docker-compose.yml. New optional variables only reach the container if they are mapped in the api service’s environment: block, so a value added to .env alone silently does nothing. Compare your deployed Compose file against the one published at the release tag and merge in what you need, keeping your own domains, mounts, and overrides.

Database migrations

54 migrations. Take a database backup first. All are idempotent and apply automatically on API startup (unless you have set AUTO_MIGRATE=false). Every index on a table that can be large is built concurrently, so writes are not blocked while they run — the trade-off is that an interrupted build leaves an invalid index behind. If that happens the next boot reports it loudly; recover with DROP INDEX CONCURRENTLY <name> and restart.

Three migrations clean up or stamp existing rows and record what they touched as a Postgres WARNING. Read these in the API log after the first boot:

Log line	What to do if the count is above zero
cleaned N contract_lines rows whose site belonged to another org	Re-scope those contract lines in the contract editor before your next billing run.
The site-name stamp on contract_lines	This records how many existing site-scoped lines were protected. A line whose site was deleted before this release cannot be recovered and keeps billing organization-wide until a technician re-scopes it.
rls-scoped replay	Informational. Five earlier migrations wrote rows without the right database scope and are replayed correctly here; the counts tell you how many rows were fixed.

New optional settings in v0.110.0

None of these are required and none of them fail boot when missing. Full descriptions are in Environment Variables.

Variable	Why you might set it
QBO_WEBHOOK_VERIFIER_TOKEN	Makes recorded QuickBooks payments come back within seconds instead of on the 15-minute sweep.
FIREBASE_SERVICE_ACCOUNT	Enables push notifications to the Android app.
AUDIT_CHAIN_VERIFY_MODE, AUDIT_CHAIN_VERIFY_RESCAN_SLICES	Nightly audit-chain verification is now incremental by default rather than a full-table walk. AUDIT_CHAIN_VERIFY_ENABLED=false now also stops a sweep that is already running.
TICKET_OUTBOX_RETENTION_DAYS, INTENT_OUTBOX_RETENTION_DAYS, METRIC_ANOMALY_INCIDENT_RETENTION_DAYS (and their batch knobs)	Tune how long the new nightly retention sweeps keep delivery and incident history. Defaults to 14 days.
SMTP_TIMEOUT_MS, MAILGUN_TIMEOUT_MS	Bound how long a wedged mail host can hold an outbound send open.
BREEZE_AUTHENTICATOR_ATTESTATION_ENFORCED	Already on by default. Only set it if you need to accept an approver device’s unverified platform-bound claim.
PARTNER_TRUST_MODE and friends	Hosted service only — inert on a self-hosted install.

Behaviour changes to know about

• End-user restart prompts ship off. Turn them on per policy on the Patch tab with Let users postpone the restart. The deadline is the guarantee; the postponement count is a courtesy budget. Linux endpoints need zenity installed to show a dialog the user can postpone from — without it they get a plain notification and the restart proceeds on schedule. See Patch Management.
• Unrated patches are never treated as Low and are excluded from auto-approval unless an update ring opts in. Expect a backlog of pending patches to appear that were previously auto-approved as Low. See Update Rings.
• Deleting a site used by a site-scoped contract line now fails invoice generation with a clear error instead of quietly billing every device in the organization. Re-scope or remove the line, then generate again.
• Audit-log retention now runs for every organization that has a policy. New organizations get a default policy automatically; existing organizations have no policy until an operator saves one, and nothing is deleted until then. See Audit Logs → Retention.
• New partners are created with inbound email-to-ticket turned off. Existing partners are unchanged.
• Windows 10 and Windows Server 2016 are the declared agent OS floor. Older Windows builds are refused by the installer rather than enrolling and failing later.

AI agent graduation and impact accounting

Both features run under the flags you already have — BREEZE_AI_AGENTS_ENABLED for the nightly impact rollup, and BREEZE_AI_AGENTS_POLICY_DECIDE_ENABLED (still off, still not generally available) for promoting an operation to pre-authorized execution. Automatic demotion is not gated by anything and is always on.

Behaviour change: partner-level pre-authorized operations stop applying on their own

A partner-wide agent’s supervisedActionKeys are now a ceiling — what an organization may be granted — not an inherited grant. An organization whose agent has no override row of its own now resolves to an empty set of pre-authorized operations, whatever the partner baseline lists.

Nothing is silently revoked and no data changes, but any organization that was relying on the partner list has to be granted the operations explicitly: either through the Graduation panel’s approval flow, or by editing that organization’s own agent. Do this before you turn BREEZE_AI_AGENTS_POLICY_DECIDE_ENABLED on, not after. See AI Agents → Graduation.

Three new daily background jobs are registered. They run in the container that owns background work — the api container by default, or the worker container if you have enabled the worker split:

Job	Runs daily at (UTC)	What it does
Graduation evaluation	18:28	Re-evaluates which agent operations have earned enough clean evidence to be promoted. Read-only with respect to what agents may do — it never grants or revokes anything.
Impact rollup	18:33	Rebuilds the last 7 complete UTC days of the AI Impact report per organization, or the full 90 days the first time it sees an organization. Skipped entirely while BREEZE_AI_AGENTS_ENABLED is off.
Evidence retention	18:48	Prunes agent outcome evidence older than 400 days.

All three sit in the existing daily lane and are deliberately spaced apart from each other and from the retention sweeps, so they do not add a new nightly spike.

Note

The graduation view returns 404 — “No active agent policy for this organization/kind” for an organization that has no effective agent, which includes an organization-only agent with no partner-wide baseline behind it. That is by design, not a fault: without a baseline there is no ceiling to graduate against.

v0.109.0 — AI agents, ticket attachments and org archiving

Mostly additive. Nothing in this release requires a new setting before you upgrade, but note the following:

• Migrations. 53 idempotent migrations auto-apply on boot. Two touch tables that can be very large on a long-running self-hosted instance: one builds an index on remote-session history, and one adds a validated constraint to time entries. Both take a brief lock while they run, so expect a slightly longer boot than a routine upgrade — proportional to how much remote-session and time-entry history you hold. Take the database backup in step 1 above and upgrade during a quiet window.
• New AI agent capabilities are off by default. BREEZE_AI_AGENTS_ENABLED remains false, and even with it on, each agent starts in Off — an agent only acts after you deliberately move it to Act and confirm with multi-factor. See AI Agents.
• New partners start with inbound email-to-ticket turned off. Existing partners are unchanged. See Ticketing.
• Optional worker split. You can now move background jobs to their own container. Entirely opt-in and off unless you enable it — see Worker Split.

v0.95.0 — authentication & tenancy hardening

This release resets the authentication surface. Plan for a short low-traffic window and expect the following the first time the new API boots:

• Everyone is signed out once. Existing web and mobile sessions are invalidated and everyone signs in again — there is no silent-refresh recovery and no kill switch. Confirm you and your admins have a working multi-factor device or recovery codes before upgrading, since signing in is the only way back in. Agents, the Helper, the portal, and installers are unaffected.
• Connected AI / automation apps must re-authorize. Legacy OAuth refresh tokens are removed; any MCP/OAuth client reconnects and re-consents once.
• Microsoft 365 ticket mailboxes need re-consent. Non-disabled mailboxes drop to a re-auth state; a Partner Admin re-approves each one. Mail arriving during the gap is picked up after re-consent, not lost.
• New required database credential. Production now runs request handlers as an unprivileged, RLS-enforced role and will not start unless one of DATABASE_URL_APP, BREEZE_APP_DB_PASSWORD, or POSTGRES_PASSWORD is set, and it refuses to run as a SUPERUSER/BYPASSRLS role. See Environment Variables → Database. This check runs even when AUTO_MIGRATE=false.
• Behind Cloudflare? Set TRUST_CF_CONNECTING_IP=true (see Environment Variables).
• Migrations. 26 idempotent migrations auto-apply on boot. Most are metadata-only, but one backfills the refresh_token_families table — on a large, long-running deployment, check that table’s size first and expect a slightly longer boot while it applies.

Caution

Any value you set in .env only takes effect if it is also mapped in the api service’s environment: block of your compose file. The shipped compose files map the database-credential variables; if you maintain your own compose file, add them there too.

Agent Updates

Agents update themselves automatically. On each heartbeat (~60s), the API checks if a newer version is available and signals the agent to upgrade. The agent:

1. Downloads the new binary and verifies its SHA-256 checksum
2. Backs up the current binary
3. Replaces the executable and restarts the service
4. Rolls back automatically if the new binary fails to start
5. Swaps the bundled breeze-backup helper to the matching version in the same upgrade; a periodic reconcile re-fetches it later if the swap was skipped or the helper is missing or stale

Since v0.105.0, release manifests can label each asset with an agent edition (self-host or hosted); the updater refuses an asset whose declared edition doesn’t match the running build, and accepts manifests without edition labels for backward compatibility. See Self-Host Agent Migration.

During an update, the device status changes to Updating in the dashboard so you can distinguish active updates from offline agents. If the update takes longer than 5 minutes, the device reverts to Offline. After a successful restart, the agent immediately reports its new version (bypassing the normal 0–60 second heartbeat jitter).

Linux Binary-In-Use Handling

On Linux, the updater handles the case where the agent binary is still in use (common with manual installs that don’t use a package manager). The updater removes the old binary before writing the new one to avoid ETXTBSY (“text file busy”) errors. If a file lock error still occurs, it is treated as transient and retried on the next heartbeat.

Watchdog Recovery

The watchdog service automatically restarts the agent after a failed update or crash. If an update leaves the agent in a bad state, the watchdog detects the failure and restarts the previous (backed-up) binary.

The binary source depends on your BINARY_SOURCE setting:

Mode	Agent downloads from
github (default)	GitHub Releases CDN
local	Your API server (or S3 if configured)

See Binary Distribution for details.

Disabling Auto-Update on Specific Devices

Some devices — production database servers, devices in a change-controlled window — should not self-update on the normal heartbeat cadence. Toggle auto-update per device without editing config on the machine:

This endpoint requires a signed-in user session with MFA satisfied, not an API key — $TOKEN below is a Breeze access token, not a partner API key.

# Disable auto-update for one device
curl -X POST "https://breeze.example.com/api/v1/devices/<device-id>/auto-update" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'

# Re-enable later
curl -X POST "https://breeze.example.com/api/v1/devices/<device-id>/auto-update" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}'

The agent persists this setting across restarts. This is also the way to recover agents that were left with auto-update disabled by a manual or recovery update — see the set_auto_update command.

Controlled agent fleet rollout

By default (AGENT_AUTO_PROMOTE=true), syncing a new agent release immediately makes it the fleet’s upgrade target — every auto-updating device moves to it on its next heartbeat. Boot-time sync only ever registers the release matching your own BREEZE_VERSION (or BINARY_VERSION if set), so a newly published GitHub release does not move your fleet until you upgrade your server images — the fleet follows the server, not the public release feed. If you’d rather stage rollouts (canary first, then the fleet), set AGENT_AUTO_PROMOTE=false to decouple registering a release from promoting it.

1. Turn on controlled rollout. Set AGENT_AUTO_PROMOTE=false in .env and map it in the api service environment: block of your compose file, then restart the API. (Compose only passes through variables you list there.)

2. Publish or sync the release. New binaries register as downloadable but are not promoted — the fleet stays on the current version. Existing devices are unaffected.

3. Canary it. Update a few non-production devices to the new version and verify (you can target an individual device’s update directly).

4. Promote to the fleet. When you’re satisfied, promote the version. This is a platform-admin action and requires MFA:

   curl -X POST "https://breeze.example.com/api/v1/agent-versions/promote" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"version": "0.84.0"}'

   Auto-updating devices then move to the promoted version on their next heartbeat. To promote a single component (e.g. just the helper or watchdog) while holding the rest, add "component": "helper" to the body.

Tip

Leave AGENT_AUTO_PROMOTE unset (or true) to keep the default behavior where the release matching your server version rolls out as soon as the API syncs it. The promote action is audit-logged either way.

Pinning Agent and Watchdog Versions

Instead of always tracking the latest promoted release, you can pin the fleet — or a single customer — to a specific version. Settings → Partner Defaults and Settings → Org Defaults each expose an Update version targets card with independent Agent target and Watchdog target selectors. Each selector offers every registered version plus Latest promoted (track the globally promoted release, the default behavior).

Pin precedence is inherit-with-override — deliberately different from the lock model used by the update policy and maintenance window below:

• A partner pin is the inherited default for all of that partner’s organizations.
• Any org can override the partner pin with its own value — including explicitly re-selecting Latest promoted to move that org back to the promoted release while the rest of the fleet stays pinned.
• Agent and watchdog pins resolve independently.

This is what lets a partner pilot a new version on one customer: pin the fleet to the known-good version at the partner level, then override a single org to the candidate version (or to Latest promoted).

Caution

Pins fail closed. If the resolved pin has no registered build for a device’s platform and architecture, the update — including watchdog-recovery pushes — is withheld: the device simply stays on its current version and retries on the next heartbeat. This is surfaced only in server logs and error tracking; there is no dashboard badge for a frozen device. Before pinning broadly, verify the pinned version has builds registered for every platform/architecture in your fleet.

Pins gate alongside the update policy and maintenance window, not instead of them — a pinned update still waits for the maintenance window and respects the update policy.

Agent Update Scheduling

In addition to the per-device auto-update toggle and the partner-level controlled rollout, Breeze supports a structured maintenance window for agent updates. When a maintenance window is configured, agents only accept update signals from the server during the allowed window — outside of it, update prompts are deferred until the next eligible window.

Setting a maintenance window

Agent update maintenance windows are set in Settings → Org Defaults (and optionally in Settings → Partner Defaults to apply fleet-wide). The window configuration includes:

Field	Description
Day	A single day of the week the window applies to, or left unset to apply the window every day — you cannot select more than one specific day
Start time	Window start time in UTC
End time	Window end time in UTC

Note

All times are specified and stored in UTC. Communicate the UTC equivalent to your team to avoid scheduling confusion across time zones.

The update policy and maintenance window follow the same partner-locks precedence that governs other policy settings: a field set at the Partner level wins and locks — child organizations cannot override it. The org value applies only where the partner has left that field unset, and each field is merged independently. The update gate that agents check on heartbeat honors this merged result.

Note

The maintenance window gates when the update signal is acted on, not when the binary is downloaded. Agents can pre-download an available update during any window but will only apply it (restart the service) during the allowed window.

When no window is configured

If no maintenance window is set, agents update on their normal heartbeat cadence (~60 seconds after a new version is promoted). Configure a window when you need updates to avoid business hours or to concentrate restarts into a known change window.

Migration Failure Recovery

If the API container fails to start after an upgrade due to a migration error, follow these steps:

1. Check the container logs to identify the failing migration.

   docker compose logs api --tail 100

   Look for lines starting with [auto-migrate]. Common failure messages include:

   • Migration checksum mismatch – a previously applied migration file was modified. This should never happen in normal operation.
   • SQL errors during a specific migration file (e.g., constraint violations, missing columns).

2. Do not manually edit migration files. Migrations are tracked by checksum. Modifying a previously applied migration will cause all future startups to fail.

3. Fix and restart. The most common causes and resolutions:

   • Incompatible extension (e.g., TimescaleDB not installed): Install the required extension and restart.
   • Permissions issue: Ensure the database user has sufficient privileges.
   • Data conflict: Manually resolve the conflicting data in the database, then restart the API container.

   docker compose restart api

4. If unrecoverable, restore your database from a pre-upgrade backup and re-attempt the upgrade after resolving the underlying issue.

Caution

Migrations run inside a transaction. If a migration fails partway through, the transaction is rolled back and no partial changes are applied. The API will not start until all migrations succeed.

Remote-access lease cutover

The first release that enables shared remote-session ownership is a barrier deployment. Do not use a rolling or blue-green handoff for that release: legacy API processes can mint credentials that the new pre-upgrade admission path must not accept.

1. Set REMOTE_ACCESS_ADMISSION_MODE=closed, REMOTE_WS_AUTH_MODE=post_upgrade, and REMOTE_WS_REDIS_TOPOLOGY=standalone-single-primary. Recreate Caddy and verify every terminal, desktop, VNC, WebSocket-tunnel, and HTTP-tunnel issuer/consumer returns 503 with Retry-After.

2. Stop the entire old API pool. Verify no old API process, WebSocket, or tunnel-HTTP consumer remains reachable. Record the verified UTC stop times in REMOTE_WS_LEGACY_TICKET_WRITER_DRAINED_AT and REMOTE_WS_LEGACY_VIEWER_ISSUER_DRAINED_AT.

3. Keep admission closed for the independent V0 credential drains: 60 seconds for legacy WebSocket tickets and 300 seconds for tunnel-HTTP tickets/cookies. The 60-second milestone does not permit reopen. At exactly 300 seconds, a uniform new pool may reopen in post_upgrade.

4. Deploy the new pool, verify every API instance uses the release digest, and run the Redis preflight (standalone primary, cluster disabled, AOF enabled, maxmemory-policy=noeviction). Only then set REMOTE_ACCESS_ADMISSION_MODE=open and recreate Caddy.

5. Independently wait 7,200 seconds plus a final 60 seconds for legacy viewer tokens and their last compatibility ticket. Only after that boundary may the complete pool switch to REMOTE_WS_AUTH_MODE=pre_upgrade. Startup rejects this mode when either verified timestamp is missing, malformed, future-dated, or too recent.

Danger

A health check is not cutover proof. Any failed probe, mixed image digest, old process/socket, reachable tunnel-HTTP consumer, unsupported Redis mode, or incomplete drain must leave remote-access admission closed.

Zero-Downtime Upgrades

For production deployments where any downtime is unacceptable, you can perform a blue-green upgrade using two parallel Compose stacks:

Caution

Do not use this procedure for the first shared-lease-aware remote-access release. That release requires the closed admission barrier and full-pool replacement above.

1. Deploy the new version alongside the old one. Use a separate Compose project name or directory with the new BREEZE_VERSION and point it at the same database and Redis.

   COMPOSE_PROJECT_NAME=breeze-new BREEZE_VERSION=0.12.0 \
     docker compose -f docker-compose.yml -f docker-compose.override.yml up -d

2. Wait for migrations and health check. The new API container applies any pending migrations on startup. Once the health endpoint returns the expected version, the new stack is ready.

   curl -s http://localhost:NEW_PORT/health | jq .version

3. Switch traffic. Update your reverse proxy (Caddy, nginx, Cloudflare tunnel) to route traffic to the new stack.

4. Tear down the old stack. Once traffic is fully migrated, stop the old containers.

Rolling Back

1. Revert the version

   # In .env
   BREEZE_VERSION=0.11.2

2. Pull and restart

   docker compose pull
   docker compose up -d

Danger

Database migrations are forward-only. If the newer version ran a migration, reverting the API version may cause schema mismatches. Restore from a database backup if needed.

Checking Versions

# API version (the running image's baked version)
curl -s https://breeze.yourdomain.com/health | jq .version

# The version .env says is deployed; must match the line above
grep '^BREEZE_VERSION=' .env

# Agent version (on the device)
breeze-agent version

# Docker image versions
docker compose images

Pre-Release Testing

You can test a release candidate before making it available to users:

1. Tag a prerelease

   git tag v0.11.3-rc.1 && git push --tags

   CI builds all images but does not tag them as latest.

2. Deploy the RC locally

   # In .env
   BREEZE_VERSION=0.11.3-rc.1
   
   docker compose pull
   docker compose up -d

3. Sync agents to the RC using the sync-github endpoint. This fetches the release assets from GitHub and registers them in the Breeze database so agents know a new version is available. It requires a signed-in admin session with MFA satisfied — $TOKEN is a Breeze access token, not an API key.

   curl -X POST "https://your-domain/api/v1/agent-versions/sync-github?version=v0.11.3-rc.1" \
     -H "Authorization: Bearer $TOKEN"

   The response includes the version that was synced and the list of platform/architecture targets that were registered:

   {
     "version": "0.11.3-rc.1",
     "synced": ["agent:linux/amd64", "agent:macos/arm64", "agent:windows/amd64", "helper:windows/amd64"]
   }

   Agents will pick up the new binary on their next heartbeat (~60 seconds). Omit the ?version= parameter to sync the latest stable release instead of a specific version. This endpoint requires system-level (admin) authentication.

4. Test, then promote

   git tag v0.11.3 && git push --tags

   This pushes latest tags — all users get the update.