Incident history

Every incident, newest first, including post-mortems written before automated monitoring started (2026-11-22).

October 2026

Maintenance

Network policy and ledger backup

Oct 9, 2026 · Tenant apps, API

CompletedCarried out alongside the tenant node move: tightened the network policy for build jobs and for the tenant namespace, and repointed the nightly ledger dump at the live database. The database itself had continuous backups throughout.Oct 9, 2026

August 2026

Critical

Tenant compute unavailable

Aug 8, 2026 → Oct 9, 2026 · 63 d · Tenant apps

ResolvedTenant compute moved to a new dedicated node. Network isolation was verified on the node before any workload was admitted, and a deployment was tested end to end through the public API. The new node is x86 (the old one was ARM): workloads built for the old node need a redeploy.Oct 9, 2026

Post-mortem

Impact window: 2026-08-08 to 2026-10-09.

What happened. Tenant compute was unavailable for the whole window: new deployments could not start and tenant workloads were not running. The website, API, dashboard and sign-in stayed up, and no account or ledger data was lost.

Cause. The hosting account for our single tenant node lapsed and the node was lost. Monitoring flagged it, but the alert was not acted on for two months.

Fix. On 2026-10-09 tenant compute moved to a new dedicated node at a different provider. Network isolation was verified on the node before any workload was admitted, and a deployment was tested end to end through the public API.

What you need to do. The new node is x86; the old one was ARM. Workloads built for the old node need a redeploy.