v3.0.0

Major release — Consul is fully removed from the agent. The embedded SQLite control.db
becomes the sole coordination plane for tasks, volumes, firewall rules, borg repositories, and
backup schedules. This completes the Consul-retirement / node-autonomy re-architecture. One
big-bang cutover in a maintenance window (controller-off-first); coordinated with a controller
release (see the controller repo's changelog for its side).

• [CHANGE] Consul removed. The binary no longer links hashicorp/consul/api and never
  contacts Consul. control.db is the source of truth for all node coordination state.
• [CHANGE] In-process task dispatch. A single dispatcher goroutine drains pending tasks
  from control.db with an at-most-once claim (status CAS), replacing the Consul long-poll job
  watcher. Task kinds: volume.backup, volume.restore, backup.delete, backup.export,
  volume.trash. A boot reconcile fails orphaned running tasks; restore/delete/export/trash
  never auto-replay.
• [CHANGE] Durable backup scheduler. A schedules table + tick loop fires backups
  exactly once (with a single catch-up and no backfill storm), replacing the in-memory cron
  runner and the Consul schedule mirror. robfig/cron is kept only as the cron-string parser.
• [FEATURE] Controller DOWN endpoints (per-node admin Bearer): POST /v1/admin/tasks
  (controller-supplied idempotent id), PUT/DELETE /v1/admin/projects/{pid}/volumes/{name},
  PUT/DELETE /v1/admin/nodes/{host}/firewall_rules (firewall reconciles on the PUT — there
  is no firewall task), and POST /v1/admin/changelog/ack. UP state (task status + results,
  observed repositories) rides the existing GET /v1/admin/changelog pull.
• [CHANGE] csevent retired. The agent no longer POSTs to /api/system/events;
  backup/restore/export/delete outcomes ride the task's status + result_json (terminal
  outcome plus captured failure output — no live per-step stream).
• [CHANGE] Populate-before-enforce (no cutover outage). On a fresh/empty node the firewall
  and volume domains skip reconcile — leaving the live cs_agent nftables table and running
  workloads untouched — until the controller backfills desired state; cross-project isolation
  still applies. Published ports stay up across the upgrade gap.
• [FEATURE] Bounded control.db. Changelog prune (ack-gated, plus an ack-independent age
  fallback) and terminal task-row retention keep the DB bounded even before the controller
  starts acking.
• [REMOVED] consul.* config, the Consul auth proxy (proxy_to_consul), the schedule_source
  and cutover.* options, and the consul.service dependency in the systemd unit.

Migrations run additively (control.db schema v1→v4); an older binary refuses a v4 DB via
the schema-version guard, so downgrades require restoring a pre-upgrade snapshot.

Upgrading a node to v3.0.0

Maintenance window, controller-off-first. The rollback anchors are a per-node control.db
snapshot and Consul left running until the very end. No agent.yml changes are required —
metadata.admin_token_hash (already set) authenticates the controller's DOWN writes, and any
leftover consul: keys are ignored.

1. Pre-window (per node): snapshot the DB and drain in-flight jobs. Leave Consul running.

   cp -a /var/lib/cs-agent/control.db /var/lib/cs-agent/control.db.pre-v3
   

2. Stop the old controller (once). No more Consul writes; customer containers keep running.
3. Install v3.0.0 on all nodes and start. Migrations run v1→v4; the agent boots with
   empty coordination tables and, sentinels unlatched, leaves the live firewall table +
   workloads untouched.

   sudo apt-get update && sudo apt-get install -y cs-agent
   

4. Boot the new controller (once). It backfills each node (firewall rules + every volume)
   via the DOWN endpoints. The sentinels latch and the agent reconciles: the firewall
   re-renders the same rules, and schedules rebuild with a future next_fire_at (no backup
   storm).
5. Verify per node: cs-agent -version (v3.0.0), control.db at schema v4, firewall
   renders (nft list table inet cs_agent), tasks dispatch, and task status/results flow up
   the changelog with ack + prune working.
6. Tear down Consul (per node, once confirmed) and remove it from the provisioner. The v3
   agent has no Consul client and its unit no longer orders after consul.service, so this is
   safe; the old :8502 HTTP relocation is obsolete.

   sudo systemctl disable --now consul
   

Rollback (until step 6, all nodes): restore the control.db.pre-v3 snapshot and
downgrade the binary — order matters: an old binary against a migrated (v4) DB refuses to
boot, so restore the snapshot first (or do both atomically). Consul is still live as the other
anchor. Restart the old controller last.

v2.0.0

Major release — the agent becomes the node's data plane (part of the Consul-retirement /
node-autonomy re-architecture). Three independent changes ship together; production rolls out staged
(native deploy first, then the firewall and metadata cutovers, validated on a canary).

• [CHANGE] Native deployment. The agent now runs as a native systemd binary installed from a
  self-hosted, GPG-signed apt repo, replacing the docker run container unit. The container image
  is kept for local dev only. cs-agent -version reports the build version/commit/date.
• [CHANGE] nftables firewall. Published-port DNAT/forwarding is rendered into a native cs_agent
  nftables table via netlink, replacing the iptables shell-out + string-diff. Cross-project isolation
  stays in DOCKER-USER. Fail-closed: published ports are closed until the first reconcile. Reads
  the same ingress_rules desired state. (Relies on the project bridges' nat-unprotected mode, under
  which Docker already accepts the forwarded ingress.)
• [FEATURE] Customer metadata served by the agent. A new HTTP API on node.primary_ip:8500 serves
  per-project customer metadata from embedded SQLite — no more Consul KV for the /db/ space, and
  no value size cap (kills the 512 KB ceiling). Bearer→tenant auth + a per-node admin Bearer; a
  compatibility shim serves the legacy …/metadata?raw=true read. Migrations are rollback-tolerant
  (additive + schema-version guard).

Upgrading a node to v2.0.0

Take a maintenance window — the firewall cutover (+ optional reboot) briefly closes published ports.
All nodes must be Debian 12/13 (iptables = the nft backend). Snapshot the firewall first:
iptables-save > /root/iptables.pre-upgrade.

1. Native binary — add the apt source + keyring, stop the old container unit, install:

   curl -fsSL https://repo.computestacks.com/public/computestacks.gpg.asc \
     | gpg --dearmor | sudo tee /etc/apt/keyrings/computestacks.gpg >/dev/null
   echo "deb [signed-by=/etc/apt/keyrings/computestacks.gpg] https://repo.computestacks.com/public stable main" \
     | sudo tee /etc/apt/sources.list.d/computestacks.list
   sudo systemctl disable --now cs-agent; docker rm -f cs-agent 2>/dev/null || true
   sudo rm -f /etc/systemd/system/cs-agent.service   # the package unit lives in /lib/systemd/system
   sudo apt-get update && sudo apt-get install -y cs-agent
   

2. Metadata / Consul port — the agent binds :8500, so Consul's HTTP listener moves to :8502
   (provisioner); confirm the agent's consul.host + the admin-token hash are configured. New
   containers receive CS_NODE_ID; existing ones use the compatibility shim — no recreation needed.
3. Firewall — the agent renders the cs_agent nft table on start (nft list table ip cs_agent).
   The host firewall itself is applied at boot by cs-iptables.service (a oneshot that runs
   /usr/local/bin/cs-recover_iptables); the agent does not manage that file. Edit that file
   directly to delete the lines the agent has now taken over — the expose-ports/container-inbound chain setup — then reboot
   so the oneshot re-applies the trimmed ruleset from a clean slate (or, to avoid a reboot, delete
   those rules from the live ruleset by hand). Verify published ports still reach containers and
   iptables -S shows none of the old expose-ports/container-inbound artifacts.
   • Rollback — v2.0.0 is the first native release, so there is no previous .deb; the
     prior version ran as a Docker container, so rolling back means undoing the deployment-model
     change, not just downgrading a package:
     1. sudo apt-get purge cs-agent (removes the native binary + the /lib/systemd/system unit).
     2. Restore the old containerized cs-agent.service (the docker run unit) and pull the agent
        image — i.e. re-apply the previous provisioner config.
     3. Restore the host firewall: sudo iptables-restore < /root/iptables.pre-upgrade, and
        revert /usr/local/bin/cs-recover_iptables to the version that re-creates the
        expose-ports/container-inbound chains — the old containerized
        agent appends to those chains and silently loses published ports without them.
     4. Re-bind Consul's HTTP listener to :8500 (the old agent and customer containers reach
        metadata via Consul there).
        From v2.0.1 onward rollback is a normal apt-get install --allow-downgrades cs-agent=<prev>;
        never roll back across a non-additive DB migration.

The controller/provisioner changes (CS_NODE_ID injection, the Consul port move, the host-firewall
trim, the apt source) ship alongside — coordinate per the rollout runbook.

v2.0.0-rc1

Major release — the agent becomes the node's data plane (part of the Consul-retirement /
node-autonomy re-architecture). Three independent changes ship together; production rolls out staged
(native deploy first, then the firewall and metadata cutovers, validated on a canary).

• [CHANGE] Native deployment. The agent now runs as a native systemd binary installed from a
  self-hosted, GPG-signed apt repo, replacing the docker run container unit. The container image
  is kept for local dev only. cs-agent -version reports the build version/commit/date.
• [CHANGE] nftables firewall. Published-port DNAT/forwarding is rendered into a native cs_agent
  nftables table via netlink, replacing the iptables shell-out + string-diff. Cross-project isolation
  stays in DOCKER-USER. Fail-closed: published ports are closed until the first reconcile. Reads
  the same ingress_rules desired state. (Relies on the project bridges' nat-unprotected mode, under
  which Docker already accepts the forwarded ingress.)
• [FEATURE] Customer metadata served by the agent. A new HTTP API on node.primary_ip:8500 serves
  per-project customer metadata from embedded SQLite — no more Consul KV for the /db/ space, and
  no value size cap (kills the 512 KB ceiling). Bearer→tenant auth + a per-node admin Bearer; a
  compatibility shim serves the legacy …/metadata?raw=true read. Migrations are rollback-tolerant
  (additive + schema-version guard).

Upgrading a node to v2.0.0

Take a maintenance window — the firewall cutover (+ optional reboot) briefly closes published ports.
All nodes must be Debian 12/13 (iptables = the nft backend). Snapshot the firewall first:
iptables-save > /root/iptables.pre-upgrade.

1. Native binary — add the apt source + keyring, stop the old container unit, install:

   curl -fsSL https://repo.computestacks.com/public/computestacks.gpg.asc \
     | gpg --dearmor | sudo tee /etc/apt/keyrings/computestacks.gpg >/dev/null
   echo "deb [signed-by=/etc/apt/keyrings/computestacks.gpg] https://repo.computestacks.com/public stable main" \
     | sudo tee /etc/apt/sources.list.d/computestacks.list
   sudo systemctl disable --now cs-agent; docker rm -f cs-agent 2>/dev/null || true
   sudo rm -f /etc/systemd/system/cs-agent.service   # the package unit lives in /lib/systemd/system
   sudo apt-get update && sudo apt-get install -y cs-agent
   

2. Metadata / Consul port — the agent binds :8500, so Consul's HTTP listener moves to :8502
   (provisioner); confirm the agent's consul.host + the admin-token hash are configured. New
   containers receive CS_NODE_ID; existing ones use the compatibility shim — no recreation needed.
3. Firewall — the agent renders the cs_agent nft table on start (nft list table ip cs_agent).
   The host firewall itself is applied at boot by cs-iptables.service (a oneshot that runs
   /usr/local/bin/cs-recover_iptables); the agent does not manage that file. Edit that file
   directly to delete the lines the agent has now taken over — the 10000:50000 INPUT range and
   the expose-ports/container-inbound chain setup — then reboot
   so the oneshot re-applies the trimmed ruleset from a clean slate (or, to avoid a reboot, delete
   those rules from the live ruleset by hand). Verify published ports still reach containers and
   iptables -S shows none of the old expose-ports/container-inbound/10000:50000 artifacts.
   • Rollback — v2.0.0 is the first native release, so there is no previous .deb; the
     prior version ran as a Docker container, so rolling back means undoing the deployment-model
     change, not just downgrading a package:
     1. sudo apt-get purge cs-agent (removes the native binary + the /lib/systemd/system unit).
     2. Restore the old containerized cs-agent.service (the docker run unit) and pull the agent
        image — i.e. re-apply the previous provisioner config.
     3. Restore the host firewall: sudo iptables-restore < /root/iptables.pre-upgrade, and
        revert /usr/local/bin/cs-recover_iptables to the version that re-creates the
        expose-ports/container-inbound chains + the 10000:50000 range — the old containerized
        agent appends to those chains and silently loses published ports without them.
     4. Re-bind Consul's HTTP listener to :8500 (the old agent and customer containers reach
        metadata via Consul there).
        From v2.0.1 onward rollback is a normal apt-get install --allow-downgrades cs-agent=<prev>;
        never roll back across a non-additive DB migration.

The controller/provisioner changes (CS_NODE_ID injection, the Consul port move, the host-firewall
trim, the apt source) ship alongside — coordinate per the rollout runbook.