Skip to content

fix(ns-ha): move HA rsync staging to tmpfs, sync every 10 minutes#1772

Merged
gsanchietti merged 2 commits into
mainfrom
issue1771
Jul 8, 2026
Merged

fix(ns-ha): move HA rsync staging to tmpfs, sync every 10 minutes#1772
gsanchietti merged 2 commits into
mainfrom
issue1771

Conversation

@gsanchietti

@gsanchietti gsanchietti commented Jul 7, 2026

Copy link
Copy Markdown
Member

Fixes #1771

Prevents HA config corruption (null bytes) on failover, and stops the telegraf log flood on the backup node.

1. HA rsync staging on tmpfs

  • Move rsync staging dir from /usr/share/keepalived/rsync (overlayfs) to /tmp/ns-ha (tmpfs)
  • Reduce sync interval 60s → 600s
  • Fix sender mkdir so tmpfs staging dir is recreated each run (survives reboot)
  • uci-defaults migration for existing installs (both nodes)

2. Disable telegraf internet inputs on the HA backup node

The backup node has no Internet access and dnsmasq answers only locally, so telegraf's inputs.ping and inputs.dns_query flood /var/log/messages with network is unreachable / DNS REFUSED errors.

  • New keepalived hotplug 900-telegraf: on NOTIFY_BACKUP move the generated ping.conf/dns.conf drop-ins aside; on NOTIFY_MASTER restore them
  • UCI config untouched (nothing lost); master keeps internet monitoring

Testing

Tested on a live 2-node cluster (primary 192.168.100.238, backup 192.168.100.239, VIP 192.168.100.240), configured via ns-ha-config per the ns-ha README.

tmpfs staging (#1)

  • Config applied: interval=600, sync_dir=/tmp/ns-ha, keepalived.conf vrrp interval 600 on both nodes; no legacy /usr/share/keepalived/rsync
  • Sync Successful, staging lands in /tmp/ns-ha (159 files); dir is on tmpfs, staged configs have zero null bytes
  • NOTIFY_SYNC restore copies staged files to real paths (verified with a custom sync_list marker)
  • Failover both directions (VIP + master move to backup and back)
  • Reboot backup → /tmp/ns-ha wiped by tmpfs → next sync recreates it (validates the mkdir "$sync_dir" fix)

telegraf inputs on backup (#2)

  • NOTIFY_BACKUPping.conf/dns.conf become *.disabled; telegraf reloads without those inputs
  • Causal check: with inputs enabled the ping/dns errors reappear within seconds; after disable, zero new errors
  • NOTIFY_MASTER restores the active drop-ins
  • Real failover: backup→master re-enables inputs, master→backup disables them; primary keeps inputs active throughout

Comment thread packages/ns-api/files/ns.ha
@gsanchietti gsanchietti marked this pull request as ready for review July 8, 2026 06:29
Comment thread packages/ns-ha/Makefile Outdated
Prevent HA config file corruption (null bytes) on slave failover
caused by concurrent rsync writes to the persistent overlayfs.

Move the rsync staging area from /usr/share/keepalived/rsync to
/tmp/ns-ha (tmpfs) and reduce the sync interval from 60s to 600s.
Fix the sender mkdir so the tmpfs staging dir is recreated on each
run, since tmpfs is emptied at reboot. Add a uci-defaults migration
that updates interval and sync_dir on both nodes for existing
installs and drops the legacy staging directory.

Refs #1771

Assisted-by: Claude Code:claude-opus-4-8
The backup node has no Internet access and dnsmasq answers only
locally, so telegraf's inputs.ping and inputs.dns_query continuously
flood the log with "network is unreachable" and DNS "REFUSED" errors.

Add a keepalived hotplug script that moves the generated ping.conf and
dns.conf drop-ins aside on NOTIFY_BACKUP and restores them on
NOTIFY_MASTER. The UCI config is left untouched, so nothing is lost and
the master node keeps its internet monitoring.

Refs #1771

Assisted-by: Claude Code:claude-opus-4-8
@gsanchietti gsanchietti merged commit fe36b63 into main Jul 8, 2026
3 checks passed
@gsanchietti gsanchietti deleted the issue1771 branch July 8, 2026 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HA: Corrupted config files with null bytes on slave failover

2 participants