feat: hot-reload config.toml via polling

[taylor-osler-sentry]

Summary

Adds hot-reloading of config.toml: when the file changes on disk, config-derived Django settings are re-applied in place without a restart.

Mechanism: a daemon thread runs inside every Django process (started from IncidentsConfig.ready) and periodically reads config.toml, comparing a sha256 of its contents. Each process holds its own in-memory settings, so the watcher must run per-process to propagate changes to web workers, the Slack bot, and q-cluster workers. A standalone watcher process could only mutate its own settings.

Why polling, not inotify: Cloud Run runs under the gVisor sandbox, which does not deliver inotify filesystem events (an event-based watchfiles/notify watcher just logs rust notify timeout and never fires). Hashing file contents (vs comparing mtime) also transparently handles the symlink swap that Kubernetes uses when updating a mounted ConfigMap.

Changes

• settings.py — refactor so all config-derived settings come from a single _build_config_settings(config) (one source of truth for import-time and reload). Add:
  • apply_config(config) — reload entrypoint. Rebuilds + validates first (a bad config never partially applies), rebinds each setting wholesale onto the live settings object (GIL-atomic, no torn reads), re-runs logging.config.dictConfig, re-inits Datadog (not Sentry — see below), and refreshes the DB connection handler cache so new DATABASES take effect.
  • CONFIG setting exposing the active ConfigFile (used to pass old to new to the hook).
  • CONFIG_WATCH_ENABLED — gates the watcher off for one-shot management commands and pytest.
  • CONFIG_WATCH_POLL_SECONDS — poll interval (default 15s, env-overridable).
  • Config-derived names kept as annotated assignments so django-stubs/mypy still see every setting's type.
• config_reload.py — start_config_watcher() (idempotent, per-process), the poll loop (_poll_loop / _hash_file), and reload_config(). A persistently-invalid file isn't retried every tick; it reloads on the next edit. Load/apply/hook failures are logged, not fatal. Re-arms the watcher in forked children (django-q workers, preloaded gunicorn workers) via os.register_at_fork, since fork() doesn't clone the watcher thread.
• config_hooks.py — on_config_reload(old_config, new_config) extension point.
• incidents/apps.py — start the watcher from ready() when enabled.
• test_config_reload.py — tests for apply_config, reload_config, and the poll/hash detection loop.

Scope / risks

Reload covers everything derived from config, including SECRET_KEY / SALT_KEY / DATABASES:

• Rotating SECRET_KEY invalidates active sessions/signed cookies.
• Rotating SALT_KEY breaks decryption of existing encrypted DB fields unless the old key stays in salt_keys.
• DATABASES changes take effect on the next connection: the reload refreshes the connection handler's cached settings, and each worker retires its own connection via the normal end-of-request cycle. In-flight requests finish on their existing connection.

Datadog is re-initialized on each reload; Sentry is not — re-running sentry_sdk.init on a live process is unsupported, so sentry_dsn changes require a restart. Changes are detected within one poll interval (default 15s) rather than instantly.

Testing

• mypy src, ruff check, ruff format, full pre-commit: pass.
• Full suite: 1240 tests pass (requires local Postgres).

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1ea3613. Configure here.}

[linear-code]

RELENG-946

[sentry-warden]

Config watcher polling interval lacks lower bound

CONFIG_WATCH_POLL_SECONDS lacks a positive lower bound, so a misconfigured value of 0 or negative causes _poll_loop to spin-wait and max out CPU.

Evidence

• settings.py line 633 defines CONFIG_WATCH_POLL_SECONDS from the environment via float() without clamping or validation.
• config_reload._poll_loop at line 91 passes the interval directly to _stop_event.wait(interval); Event.wait(0) (or negative) returns immediately, creating a busy loop.
• start_config_watcher() spawns the thread that executes this loop inside every Django process.
• Default is 15s, but a bad env value silently exhausts CPU in every process.

Also found at 1 additional location

• src/firetower/settings.py:633

_{Identified by Warden · find-bugs · T9Q-X8C}