Skip to content

fix(client create): never mint over a live cluster — fail closed on unreadable discovery#190

Open
LukasWodka wants to merge 1 commit into
developfrom
fix/no-mint-over-live-cluster
Open

fix(client create): never mint over a live cluster — fail closed on unreadable discovery#190
LukasWodka wants to merge 1 commit into
developfrom
fix/no-mint-over-live-cluster

Conversation

@LukasWodka

Copy link
Copy Markdown
Contributor

Summary

tracebloc client create could mint a new backend client and stamp the cluster's cluster_id anchor onto it while a healthy different client was already running on that cluster. The installer then refused to deploy the new client (one-client-per-machine guard), leaving an orphaned phantom that owns the anchor. Every later re-provision then 409s (cluster_conflict, mislabeled "different account"), and the real client can never reclaim the anchor — the machine is wedged with no self-service recovery.

Confirmed in the field on a dev laptop: edge_device 1060 was minted + anchored to the cluster but never deployed; the cluster actually runs edge_device 1044. Full investigation trail: the phantom was born by a pre-R7 CLI (v0.5.1) minting straight through the backend's get-or-create against an unclaimed anchor.

Root cause of the reproducible class

DiscoverInClusterClientID swallowed List/RBAC errors into (nil, nil)"nothing installed" — which is indistinguishable from a genuinely fresh cluster, so runClientCreate fell through to a mint. A transient/RBAC discovery blip over a live cluster was enough to mint a duplicate and strand the anchor.

Fix (two edits)

  • internal/cluster/discover.goDiscoverInClusterClientID is now three-valued:
    • (client, nil) — a live client was found;
    • (nil, nil) — reachable and genuinely no client (an empty cluster reports this via an empty list, not an error);
    • (nil, err)couldn't determine (deployments List error, secrets List error, or a release present with an unreadable CLIENT_ID).
  • internal/cli/client.goadoptLiveInClusterClient fails closed on a discovery error when the cluster is reachable (clusterID != "" — the kube-system UID read succeeded over the same kubeconfig). Only a genuinely unreachable cluster (clusterID == "", UID read failed too) keeps the old fall-through to a non-anchored mint — which stamps no anchor, so it can't orphan one (the deliberate headless path).

Why it's safe

Fresh installs use an admin kubeconfig; discovery lists succeed and return empty → (nil,nil) → mint. A List only errors on RBAC-denial or transport failure — never as the way an empty cluster reports emptiness — so no normal fresh install is blocked. clusterID != "" reliably implies reachability because readClusterID and readInClusterClient share the same Load+NewClientset path (cluster/identity.go). The only non-test caller of the discovery functions is the adopt path itself (verified).

Tests

Verified to fail against the pre-fix code:

  • discover_test.go: DeploymentsListError / SecretsListError / ReleaseButNoSecret now expect (nil, error).
  • client_test.go: DiscoveryErrorReachableFailsClosed (reachable + discovery error → exitError, no mint), and DiscoveryErrorUnreachableMintsNonAnchored (unreachable → still mints with empty cluster_id) — the latter guards against over-tightening into the legitimate headless path.

Full repo go test ./... green; go vet clean; adversarially reviewed.

Type / scope

Bug fix (correctness / data-integrity) · cli. Fix #1 of the phantom-client investigation — never mint over a live cluster. Separate follow-ups (not in this PR): backend same-account re-anchor + honest cluster_conflict message (stop saying "different account" when it isn't), and an orphan reaper (backend#970) for anchored-but-never-deployed clients.

…nreadable discovery

`tracebloc client create` could mint a NEW backend client and stamp the
cluster's cluster_id anchor onto it, even when a healthy DIFFERENT client was
already running on the cluster. The installer then refused to deploy the new
client (one-client-per-machine), leaving an orphaned "phantom" that owns the
anchor — so every later re-provision 409s ("cluster_conflict", mislabeled as
cross-account) and the real client can never reclaim the anchor. Confirmed in
the field: edge_device 1060 minted + anchored, never deployed, wedging a
cluster that actually runs 1044.

Root of the reproducible class: DiscoverInClusterClientID swallowed List/RBAC
errors into (nil, nil) — "nothing installed" — which is indistinguishable from
a genuinely fresh cluster, so runClientCreate fell through to a mint.

Fix (two edits):
- cluster/discover.go: DiscoverInClusterClientID is now three-valued. It
  returns (nil, err) when it CANNOT determine — a deployments List error, a
  secrets List error, or a release present whose CLIENT_ID is unreadable.
  (nil, nil) now means only "reachable and genuinely no client". An empty
  cluster still reports emptiness via an empty list, not an error, so fresh
  installs are unaffected.
- cli/client.go: adoptLiveInClusterClient fails closed on a discovery error
  when the cluster is REACHABLE (clusterID != "", i.e. the kube-system UID read
  succeeded over the same kubeconfig) — refusing to mint a duplicate that could
  strand the anchor. Only a genuinely unreachable cluster (clusterID == "",
  where the UID read failed too) keeps the old fall-through to a non-anchored
  mint (which stamps no anchor, so it can't orphan one) — the deliberate
  headless/no-cluster path.

Tests (verified to FAIL against the pre-fix code):
- discover_test.go: DeploymentsListError / SecretsListError / ReleaseButNoSecret
  now expect (nil, error).
- client_test.go: DiscoveryErrorReachableFailsClosed (reachable + discovery
  error -> exitError, no mint) and DiscoveryErrorUnreachableMintsNonAnchored
  (unreachable -> still mints, cluster_id empty) — the latter guards against
  over-tightening into the legitimate headless path.

Full repo suite green; adversarially reviewed. This is fix #1 of the
phantom-client investigation (never mint over a live cluster). Follow-ups
(separate): backend same-account re-anchor + honest 409 message; orphan reaper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant