fix(client create): never mint over a live cluster — fail closed on unreadable discovery#190
Open
LukasWodka wants to merge 1 commit into
Open
fix(client create): never mint over a live cluster — fail closed on unreadable discovery#190LukasWodka wants to merge 1 commit into
LukasWodka wants to merge 1 commit into
Conversation
…nreadable discovery
`tracebloc client create` could mint a NEW backend client and stamp the
cluster's cluster_id anchor onto it, even when a healthy DIFFERENT client was
already running on the cluster. The installer then refused to deploy the new
client (one-client-per-machine), leaving an orphaned "phantom" that owns the
anchor — so every later re-provision 409s ("cluster_conflict", mislabeled as
cross-account) and the real client can never reclaim the anchor. Confirmed in
the field: edge_device 1060 minted + anchored, never deployed, wedging a
cluster that actually runs 1044.
Root of the reproducible class: DiscoverInClusterClientID swallowed List/RBAC
errors into (nil, nil) — "nothing installed" — which is indistinguishable from
a genuinely fresh cluster, so runClientCreate fell through to a mint.
Fix (two edits):
- cluster/discover.go: DiscoverInClusterClientID is now three-valued. It
returns (nil, err) when it CANNOT determine — a deployments List error, a
secrets List error, or a release present whose CLIENT_ID is unreadable.
(nil, nil) now means only "reachable and genuinely no client". An empty
cluster still reports emptiness via an empty list, not an error, so fresh
installs are unaffected.
- cli/client.go: adoptLiveInClusterClient fails closed on a discovery error
when the cluster is REACHABLE (clusterID != "", i.e. the kube-system UID read
succeeded over the same kubeconfig) — refusing to mint a duplicate that could
strand the anchor. Only a genuinely unreachable cluster (clusterID == "",
where the UID read failed too) keeps the old fall-through to a non-anchored
mint (which stamps no anchor, so it can't orphan one) — the deliberate
headless/no-cluster path.
Tests (verified to FAIL against the pre-fix code):
- discover_test.go: DeploymentsListError / SecretsListError / ReleaseButNoSecret
now expect (nil, error).
- client_test.go: DiscoveryErrorReachableFailsClosed (reachable + discovery
error -> exitError, no mint) and DiscoveryErrorUnreachableMintsNonAnchored
(unreachable -> still mints, cluster_id empty) — the latter guards against
over-tightening into the legitimate headless path.
Full repo suite green; adversarially reviewed. This is fix #1 of the
phantom-client investigation (never mint over a live cluster). Follow-ups
(separate): backend same-account re-anchor + honest 409 message; orphan reaper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
e4a73c2 to
8a6f721
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tracebloc client createcould mint a new backend client and stamp the cluster'scluster_idanchor onto it while a healthy different client was already running on that cluster. The installer then refused to deploy the new client (one-client-per-machine guard), leaving an orphaned phantom that owns the anchor. Every later re-provision then409s (cluster_conflict, mislabeled "different account"), and the real client can never reclaim the anchor — the machine is wedged with no self-service recovery.Confirmed in the field on a dev laptop:
edge_device 1060was minted + anchored to the cluster but never deployed; the cluster actually runsedge_device 1044. Full investigation trail: the phantom was born by a pre-R7 CLI (v0.5.1) minting straight through the backend's get-or-create against an unclaimed anchor.Root cause of the reproducible class
DiscoverInClusterClientIDswallowed List/RBAC errors into(nil, nil)— "nothing installed" — which is indistinguishable from a genuinely fresh cluster, sorunClientCreatefell through to a mint. A transient/RBAC discovery blip over a live cluster was enough to mint a duplicate and strand the anchor.Fix (two edits)
internal/cluster/discover.go—DiscoverInClusterClientIDis now three-valued:(client, nil)— a live client was found;(nil, nil)— reachable and genuinely no client (an empty cluster reports this via an empty list, not an error);(nil, err)— couldn't determine (deployments List error, secrets List error, or a release present with an unreadableCLIENT_ID).internal/cli/client.go—adoptLiveInClusterClientfails closed on a discovery error when the cluster is reachable (clusterID != ""— thekube-systemUID read succeeded over the same kubeconfig). Only a genuinely unreachable cluster (clusterID == "", UID read failed too) keeps the old fall-through to a non-anchored mint — which stamps no anchor, so it can't orphan one (the deliberate headless path).Why it's safe
Fresh installs use an admin kubeconfig; discovery lists succeed and return empty →
(nil,nil)→ mint. A List only errors on RBAC-denial or transport failure — never as the way an empty cluster reports emptiness — so no normal fresh install is blocked.clusterID != ""reliably implies reachability becausereadClusterIDandreadInClusterClientshare the sameLoad+NewClientsetpath (cluster/identity.go). The only non-test caller of the discovery functions is the adopt path itself (verified).Tests
Verified to fail against the pre-fix code:
discover_test.go:DeploymentsListError/SecretsListError/ReleaseButNoSecretnow expect(nil, error).client_test.go:DiscoveryErrorReachableFailsClosed(reachable + discovery error → exitError, no mint), andDiscoveryErrorUnreachableMintsNonAnchored(unreachable → still mints with emptycluster_id) — the latter guards against over-tightening into the legitimate headless path.Full repo
go test ./...green;go vetclean; adversarially reviewed.Type / scope
Bug fix (correctness / data-integrity) ·
cli. Fix #1 of the phantom-client investigation — never mint over a live cluster. Separate follow-ups (not in this PR): backend same-account re-anchor + honestcluster_conflictmessage (stop saying "different account" when it isn't), and an orphan reaper (backend#970) for anchored-but-never-deployed clients.