Suggested title
X509FederationClient leaves inFlightRefresh incomplete when the refresh leader fails before cleanup
Description
Summary
X509FederationClient can become permanently stuck after an exception during session-key or
certificate refresh.
The single-flight implementation assigns a new CompletableFuture to inFlightRefresh, but several
operations that can throw are executed before the try/catch/finally responsible for completing and
clearing that future.
If one of those operations throws:
- The Leader exits without completing
inFlightRefresh.
inFlightRefresh remains non-null and incomplete.
- Every subsequent caller becomes a Follower.
- Each Follower waits one minute and fails with
Timed out waiting for security token refresh.
- The affected
X509FederationClient does not recover. Recreating the authentication provider or
restarting the process is required.
Affected versions
Confirmed by a standalone reproducer:
oci-java-sdk-common:2.91.0
oci-java-sdk-common:2.93.0
Production stack traces from both observed incidents reported Oracle-JavaSDK/2.92.0.
The issue was introduced by commit
6a737779038869e1ca2948868a225736befdae0e,
which released version 2.91.0 and added the single-flight implementation.
The relevant file is unchanged between public tags v2.91.0 and v2.93.0.
This can be verified with:
git diff v2.91.0 v2.93.0 -- \
bmc-common/src/main/java/com/oracle/bmc/auth/internal/X509FederationClient.java
The command produces no output.
Source-code evidence
In
X509FederationClient.refreshAndGetSecurityTokenInner,
the Leader publishes the incomplete future:
future = new CompletableFuture<>();
inFlightRefresh = future;
iAmTheLeader = true;
The following operations are then executed before the cleanup try/finally:
sessionKeySupplier.refreshKeys();
((Refreshable) leafCertificateSupplier).refresh();
AuthUtils.getTenantIdFromCertificate(...);
((Refreshable) supplier).refresh();
The cleanup block starts only later:
try {
securityTokenAdapter = getSecurityTokenFromServer();
String token = securityTokenAdapter.getSecurityToken();
future.complete(token);
return token;
} catch (Exception e) {
future.completeExceptionally(e);
throw new BmcException(false, "Error refreshing security token.", e, null);
} finally {
inFlightRefresh = null;
}
An exception thrown before this try bypasses both:
future.completeExceptionally(e);
inFlightRefresh = null;
The leaf-certificate block catches only RefreshFailedException. In the production failures described
below, URLBasedX509CertificateSupplier exhausted its retries and threw:
java.lang.IllegalArgumentException: Open stream of certificate failed.
Caused by: java.io.FileNotFoundException:
http://169.254.169.254/opc/v2/identity/cert.pem
This exception escaped at:
URLBasedX509CertificateSupplier.refresh
X509FederationClient.refreshAndGetSecurityTokenInner:248
Before the introducing commit, refresh was performed entirely under the existing synchronized block
and there was no persistent inFlightRefresh state that could be left incomplete.
Minimal reproduction
Use this dependency:
<dependency>
<groupId>com.oracle.oci.sdk</groupId>
<artifactId>oci-java-sdk-common</artifactId>
<version>2.93.0</version>
</dependency>
- Construct an
X509FederationClient with an X509CertificateSupplier that also implements
javax.security.auth.Refreshable.
- Make its
refresh() method throw IllegalArgumentException, matching the terminal exception from
URLBasedX509CertificateSupplier.
- Call
refreshAndGetSecurityToken(). This caller becomes the Leader and exits with the simulated
certificate failure.
- Inspect
inFlightRefresh: it is non-null and isDone() is false.
- Call
refreshAndGetSecurityToken() from another thread. It becomes a Follower and times out after
one minute.
Observed output using the real oci-java-sdk-common:2.93.0 artifact:
Loaded SDK artifact: .../oci-java-sdk-common/2.93.0/oci-java-sdk-common-2.93.0.jar
[EXPECTED FAILURE]
Leader stopped with: Simulated failure reading IMDS cert.pem
[SDK STATE]
Does inFlightRefresh still exist? YES
Has inFlightRefresh completed? NO
[FOLLOWER RESULT]
Exception: com.oracle.bmc.model.BmcException
Message: (-1, null, false) Timed out waiting for security token refresh.
VERDICT: BUG REPRODUCED
The reproducer does not require OCI credentials or network access.
Production evidence
The same sequence was observed independently in two long-running Java processes on OCI compute
instances:
[Leader] About to refresh security token.
Refreshing session keys.
Attempt 1 to open stream of certificate failed.
Attempt 2 to open stream of certificate failed.
Attempt 3 to open stream of certificate failed.
java.lang.IllegalArgumentException: Open stream of certificate failed.
Caused by: java.io.FileNotFoundException:
http://169.254.169.254/opc/v2/identity/cert.pem
After the Leader terminated, subsequent callers repeatedly logged:
[Follower] Waiting on leader's result.
com.oracle.bmc.model.BmcException:
Timed out waiting for security token refresh.
Caused by: java.util.concurrent.TimeoutException
In one occurrence, the logs contained 62 Follower timeouts. Authenticated Monitoring operations
continued failing until the process restarted. Restarting created a new authentication provider and
immediately restored token refresh.
There is an additional timeout mismatch: the certificate supplier's three retries can take longer
than the hard-coded one-minute Follower wait, so Followers may time out even while a slow Leader is
still retrying. The persistent failure described in this issue remains after that Leader has exited
because its future is never completed or cleared.
Expected behavior
Every Leader exit path should:
- Complete the shared future, normally or exceptionally.
- Clear
inFlightRefresh.
- Allow a later caller to become a new Leader and retry.
Followers should receive the Leader's original exception rather than wait for a future that can never
complete.
Suggested fix
Move all Leader operations under the same try/catch/finally:
if (iAmTheLeader) {
try {
if (refreshKeys) {
sessionKeySupplier.refreshKeys();
}
refreshLeafCertificate();
validateTenancy();
refreshIntermediateCertificates();
securityTokenAdapter = getSecurityTokenFromServer();
String token = securityTokenAdapter.getSecurityToken();
future.complete(token);
return token;
} catch (Exception e) {
future.completeExceptionally(e);
throw new BmcException(false, "Error refreshing security token.", e, null);
} finally {
inFlightRefresh = null;
}
}
(content is generated by AI, but was read by human and it makes sense)
Suggested title
X509FederationClientleavesinFlightRefreshincomplete when the refresh leader fails before cleanupDescription
Summary
X509FederationClientcan become permanently stuck after an exception during session-key orcertificate refresh.
The single-flight implementation assigns a new
CompletableFuturetoinFlightRefresh, but severaloperations that can throw are executed before the
try/catch/finallyresponsible for completing andclearing that future.
If one of those operations throws:
inFlightRefresh.inFlightRefreshremains non-null and incomplete.Timed out waiting for security token refresh.X509FederationClientdoes not recover. Recreating the authentication provider orrestarting the process is required.
Affected versions
Confirmed by a standalone reproducer:
oci-java-sdk-common:2.91.0oci-java-sdk-common:2.93.0Production stack traces from both observed incidents reported
Oracle-JavaSDK/2.92.0.The issue was introduced by commit
6a737779038869e1ca2948868a225736befdae0e,which released version
2.91.0and added the single-flight implementation.The relevant file is unchanged between public tags
v2.91.0andv2.93.0.This can be verified with:
The command produces no output.
Source-code evidence
In
X509FederationClient.refreshAndGetSecurityTokenInner,the Leader publishes the incomplete future:
The following operations are then executed before the cleanup
try/finally:The cleanup block starts only later:
An exception thrown before this
trybypasses both:The leaf-certificate block catches only
RefreshFailedException. In the production failures describedbelow,
URLBasedX509CertificateSupplierexhausted its retries and threw:This exception escaped at:
Before the introducing commit, refresh was performed entirely under the existing synchronized block
and there was no persistent
inFlightRefreshstate that could be left incomplete.Minimal reproduction
Use this dependency:
X509FederationClientwith anX509CertificateSupplierthat also implementsjavax.security.auth.Refreshable.refresh()method throwIllegalArgumentException, matching the terminal exception fromURLBasedX509CertificateSupplier.refreshAndGetSecurityToken(). This caller becomes the Leader and exits with the simulatedcertificate failure.
inFlightRefresh: it is non-null andisDone()is false.refreshAndGetSecurityToken()from another thread. It becomes a Follower and times out afterone minute.
Observed output using the real
oci-java-sdk-common:2.93.0artifact:The reproducer does not require OCI credentials or network access.
Production evidence
The same sequence was observed independently in two long-running Java processes on OCI compute
instances:
After the Leader terminated, subsequent callers repeatedly logged:
In one occurrence, the logs contained 62 Follower timeouts. Authenticated Monitoring operations
continued failing until the process restarted. Restarting created a new authentication provider and
immediately restored token refresh.
There is an additional timeout mismatch: the certificate supplier's three retries can take longer
than the hard-coded one-minute Follower wait, so Followers may time out even while a slow Leader is
still retrying. The persistent failure described in this issue remains after that Leader has exited
because its future is never completed or cleared.
Expected behavior
Every Leader exit path should:
inFlightRefresh.Followers should receive the Leader's original exception rather than wait for a future that can never
complete.
Suggested fix
Move all Leader operations under the same
try/catch/finally:(content is generated by AI, but was read by human and it makes sense)