Skip to content

fix(bigquerystorage): accept base64 and byte[] values for BYTES columns - #13981

Open
laughingman7743 wants to merge 2 commits into
googleapis:mainfrom
laughingman7743:fix-jsontoproto-bytes-base64
Open

fix(bigquerystorage): accept base64 and byte[] values for BYTES columns#13981
laughingman7743 wants to merge 2 commits into
googleapis:mainfrom
laughingman7743:fix-jsontoproto-bytes-base64

Conversation

@laughingman7743

Copy link
Copy Markdown

JsonToProtoMessage accepts only a ByteString or a JSONArray of integer byte values for a
BYTES column, so a base64 string fails per record. That is the encoding both protobuf's
canonical JSON mapping and BigQuery's own JSON
ingestion use — Loading JSON data from Cloud Storage
states "Columns with BYTES types must be encoded as Base64" — so a document bq load accepts is
rejected here. A scalar byte[] was also rejected while the repeated path accepted one.

This change is additive. A String in a BYTES column is an error today, and so is a scalar
byte[], so no input that works now changes behaviour.

Design notes

Decoding is guarded on the TableSchema saying BYTES. Proto BYTES also carries BigQuery
NUMERIC and BIGNUMERIC, and with the overload that takes no TableSchema the converter cannot
tell them apart. Without the guard, a string on a NUMERIC column would silently decode as base64
instead of erroring. Two tests pin this — testBytesFromBase64NeedsTableSchema and
testNumericFromStringIsNotDecodedAsBase64 — and they pass both with and without the change.

Standard and URL-safe alphabets are both accepted, with or without padding, as protobuf's
mapping requires. The two alphabets are mutually exclusive on +/ versus -_, so the try-then-fall
-back shape is what makes both work; this mirrors JsonFormat.ParserImpl#parseBytes in
protobuf-java-util.

The decoder is java.util.Base64, following AGENTS.md §6's preference for the standard
library over an existing dependency. It is behaviourally identical to Guava's BaseEncoding here —
measured across padded, unpadded and both alphabets. If you would rather match JsonFormat and use
BaseEncoding, say so and I will switch it; it is a two-line change.

Scope

This fixes two separate reports because they live in the same case BYTES: branch, a few lines
apart. Happy to split it into two pull requests if you would rather review them independently.

It changes v1 only. com.google.cloud.bigquery.storage.v1beta2 carries its own copy of
JsonToProtoMessage with the same branch — tell me if you want it mirrored there and I will add it.

Testing

Seven tests added to JsonToProtoMessageTest, covering scalar base64 (padded, unpadded, both
alphabets), repeated base64, a scalar byte[], an invalid base64 string, and the two guards above.

  • mvn -pl java-bigquerystorage/google-cloud-bigquerystorage test -P quick-build passes.
  • With the change to JsonToProtoMessage reverted and the tests left in place, five of the seven
    fail — so they exercise the new behaviour rather than merely passing. The two that still pass are
    the guards, which are meant to hold in both states.
  • Formatted with the repository's fmt-maven-plugin (google-java-format 1.25.2).

Fixes #13980
Fixes #13979

JsonToProtoMessage accepts only ByteString or a JSONArray of integer byte
values for a BYTES column, so a base64 string fails per record — the encoding
protobuf's canonical JSON mapping and BigQuery's own JSON ingestion both use.
A scalar byte[] was also rejected while the repeated path accepted one.

Decoding is guarded on the TableSchema saying BYTES, because proto BYTES also
carries NUMERIC and BIGNUMERIC; without that guard a string on a NUMERIC
column would silently decode instead of erroring.

Standard and URL-safe alphabets are both accepted, with or without padding,
as that mapping requires.

Fixes googleapis#13980
Fixes googleapis#13979
@laughingman7743
laughingman7743 requested review from a team as code owners August 1, 2026 10:32

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for decoding Base64-encoded strings (both standard and URL-safe alphabets) into byte arrays when converting JSON to proto messages for fields of type BYTES. It also includes comprehensive unit tests. The reviewer suggested optimizing the parseBase64 method to avoid using exceptions for control flow. Instead of catching IllegalArgumentException to fall back to the URL-safe decoder, the reviewer proposed checking if the string contains - or _ to determine which decoder to use, which improves performance and simplifies the code.

Comment on lines +1130 to +1143
private static byte[] parseBase64(String currentScope, String value) {
try {
return Base64.getDecoder().decode(value);
} catch (IllegalArgumentException standardAlphabetFailed) {
try {
return Base64.getUrlDecoder().decode(value);
} catch (IllegalArgumentException urlSafeAlphabetFailed) {
throw new IllegalArgumentException(
"Error: "
+ currentScope
+ " could not be converted to byte[]: not a valid base64 string.");
}
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using exceptions for control flow (catching IllegalArgumentException from the standard decoder to fall back to the URL-safe decoder) can introduce significant performance overhead, especially when processing large datasets containing URL-safe base64 strings. Since standard and URL-safe base64 alphabets only differ by the characters +// versus -/_, we can check if the string contains - or _ to decide which decoder to use. This avoids throwing and catching exceptions in the happy path for URL-safe base64 strings, and also simplifies the code by removing the nested try-catch blocks.

Suggested change
private static byte[] parseBase64(String currentScope, String value) {
try {
return Base64.getDecoder().decode(value);
} catch (IllegalArgumentException standardAlphabetFailed) {
try {
return Base64.getUrlDecoder().decode(value);
} catch (IllegalArgumentException urlSafeAlphabetFailed) {
throw new IllegalArgumentException(
"Error: "
+ currentScope
+ " could not be converted to byte[]: not a valid base64 string.");
}
}
}
private static byte[] parseBase64(String currentScope, String value) {
try {
if (value.contains("-") || value.contains("_")) {
return Base64.getUrlDecoder().decode(value);
}
return Base64.getDecoder().decode(value);
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException(
"Error: "
+ currentScope
+ " could not be converted to byte[]: not a valid base64 string.",
e);
}
}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — applied.

Falling back through a caught exception would have put an exception on the happy path for every
record when a producer uses the URL-safe alphabet, not just occasionally, which is a real cost in a
streaming write path. Picking the alphabet up front is equivalent: a value containing - or _ can
never be valid under the standard alphabet, and one containing neither is handled by the standard
decoder, so the outcomes match in both the success and the failure case.

Applied with indexOf('-') >= 0 || indexOf('_') >= 0 rather than contains("-"), to use the char
overload; behaviour is identical. I also kept your propagation of the original exception as the
cause, which the previous version dropped.

JsonToProtoMessageTest still passes (58 tests), and the module suite is green (402 tests).

… fallback

Falling back through a caught IllegalArgumentException put an exception on the
happy path for every record when a producer uses the URL-safe alphabet. A value
containing - or _ is never valid standard base64, so the alphabet can be chosen
before decoding. The original exception is now kept as the cause.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant