fix(bigquerystorage): accept base64 and byte[] values for BYTES columns - #13981
fix(bigquerystorage): accept base64 and byte[] values for BYTES columns#13981laughingman7743 wants to merge 2 commits into
Conversation
JsonToProtoMessage accepts only ByteString or a JSONArray of integer byte values for a BYTES column, so a base64 string fails per record — the encoding protobuf's canonical JSON mapping and BigQuery's own JSON ingestion both use. A scalar byte[] was also rejected while the repeated path accepted one. Decoding is guarded on the TableSchema saying BYTES, because proto BYTES also carries NUMERIC and BIGNUMERIC; without that guard a string on a NUMERIC column would silently decode instead of erroring. Standard and URL-safe alphabets are both accepted, with or without padding, as that mapping requires. Fixes googleapis#13980 Fixes googleapis#13979
There was a problem hiding this comment.
Code Review
This pull request adds support for decoding Base64-encoded strings (both standard and URL-safe alphabets) into byte arrays when converting JSON to proto messages for fields of type BYTES. It also includes comprehensive unit tests. The reviewer suggested optimizing the parseBase64 method to avoid using exceptions for control flow. Instead of catching IllegalArgumentException to fall back to the URL-safe decoder, the reviewer proposed checking if the string contains - or _ to determine which decoder to use, which improves performance and simplifies the code.
| private static byte[] parseBase64(String currentScope, String value) { | ||
| try { | ||
| return Base64.getDecoder().decode(value); | ||
| } catch (IllegalArgumentException standardAlphabetFailed) { | ||
| try { | ||
| return Base64.getUrlDecoder().decode(value); | ||
| } catch (IllegalArgumentException urlSafeAlphabetFailed) { | ||
| throw new IllegalArgumentException( | ||
| "Error: " | ||
| + currentScope | ||
| + " could not be converted to byte[]: not a valid base64 string."); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Using exceptions for control flow (catching IllegalArgumentException from the standard decoder to fall back to the URL-safe decoder) can introduce significant performance overhead, especially when processing large datasets containing URL-safe base64 strings. Since standard and URL-safe base64 alphabets only differ by the characters +// versus -/_, we can check if the string contains - or _ to decide which decoder to use. This avoids throwing and catching exceptions in the happy path for URL-safe base64 strings, and also simplifies the code by removing the nested try-catch blocks.
| private static byte[] parseBase64(String currentScope, String value) { | |
| try { | |
| return Base64.getDecoder().decode(value); | |
| } catch (IllegalArgumentException standardAlphabetFailed) { | |
| try { | |
| return Base64.getUrlDecoder().decode(value); | |
| } catch (IllegalArgumentException urlSafeAlphabetFailed) { | |
| throw new IllegalArgumentException( | |
| "Error: " | |
| + currentScope | |
| + " could not be converted to byte[]: not a valid base64 string."); | |
| } | |
| } | |
| } | |
| private static byte[] parseBase64(String currentScope, String value) { | |
| try { | |
| if (value.contains("-") || value.contains("_")) { | |
| return Base64.getUrlDecoder().decode(value); | |
| } | |
| return Base64.getDecoder().decode(value); | |
| } catch (IllegalArgumentException e) { | |
| throw new IllegalArgumentException( | |
| "Error: " | |
| + currentScope | |
| + " could not be converted to byte[]: not a valid base64 string.", | |
| e); | |
| } | |
| } |
There was a problem hiding this comment.
Good catch — applied.
Falling back through a caught exception would have put an exception on the happy path for every
record when a producer uses the URL-safe alphabet, not just occasionally, which is a real cost in a
streaming write path. Picking the alphabet up front is equivalent: a value containing - or _ can
never be valid under the standard alphabet, and one containing neither is handled by the standard
decoder, so the outcomes match in both the success and the failure case.
Applied with indexOf('-') >= 0 || indexOf('_') >= 0 rather than contains("-"), to use the char
overload; behaviour is identical. I also kept your propagation of the original exception as the
cause, which the previous version dropped.
JsonToProtoMessageTest still passes (58 tests), and the module suite is green (402 tests).
… fallback Falling back through a caught IllegalArgumentException put an exception on the happy path for every record when a producer uses the URL-safe alphabet. A value containing - or _ is never valid standard base64, so the alphabet can be chosen before decoding. The original exception is now kept as the cause.
JsonToProtoMessageaccepts only aByteStringor aJSONArrayof integer byte values for aBYTEScolumn, so a base64 string fails per record. That is the encoding both protobuf'scanonical JSON mapping and BigQuery's own JSON
ingestion use — Loading JSON data from Cloud Storage
states "Columns with BYTES types must be encoded as Base64" — so a document
bq loadaccepts isrejected here. A scalar
byte[]was also rejected while the repeated path accepted one.This change is additive. A
Stringin aBYTEScolumn is an error today, and so is a scalarbyte[], so no input that works now changes behaviour.Design notes
Decoding is guarded on the
TableSchemasayingBYTES. ProtoBYTESalso carries BigQueryNUMERICandBIGNUMERIC, and with the overload that takes noTableSchemathe converter cannottell them apart. Without the guard, a string on a
NUMERICcolumn would silently decode as base64instead of erroring. Two tests pin this —
testBytesFromBase64NeedsTableSchemaandtestNumericFromStringIsNotDecodedAsBase64— and they pass both with and without the change.Standard and URL-safe alphabets are both accepted, with or without padding, as protobuf's
mapping requires. The two alphabets are mutually exclusive on
+/versus-_, so the try-then-fall-back shape is what makes both work; this mirrors
JsonFormat.ParserImpl#parseBytesinprotobuf-java-util.The decoder is
java.util.Base64, followingAGENTS.md§6's preference for the standardlibrary over an existing dependency. It is behaviourally identical to Guava's
BaseEncodinghere —measured across padded, unpadded and both alphabets. If you would rather match
JsonFormatand useBaseEncoding, say so and I will switch it; it is a two-line change.Scope
This fixes two separate reports because they live in the same
case BYTES:branch, a few linesapart. Happy to split it into two pull requests if you would rather review them independently.
It changes
v1only.com.google.cloud.bigquery.storage.v1beta2carries its own copy ofJsonToProtoMessagewith the same branch — tell me if you want it mirrored there and I will add it.Testing
Seven tests added to
JsonToProtoMessageTest, covering scalar base64 (padded, unpadded, bothalphabets), repeated base64, a scalar
byte[], an invalid base64 string, and the two guards above.mvn -pl java-bigquerystorage/google-cloud-bigquerystorage test -P quick-buildpasses.JsonToProtoMessagereverted and the tests left in place, five of the sevenfail — so they exercise the new behaviour rather than merely passing. The two that still pass are
the guards, which are meant to hold in both states.
fmt-maven-plugin(google-java-format 1.25.2).Fixes #13980
Fixes #13979