[java-bigquerystorage] JsonToProtoMessage rejects base64 strings for BYTES columns

[laughingman7743]

Issue Details

JsonToProtoMessage cannot convert a BYTES column from a JSON document that encodes the value as
a base64 string — the encoding that both protobuf's canonical JSON mapping and BigQuery's own JSON
ingestion use. It accepts only a ByteString (which cannot come out of parsed JSON text) or a
JSONArray of integer byte values such as [104, 105].

This makes JsonStreamWriter unable to accept an otherwise ordinary JSON document, and it fails per
record rather than at setup.

Why this looks like a defect rather than a missing feature

Two independent conventions say a JSON bytes value is a base64 string:

1. Protobuf's canonical JSON mapping — https://protobuf.dev/programming-guides/json/ — states
   for bytes: "JSON value will be the data encoded as a string using standard base64 encoding with
   paddings. Either standard or URL-safe base64 encoding with/without paddings are accepted."
   protobuf-java-util implements exactly this in JsonFormat.ParserImpl#parseBytes.

2. BigQuery's own JSON ingestion — Loading JSON data from Cloud Storage
   states "Columns with BYTES types must be encoded as Base64" in its data type conversion section
   (the CSV page
   carries the same line).

So a JSON document that bq load ingests without complaint fails per record through the Storage
Write API's own JSON converter. Conversely, [104, 105] — the form the converter does accept — is
not documented anywhere I could find and is not something a producer emits.

Where

google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java,
in two places — case BYTES: in fillField (scalar) and in fillRepeatedField. Both fall through
to the "wrong field type" error when the value is a String.

Note that proto BYTES also carries BigQuery NUMERIC/BIGNUMERIC, handled in a prelude before the
plain BYTES handling. So any fix has to be guarded on
fieldSchema != null && fieldSchema.getType() == TableFieldSchema.Type.BYTES — with the overload
that takes no TableSchema, the converter cannot tell BYTES from NUMERIC, and decoding
unconditionally would turn today's error on a NUMERIC column into a silently wrong value. The
reproducer below includes a NUMERIC case so that behaviour is pinned.

On the alphabet

Protobuf accepts standard and URL-safe, with and without padding. JsonFormat.ParserImpl#parseBytes
already answers how to do that — try standard, fall back to URL-safe on IllegalArgumentException.

java.util.Base64 and Guava's BaseEncoding are behaviourally identical for this purpose, so the
choice is free. Measured on the JDK below, Guava 33.5.0-jre, one run:

decoder	aGk=	aGk (unpadded)	-_8= (URL-safe)	+/8= (standard)
java.util.Base64.getDecoder()	ok	ok	error	ok
java.util.Base64.getUrlDecoder()	ok	ok	ok	error
BaseEncoding.base64()	ok	ok	error	ok
BaseEncoding.base64Url()	ok	ok	ok	error

Both tolerate missing padding, and the two alphabets are mutually exclusive on +/ versus -_, so
the try-then-fall-back shape is required rather than cosmetic.

The pull request uses java.util.Base64, following AGENTS.md §6's preference for the standard
library over an existing dependency. If you would rather match JsonFormat.ParserImpl#parseBytes
and use Guava's BaseEncoding, say so and I will switch it — the behaviour is identical, so it is a
two-line change.

The pull request

A pull request is ready and follows immediately, per CONTRIBUTING's "open an issue before doing
significant work".

It is additive: a String in a BYTES column is an error today, so no input that works now
changes behaviour.

It also fixes a second, independent problem in the same case BYTES: branch, which I have filed
separately as #13979 — fillField rejects a scalar byte[] while fillRepeatedField accepts
one. Both live in the same few lines, so one pull request seemed easier to review than two touching
the same code; happy to split it in two if you would prefer to take them independently.

Scope: the pull request changes v1 only. com.google.cloud.bigquery.storage.v1beta2 carries its
own copy of JsonToProtoMessage with the same branch — tell me if you want it mirrored there and I
will add it.

Environment

• OS Type and Version: macOS 26.5.2
• Java Version and JDK Vendor: OpenJDK 17.0.19, Eclipse Temurin (17.0.19+10)
• (If using GraalVM) GraalVM Version: n/a

Not deployed on GCP — this reproduces entirely locally, since the failure is in client-side
conversion before any RPC is made.

Dependencies

• Libraries-Bom: com.google.cloud:libraries-bom:26.85.1
• Client library: com.google.cloud:google-cloud-bigquerystorage:3.30.0 (resolved by the BOM
  above)
• com.google.protobuf:protobuf-java:4.33.2
• com.google.guava:guava:33.5.0-jre
• org.json:json (the JSONObject parameter type of convertToProtoMessage)

Gax and auth versions are whatever the BOM resolves; no RPC is made, so they are not involved in the
failure.

Reproducer

Single file, no GCP credentials or resources needed — the failure is entirely in local conversion.
Run with google-cloud-bigquerystorage on the classpath.

import com.google.cloud.bigquery.storage.v1.*;
import com.google.protobuf.Descriptors;
import org.json.JSONObject;

public class B64Probe {
  static TableFieldSchema f(String n, TableFieldSchema.Type t, TableFieldSchema.Mode m) {
    return TableFieldSchema.newBuilder().setName(n).setType(t).setMode(m).build();
  }

  public static void main(String[] args) throws Exception {
    TableSchema schema =
        TableSchema.newBuilder()
            .addFields(f("blob", TableFieldSchema.Type.BYTES, TableFieldSchema.Mode.NULLABLE))
            .addFields(f("blobs", TableFieldSchema.Type.BYTES, TableFieldSchema.Mode.REPEATED))
            .addFields(f("num", TableFieldSchema.Type.NUMERIC, TableFieldSchema.Mode.NULLABLE))
            .build();
    Descriptors.Descriptor d =
        BQTableSchemaToProtoDescriptor.convertBQTableSchemaToProtoDescriptor(schema);

    probe(d, schema, "scalar base64", "{\"blob\":\"aGk=\"}");
    probe(d, schema, "scalar jsonarray", "{\"blob\":[104,105]}");
    probe(d, schema, "repeated base64", "{\"blobs\":[\"aGk=\"]}");
    probe(d, schema, "repeated jsonarray", "{\"blobs\":[[104,105]]}");
    probe(d, schema, "numeric string", "{\"num\":\"1.5\"}");
  }

  static void probe(Descriptors.Descriptor d, TableSchema s, String label, String json) {
    try {
      Object msg =
          JsonToProtoMessage.INSTANCE.convertToProtoMessage(d, s, new JSONObject(json), false);
      System.out.println("OK   " + label + " :: " + msg.toString().replace("\n", " ").trim());
    } catch (Exception e) {
      System.out.println("FAIL " + label + " :: " + e.getClass().getName() + ": " + e.getMessage());
    }
  }
}

Steps:

1. Put google-cloud-bigquerystorage:3.30.0 (and its transitive dependencies) on the classpath.
2. Run the class above.
3. Observe that the two base64 cases fail while the JSONArray cases succeed.

"aGk=" is base64 for hi, which is what [104, 105] produces.

Logs and Stack Trace

Output of the reproducer, verbatim:

FAIL scalar base64 :: com.google.cloud.bigquery.storage.v1.Exceptions$RowIndexToErrorException: The map of row index to error message is {0=Field root.blob failed to convert to BYTES. Error: JSONObject does not have a bytes field at root.blob.}
OK   scalar jsonarray :: blob: "hi"
FAIL repeated base64 :: com.google.cloud.bigquery.storage.v1.Exceptions$RowIndexToErrorException: The map of row index to error message is {0=Field root.blobs failed to convert to BYTES. Error: JSONObject does not have a bytes field at root.blobs[0].}
OK   repeated jsonarray :: blobs: "hi"
OK   numeric string :: num: "\000/hY"

The last line is the NUMERIC case that a fix must not disturb.

Behavior

• When did the issue begin? Not a regression. The branch already had this
  ByteString/JSONArray shape in April 2021 — see
  googleapis/java-bigquerystorage#984.
• Is it flaky? No, fully deterministic and reproducible offline.
• Related to volume of data? No.