Is your feature request related to a problem or challenge?
Writing Parquet V2 can benefit from set_data_page_v2_compression_ratio_threshold, which is not exposed in Datafusion.
Describe the solution you'd like
Expose data_page_compression_ratio_threshold on datafusion_common::config::ParquetOptions and ParquetColumnOptions so it can be passed along to WriterPropertiesBuilder.
This primarily enables datafusion_datasource_parquet::file_format::ParquetFormat to use this setting.
Describe alternatives you've considered
We are evaluating if we should always disable compression for sorted delta-encoded columns. This can easily be done with ParquetColumnOptions.compression today.
We can also create a wrapper ParquetSink to set more options on the WriterPropertiesBuilder but that's quite a painful workaround for little gain.
Additional context
We experience poor compression ratios (<5%) with Zstd for delta-encoded columns of timestamps and other types. Disabling compression with a threshold would be ideal to avoid the decompression overhead at query time.
Is your feature request related to a problem or challenge?
Writing Parquet V2 can benefit from set_data_page_v2_compression_ratio_threshold, which is not exposed in Datafusion.
Describe the solution you'd like
Expose
data_page_compression_ratio_thresholdondatafusion_common::config::ParquetOptionsandParquetColumnOptionsso it can be passed along toWriterPropertiesBuilder.This primarily enables
datafusion_datasource_parquet::file_format::ParquetFormatto use this setting.Describe alternatives you've considered
We are evaluating if we should always disable compression for sorted delta-encoded columns. This can easily be done with
ParquetColumnOptions.compressiontoday.We can also create a wrapper
ParquetSinkto set more options on theWriterPropertiesBuilderbut that's quite a painful workaround for little gain.Additional context
We experience poor compression ratios (<5%) with Zstd for delta-encoded columns of timestamps and other types. Disabling compression with a threshold would be ideal to avoid the decompression overhead at query time.