Skip to content

Implement Spark SQL parser extensions and enhance analytic functions#38870

Draft
damccorm wants to merge 1 commit into
apache:masterfrom
damccorm:feature/sql-analytic-functions
Draft

Implement Spark SQL parser extensions and enhance analytic functions#38870
damccorm wants to merge 1 commit into
apache:masterfrom
damccorm:feature/sql-analytic-functions

Conversation

@damccorm

@damccorm damccorm commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

This PR brings in several enhancements for analytic/window functions and adds Spark-compatible SQL parser extensions:

  • Analytic Functions:
    • Added support for CUME_DIST().
    • Implemented IGNORE NULLS routing for FIRST_VALUE and LAST_VALUE (needed for pandas ffill/bfill compatibility).
    • Switched to a ValueHolder accumulator in these combiners to safely handle null values without throwing NPEs.
    • Fixed BeamWindowRel to support functions with no input columns (like COUNT(*)).
  • Planner/Parser Fixes:
    • Fixed a Volcano planning failure in BeamWindowRule by explicitly converting the input to BEAM_LOGICAL convention (mirrors what we do in Sort/Aggregation rules).
    • Added Spark-style parser extensions to parserImpls.ftl and config.fmpp: support for angle-bracket types (e.g., array<int>, map<string, int>), type constructors (e.g., float(x)), and the :: infix cast operator.
  • Spark-style Collection Constructors:
    • Registered ARRAY(...) and MAP(...) function-form constructors in CalciteQueryPlanner so they validate correctly (bracket form ARRAY[...] is standard, but function form is Spark-specific).
    • Added tests in BeamSqlDslArrayTest and BeamSqlMapTest.
  • Intervals: Added NanosDuration decoding to BeamTableUtils to correctly handle Calcite's interval representations.

Added tests to BeamAnalyticFunctionsTest for CUME_DIST and COUNT(*) OVER ().

@damccorm damccorm force-pushed the feature/sql-analytic-functions branch from 211ee94 to e78d80c Compare June 11, 2026 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant