Skip to content

fix(api): scope NULL branch in NullableDatetimeRangeFilter for start_date lower bounds [fj4WqyCCw3C5ShR1RfB7MoBPTpkRrBFYP1uT35g3MvT] - #70629

Open
waterWang wants to merge 2 commits into
apache:mainfrom
waterWang:fix/nullable-datetime-range-filter-start-date-gte
Open

fix(api): scope NULL branch in NullableDatetimeRangeFilter for start_date lower bounds [fj4WqyCCw3C5ShR1RfB7MoBPTpkRrBFYP1uT35g3MvT]#70629
waterWang wants to merge 2 commits into
apache:mainfrom
waterWang:fix/nullable-datetime-range-filter-start-date-gte

Conversation

@waterWang

Copy link
Copy Markdown

Description

NullableDatetimeRangeFilter was added in #66696 to replace COALESCE(column, now()) with index-friendly OR predicates. For lower bounds (gte/gt), the NULL branch passes unconditionally:

select = select.where(or_(self.attribute >= x, self.attribute.is_(None)))

This is correct for non-terminal rows (a not-yet-started task will eventually start). But it's wrong for terminal dag runs whose start_date is NULL — a failed/success run that never started will never have a start_date, yet it matches every start_date_gte filter forever.

Root cause

When querying GET /dags/~/dagRuns?start_date_gte=<recent-timestamp>, the response includes terminal dag runs from years ago (state: failed, start_date: null, populated end_date) as if they had just occurred. Roughly 99% of a -logical_date-ordered page are these stale null-start_date rows.

Fix

Add an extra_null_condition parameter to NullableDatetimeRangeFilter. For start_date lower bounds, the factory passes model.end_date.is_(None), so the NULL branch becomes:

or_(start_date >= x, and_(start_date IS NULL, end_date IS NULL))

A genuinely pending run has start_date IS NULL AND end_date IS NULL. A terminal run with NULL start_date has end_date IS NOT NULL and is correctly excluded.

Changes

  • parameters.py: Add extra_null_condition to NullableDatetimeRangeFilter; update datetime_range_filter_factory to pass it for start_date
  • test_parameters.py: Update existing lower-bound test to verify end_date IS NULL; add test for terminal-run exclusion

Related issues

Fixes #70627

…date lower bounds

For start_date lower bounds (gte/gt), the NULL branch now also requires
end_date IS NULL. This prevents terminal dag runs with a NULL start_date
(e.g. failed runs that never started) from matching every start_date_gte
filter indefinitely.

Closes apache#70627
@boring-cyborg

boring-cyborg Bot commented Jul 28, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@ColtenOuO

Copy link
Copy Markdown
Contributor

The suffix of the PR title needs to be changed.

There have some garbled text.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dagRuns API: start_date_gte filter matches terminal (failed/success) runs with a NULL start_date regardless of age

2 participants