Limit pandas to < 3 for DataFrame XComs

[potiuk]

A pandas.DataFrame does not survive an XCom round trip under pandas 3 the way it
does under pandas 2, so this version of Airflow constrains pandas to < 3 and
refuses DataFrame XComs at runtime when a newer pandas is installed anyway.

What breaks under pandas 3

The serializer is never reached. pandas 3 exposes its public classes from the
pandas namespace, so a DataFrame qualifies as pandas.DataFrame rather than
pandas.core.frame.DataFrame. The serde registry is keyed on the latter, so it does
not dispatch to the pandas serializer at all and the caller gets serde's generic

TypeError: cannot serialize object of type <class 'pandas.DataFrame'>

with nothing in it pointing at pandas or at what to do about it.

The dtypes differ. Even once dispatch is fixed, the same payload reads back
differently depending on the reader's pandas version — an object column becomes
str, and missing values become nan rather than None. The reader's pandas
version decides what a task receives, not the writer's, so a mixed-version
deployment silently hands tasks data that differs from what was pushed.

Approach

pandas<3 is declared everywhere Airflow declares pandas — 45 specifiers across
14 pyproject.toml files: the pandas extra on common-sql (what
apache-airflow[pandas] resolves to), the task-sdk development group, and the
thirteen providers that pin pandas independently of common-sql — amazon,
apache/hdfs, apache/hive, databricks, exasol, google, papermill,
postgres, presto, salesforce, snowflake, trino and weaviate.

Capping only common-sql would have left a gap: installing any of those providers
could still resolve pandas 3, so the constraint would not hold and the user would
meet the runtime refusal instead of simply being kept on a supported version.

pandas-gbq in providers/google is deliberately not capped — a different
package, unaffected by this. Provider README.rst files and uv.lock are
regenerated to match.

That constraint cannot be relied on by itself. pandas is an optional
dependency: a deployment can install pandas 3 directly, or pull it in through
another package, and nothing in Airflow's metadata prevents it. So the constraint
is backed by a runtime check that fails with a message naming pandas, the version
found, and what to do.

The check needs the pandas 3 qualname registered to be reachable at all.
pandas.DataFrame is added to the registry only so that the request reaches this
module and can be refused deliberately — not to support it. Without that entry the
guard is dead code, because serde never dispatches.

Reads are refused as well as writes. A payload written under pandas 2 comes
back with pandas 3 dtypes, so accepting it would give the task different data than
was pushed, with no signal at all. An error on both sides is the safer outcome, and
the write-side failure is the more recoverable of the two.

Behaviour change

With pandas >= 3 installed, pushing or pulling a DataFrame through XCom raises:

RuntimeError: DataFrame XComs are not supported with pandas 3.0.5: this version of
Airflow supports pandas < 3 for XCom serialization. Pin pandas < 3 in the
environment that runs your tasks, or avoid passing DataFrames through XCom. ...

Previously the same situation produced an opaque serde TypeError on write, and —
once dispatch was corrected — silently different dtypes on read. Nothing changes for
pandas 2 users.

This defers pandas 3 support, it does not drop it. Moving from pandas 2 to
pandas 3 is a deliberate migration for users to make, given the dtype changes.
Airflow will enable it in a future release; see the follow-up PR linked in the
comments.

Test plan

• task-sdk/tests/task_sdk/serde/test_serializers.py — 11 pandas cases pass
• test_pandas_3_dataframe_xcom_is_refused — 3.0.0, 3.0.5 and 4.1.2 all
  refused on both serialize and deserialize
• test_pandas_3_dataframe_name_is_registered — guards the reachability
  precondition, so the guard cannot silently become dead code if the registry
  entry is dropped
• test_pandas_2_dataframe_xcom_still_round_trips — 2.1.2 and 2.3.3
  unaffected
• Existing pandas serde tests unchanged and passing
• All 45 pandas specifiers verified capped and none left uncapped; every changed
  pyproject.toml re-parsed with tomllib to confirm it is still valid
• pandas-gbq confirmed unchanged
• uv.lock regenerated (947 packages resolved) with no uncapped pandas entry
  remaining, and provider README.rst files regenerated from the new pins
• ruff check / ruff format clean

Was generative AI tooling used to co-author this PR?

• Yes — Claude Opus 5 (1M context)

Generated-by: Claude Opus 5 (1M context) following the guidelines at
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions

[potiuk]

Cross-reference: #70558 is where pandas 3 support starts, and it is milestoned for 3.4.0. This PR is the deliberate limit we apply until then.

The two are complements rather than alternatives:

• Support pandas 3 for DataFrame XComs #70558 registers both DataFrame qualnames (pandas.core.frame.DataFrame and pandas.DataFrame) so that a DataFrame written by either pandas version can be read by either, and documents that the reader's pandas version decides the resulting dtypes.
• this PR registers the pandas 3 qualname for the opposite purpose — so the request reaches the serializer and can be refused with an actionable message instead of dying on serde's generic cannot serialize object of type.

The reason for limiting rather than shipping support now is that moving from pandas 2 to pandas 3 should be a deliberate decision by the user, not something an Airflow upgrade does to them. Under pandas 3 the same XCom payload reads back with different dtypes — an object column becomes str, missing values become nan rather than None — so data already written under pandas 2 is read differently afterwards. That is a migration to plan, not a side effect to absorb.

So this is a deferral, not a drop. Users who want pandas 3 today can still choose it; they just cannot pass DataFrames through XCom while doing so, and they now get told that explicitly rather than discovering it through an unrelated-looking TypeError. Support will be enabled in a future release once #70558 (or its successor) lands.

[shahar1]

AI Summary

Approach is right and the guard is correctly wired. Three findings, none a correctness bug: two I'd like resolved before this merges, one nit.

What I checked rather than took on trust

• Registration is string-keyed (task-sdk/src/airflow/sdk/serde/__init__.py:477-483), so the extra pandas.DataFrame entry costs no eager pandas import, and _register() only raises when the same name maps to a different module — no duplicate-registration risk from registering an alias of an already-registered class.
• Nothing between serde.serialize() and the serializer swallows the RuntimeError (task-sdk/src/airflow/sdk/serde/__init__.py:245-246), so the actionable message actually reaches the caller on both the write and the read path rather than being retried into the custom-serializer fallback.
• The guard sits after the isinstance(o, pd.DataFrame) check on write and after the cls is not pd.DataFrame check on read — both correct placements, and non-DataFrame values are unaffected.
• No sibling serializer was missed: task-sdk/src/airflow/sdk/serde/serializers/pandas.py is the only DataFrame serializer in the repo.

1. The upper bounds carry no rationale comment, and no tracking-issue link

The project's rule on upper bounds, from contributing-docs/13_airflow_dependencies_and_extras.rst:

We very rarely upper-bind dependencies - only when there is a known conflict with a new or upcoming version of a dependency that breaks the installation or tests (and we always make a comment why we are upper-bounding a dependency).

and providers/AGENTS.md:

Don't upper-bound dependencies by default; add limits only with justification.

45 specifiers across 14 files gain <3, and not one of them carries a comment saying why. AGENTS.md § Tracking issues for deferred work is more specific still — a version cap needs a tracking issue and the full issue URL as a comment at the cap site, precisely so the reference outlives the PR. It also rules out the pointer this PR uses:

Do not write vague forward-looking phrases like "will open a tracking issue" or "to be filed later" in the PR body or in code comments.

"see the follow-up PR linked in the comments" in the description is that phrase, and a GitHub cross-reference to #70558 does not reach whoever opens providers/snowflake/pyproject.toml in six months, or the release manager deciding whether the cap can come off. Inline suggestion on the common-sql extra below; the same one-liner at the other 13 sites would close it.

2. Worth stating deliberately how far the caps reach

The 13 provider caps are the widest part of the diff and the narrowest in benefit. The defect is in DataFrame XCom serde, which lives in task-sdk; the provider pandas extras exist for get_df()-style paths that never go near it. After this, someone on apache-airflow-providers-snowflake[pandas] who never puts a DataFrame in an XCom cannot install pandas 3 at all.

That cost also lands outside this repo's release train. Providers ship from main and support Airflow 2.11 through 3.3, so the cap binds users on Airflow versions that do not have — and will never get — the guard it is protecting. It is a permanent property of those released distributions, not something a later Airflow release can lift.

I take the point in the description that capping only common-sql leaves a gap. But the paragraph right after it concedes the constraint cannot be relied on regardless — pandas can be installed directly or pulled in through another package — and the task-sdk pin is in the dev dependency group, so it never reaches a user environment at all. The runtime guard is what actually closes the hole; the caps buy partial coverage at a cost paid largely by users who are not exposed to the bug. Capping common-sql[pandas] alone, which is what apache-airflow[pandas] resolves to, plus the guard, looks like the narrower trade.

Your call and I'm not blocking on it — but with backport-to-v3-3-test putting these caps into a patch-level provider release, I'd rather see the breadth chosen deliberately than inherited.

3. Nit on the new pandas 2 test

Inline below — it cannot fail without this change. The other two new tests are good; test_pandas_3_dataframe_name_is_registered in particular, since it pins the reachability precondition the entire guard depends on and would otherwise be the silent way this feature rots.

---

This review was drafted by an AI-assisted tool and
confirmed by an Airflow maintainer. The findings
above are observations, not blockers; an Airflow
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.

---

Drafted-by: Claude Opus 5 (1M context); reviewed by @shahar1 before posting

[potiuk]

Thanks @shahar1 — both correct, both fixed.

• Rationale comments added above every one of the 45 specifiers, across all 15 files, with the tracking pointer as a full URL so it resolves from a grep of a checkout. There is no tracking issue for pandas 3 support, so the comments point at Support pandas 3 for DataFrame XComs #70558, which is the PR that removes the cap — that seemed more useful than opening an issue whose only content would be "see that PR". Say the word if you would rather have a real issue.
• test_pandas_2_dataframe_xcom_still_round_trips dropped. You were right that it passes unchanged on main — nothing read pd.__version__ before this PR, so monkeypatching it to a 2.x value while the installed pandas is already 2.x asserted nothing, and test_pandas_serializers already covers the round trip.

That is the second vacuous test of mine you have caught in this batch, which is a useful signal about how I was writing them: I was checking that the new code path is reachable rather than that the new behaviour is required. The reverting-the-change test is the right bar.

[github-actions]

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-3-test

You can attempt to backport this manually by running:

cherry_picker 415cb70 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.