Support pandas 3 for DataFrame XComs

[potiuk]

pandas 3 exposes its public classes from the pandas namespace, so a DataFrame is qualified as pandas.DataFrame rather than pandas.core.frame.DataFrame. The serde registry is keyed on that name and only knew the old one, so under pandas 3 no DataFrame could be pushed through XCom at all:

TypeError: cannot serialize object of type <class 'pandas.DataFrame'>

Both names are now registered, so a DataFrame written by either pandas version can be read by either. Verified across all four combinations (writer x reader), with the payload written by one interpreter and read by the other:

Reader	pandas-2 payload	pandas-3 payload
pandas 2.3.3	object, missing = None	object, missing = None
pandas 3.0.5	str, missing = nan	str, missing = nan

The reader's pandas version decides the dtypes, not the writer's — that, and the fact that a component without this change cannot read a pandas-3-written DataFrame XCom, is what the significant newsfragment documents.

Two provider tests also asserted on pandas 2 behaviour (object dtype, missing values stringified to "nan" / "None") and are now version-aware. The production paths were already correct.

Split out of #70501, which holds the development environment on pandas 2 in the meantime.

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

[potiuk]

Hey @vatsrahul1001 -> I think that one might be useful to get into 3.3.1 as well. Some of the recent upgrades in our providers "freed" pandas to be updated to version 3 in a number of combinations - and if someone uses Pandas 2 directly sent as XCom, the XComs that are stored in DB will start failing tasks reading them if the tasks will get Pandas 3.

This is easily fixable - by this one - the only issue is that there is no way back after migrating to Pandas 3 and that you have to migrate everything at once to Pandas 3 (but this is the only migration scenario we support).

That's why the significant newsfragment.

We could wait to Airflow 3.4, but I am afraid, people will start to migrate to Pandas 3 whenever other Airflow dependencies allow them to, so there is a high risk between now and 3.4 lack of this will hit real users.

We could - alternatively - limit 3.3.1 to "pandas<3" - but that's only half the solution, because older versions of Airflow do not have "pandas<3" limitation - and if someone updates providers and updates pandas to 3 - they will have no easy way back. By releasing Airflow 3.3.1 with pandas 3 support, we give them the option "Upgrade to Airlfow 3.3.1+" if this happens.