Skip to content

Fix openlineage team name handling, add dag_version_data and TI note - #70772

Merged
mobuchowski merged 1 commit into
apache:mainfrom
kacpermuda:openlineage-team-name-session-fix-and-dag-version-data
Jul 31, 2026
Merged

Fix openlineage team name handling, add dag_version_data and TI note#70772
mobuchowski merged 1 commit into
apache:mainfrom
kacpermuda:openlineage-team-name-session-fix-and-dag-version-data

Conversation

@kacpermuda

Copy link
Copy Markdown
Collaborator

The previous implementation called DagBundleModel.get_team_name() without an explicit session. This triggered @provide_session to open a new create_session(), which — for ORM DagRun objects in the scheduler loop — resolves to the same scoped session and commits on exit. That commit fires Airflow's prohibit_commit HA lock guard, producing errors in multi-scheduler setups.

The fix switches to object_session(dagrun) to obtain the session the ORM object is already attached to, then passes it explicitly to DagModel.get_team_name(dag_id, session=session), bypassing @provide_session's create_session() path entirely. A similar fix was proposed in #70452; this PR simplifies it further by going directly through DagModel instead of the bundle.

Once #70760 is in, no DB call will be required for scheduler-side events in Airflow 3.3.1: the stamp is already in memory. However, the _team_name attribute is private with no stability guarantee; the check is guarded with hasattr and an isinstance check; so we might keep the db call here under try/except with fallback to None.

Two new fields added as well:

DagVersion.version_data (added in Airflow 3.3) carries arbitrary DAG-level metadata set at parse time. This PR surfaces it as dag_version_data in the OpenLineage DagRunInfo facet. No logic — just reading the attribute and attaching it to the event, guarded by AIRFLOW_V_3_3_PLUS.

TaskInstance.note is set when an operator or a user manually changes task state in the UI. Attaching it to OpenLineage task events makes manual interventions observable in lineage.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Sonnet 4.6)

Generated-by: Claude Code (Sonnet 4.6) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@SameerMesiah97 SameerMesiah97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that #70452 can be closed in favour of this. I just have 2 comments for you to address. Also, I would remind you to do a quick manual test before merge to verify this works.

@kacpermuda
kacpermuda force-pushed the openlineage-team-name-session-fix-and-dag-version-data branch from 5ffce56 to 0a4bdc7 Compare July 30, 2026 14:25
@eladkal

eladkal commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

tests are failing

@kacpermuda
kacpermuda force-pushed the openlineage-team-name-session-fix-and-dag-version-data branch from 0a4bdc7 to 2883bb1 Compare July 30, 2026 16:29
@kacpermuda
kacpermuda force-pushed the openlineage-team-name-session-fix-and-dag-version-data branch from 2883bb1 to 8d2379c Compare July 31, 2026 09:36
@uranusjr

Copy link
Copy Markdown
Member

Why do we need to bundle three things in one PR? All three look unrelated to me.

@kacpermuda

Copy link
Copy Markdown
Collaborator Author

Yes, the team_name fix is the main part here, and two other attributes are just small additions to the ol events, just added them as I've noticed while doing the main fix that they are not included in OL events now. I can separate this into two PRs is it's problematic, but given the small size of this change I just put them here. Is that okay ?

@mobuchowski

Copy link
Copy Markdown
Contributor

I agree it seems small enough?

@mobuchowski
mobuchowski merged commit 2f9eb82 into apache:main Jul 31, 2026
195 of 196 checks passed
@kacpermuda
kacpermuda deleted the openlineage-team-name-session-fix-and-dag-version-data branch July 31, 2026 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants