Skip to content

Fix OpenLineage team_name access and other failing paths - #70626

Merged
mobuchowski merged 2 commits into
apache:mainfrom
kacpermuda:fix-ol-team-name-and-other
Jul 29, 2026
Merged

Fix OpenLineage team_name access and other failing paths#70626
mobuchowski merged 2 commits into
apache:mainfrom
kacpermuda:fix-ol-team-name-and-other

Conversation

@kacpermuda

Copy link
Copy Markdown
Collaborator

Fixes a set of paths in the OpenLineage listener where a single failing detail discards an entire event, or escapes into the caller.

The headline is the team_name attr accessed with DagBundleModel, that was trying to access the DB from the worker. Now it short-circuits to the team_name in DagRun model, that is forwarded to workers.

Fix Impact before
team_name reads the Execution API value on workers All task events dropped when multi_team is on
Forked emission child always exits A BaseException (e.g. SIGINT to the process group) unwound the child back into the task runner, where it kept running as a second one sharing the supervisor connection
before_stopping detaches the pool and guards its timeout Every Dag run event after a job stopped hit an already-shut-down pool; a hung shutdown raised AirflowTaskTimeout (a BaseException) straight past the caller's except Exception
before_stopping no longer creates a pool to tear down The task runner built a ProcessPoolExecutor at the end of every task purely to shut it down
submit_callable catches RuntimeError BrokenProcessPool subclasses it, so this also covers post-shutdown submissions
get_airflow_state_run_facet guards its DB query A DB hiccup in the pool worker discarded the whole terminal Dag run event instead of just the facet
dag_version_info uses safe_getattr Lazy-loaded DagVersion columns could raise on a detached session
Pool wrappers no longer return the event An unpicklable redacted event reported a successful emission as a failed submission

Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@kacpermuda
kacpermuda requested a review from mobuchowski as a code owner July 28, 2026 14:03
@kacpermuda
kacpermuda force-pushed the fix-ol-team-name-and-other branch 2 times, most recently from 47db132 to 5929d2e Compare July 29, 2026 13:17
@mobuchowski
mobuchowski merged commit c9f06bc into apache:main Jul 29, 2026
102 checks passed
@kacpermuda
kacpermuda deleted the fix-ol-team-name-and-other branch July 29, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants