Skip to content

chore: Upgrade OVN chart/images to 2026.1.3#2143

Merged
cardoe merged 3 commits into
mainfrom
understack-ovn
Jul 22, 2026
Merged

chore: Upgrade OVN chart/images to 2026.1.3#2143
cardoe merged 3 commits into
mainfrom
understack-ovn

Conversation

@skrobul

@skrobul skrobul commented Jul 15, 2026

Copy link
Copy Markdown
Contributor
  • Upgrades the OVN chart to 2026.1.3+a659ab8a2 and points it at the renamed openstack-helm repo (was openstack-helm-infra, which no longer publishes this chart).

  • Updates the OVN images (ovn_ovsdb_nb, ovn_ovsdb_sb, ovn_northd, ovn_controller) from docker.io/openstackhelm/ovn:ubuntu_jammy-20250111 to quay.io/airshipit/ovn:ubuntu_noble-20260711.

  • While validating the upgrade, found that a baremetal/external port on a est network wasn't being claimed by any chassis. Root cause turned out to be a pre-existing (not upgrade-related) bug in neutron_understack.routers.link_vxlan_network_ha_chassis_group: it infers the gateway chassis by requiring every HA_Chassis row in the whole NB database to share one chassis_name. A single stale row left behind by a decommissioned/replaced chassis made that check ambiguous, silently blocking HA_Chassis_Group population for every vxlan network in the fleet, not just the one being tested.

  • Fixed the check to ignore HA_Chassis rows whose chassis no longer exists in the Southbound DB, so a stale row elsewhere doesn't block inference.

  • Added scripts/cleanup_dead_ovn_ha_chassis.py: a dry-run-by-default operator script that (1) removes stale HA_Chassis rows pointing at chassis no longer in the SB DB, and (2) repopulates any per-network HA_Chassis_Group left empty as a result, deriving the chassis from the router's own group — no more manually detaching/reattaching subnets per router to work around it.

Warning

Deploying this requires coordinated updated of the Network Node's OVS.

Deployment

Backup both databases

SB Database:

kubectl exec -i ovn-ovsdb-sb-0 -c ovsdb -- ovsdb-client backup unix:/var/run/ovn/ovnsb_db.sock > ovnsb_backup_$(date +%s).db

NB Database:

kubectl exec -i ovn-ovsdb-nb-0 -c ovsdb -- ovsdb-client backup unix:/var/run/ovn/ovnnb_db.sock > ovnnb_backup_$(date +%s).db

If you are not copying the commands verbatim, note that missing -t from kubectl exec is intentional, otherwise the backup does not work.

Steps

Specifically, the ovn-controller must use the compatible version of OVS libraries to talk to the openvswitch on the host.

On each network node, upgrade OVS from 2.17 to 3.3.4:

apt-mark unhold openvswitch-common openvswitch-switch python3-openvswitch
apt-get install openvswitch-common openvswitch-switch python3-openvswitch
apt-mark hold openvswitch-common openvswitch-switch python3-openvswitch

Then restart all of the OVN related pods.

Manual schema migration might be needed too (check the northd logs):

ovsdb-client convert tcp:ovn-ovsdb-nb-0.ovn-ovsdb-nb.openstack.svc.cluster.local:6641 /usr/share/ovn/ovn-nb.ovsschema
ovsdb-client convert tcp:ovn-ovsdb-sb-0.ovn-ovsdb-sb.openstack.svc.cluster.local:6642 /usr/share/ovn/ovn-sb.ovsschema

Following the restart, monitor the logs for any unusual errors (especially "OVNSB commit failed"). You may need to delete records in Chassis, Chassis_Private tables. Ideally start with ovn-sbctl --db=tcp:ovn-ovsdb-sb-0.ovn-ovsdb-sb.openstack.svc.cluster.local:6642 chassis-del <stale-chassis-name> if stale chassis is found.

If after that the routing still does not work, check the cleanup script included in this PR.

@skrobul
skrobul marked this pull request as ready for review July 16, 2026 08:24
@skrobul
skrobul requested review from cardoe and mfencik July 16, 2026 08:24
skrobul added 3 commits July 20, 2026 10:33
Cleans up stale HA_Chassis rows left behind by decommissioned/replaced
chassis, and repopulates per-network unified HA_Chassis_Groups that were
left empty as a result — avoiding the need to manually detach/reattach
subnets per router to re-trigger neutron_understack's
link_vxlan_network_ha_chassis_group workaround. Dry-run by default.
@cardoe cardoe changed the title chore: Upgrade OVN chart/images to 2026.1.3 and fix chassis-claiming regression chore: Upgrade OVN chart/images to 2026.1.3 Jul 20, 2026
@cardoe

cardoe commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

The chassis regression patch has been merged out of this series.

@cardoe
cardoe added this pull request to the merge queue Jul 22, 2026
Merged via the queue into main with commit cf77ae3 Jul 22, 2026
18 checks passed
@cardoe
cardoe deleted the understack-ovn branch July 22, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants