mt7996 pci reset recovery#193
Open
meehien wants to merge 4 commits into
Open
Conversation
…able Fatal / Surprise Down Error), the firmware stops answering and the MCU message watchdog escalates to mt7996_mac_reset_work(). That worker (and the mt7996_reset() path that queues it) issues MMIO read-modify-writes against the register window of a device that is no longer on the bus. On arm64 with the mtk-pcie-gen3 controller a read from an offline endpoint comes back as an asynchronous external abort -> SError, which is an unconditional kernel panic (no oops, no recovery): SError Interrupt on CPU0, code 0x00000000bf000002 -- SError Kernel panic - not syncing: Asynchronous SError Interrupt mt76_mmio_rmw+0x58/0x120 [mt76] mt7996_rr+0xf0/0x14c [mt7996e] mt7996_update_channel+0x9c/0x178 [mt7996e] mt76_update_survey+0x28/0xf4 [mt76] mt7996_mac_work+0x30/0xe0 [mt7996e] Add a mt7996_dev_gone() helper that reports the endpoint unreachable when either the driver has flagged it removed (MT76_REMOVED) or the AER core has taken the PCIe channel offline, and gate both recovery entry points on it. The pci_channel_offline() check is the load-bearing one for the panic above: the AER core freezes the channel at the start of pcie_do_recovery(), well before the watchdog-triggered reset worker runs, so the guard trips and the worker returns instead of touching dead MMIO.
down) gives the driver no notification that the endpoint has vanished; the log shows 'mt7996e: AER: can't recover (no error_detected callback)'. The reset worker is guarded against an offline channel by the previous patch, but relying solely on pci_channel_offline() misses controllers that do not update pci_dev->error_state, and it leaves the device wedged rather than disconnected. Add an .error_detected handler to mt7996_pci_driver. It sets MT76_REMOVED - the same idiom the mt792x PCIe driver uses on unplug - so all present and future liveness checks (including mt7996_dev_gone()) short-circuit, wakes any thread blocked in the MCU wait so it fails fast, and returns PCI_ERS_RESULT_DISCONNECT. Disconnect is the honest result: the MAC and firmware state cannot survive a PCIe link loss and this driver has no slot_reset re-init path, so an orderly removal is preferable to a doomed recovery attempt. The handler is attached to mt7996_pci_driver only. The hif2 device (second PCIe function) stores a struct mt7996_hif in its drvdata rather than the mt76_dev, so it needs a back-pointer to the owning device before the same handler can be reused there; that is left as a follow-up.
only. The second PCIe function (hif2, bound by mt7996_hif_driver) is the same physical chip, so a fatal AER event on its link downs the whole device just the same - but it had no handler, and its drvdata is a struct mt7996_hif rather than the mt76_dev, so the primary handler cannot be reused as-is. Give struct mt7996_hif a back-pointer to the owning mt7996_dev, populated when the primary function claims the hif during probe, and add an .error_detected handler for mt7996_hif_driver that follows it to the mt76 device and marks it removed. The shared marking logic (set MT76_REMOVED, wake the MCU wait, return PCI_ERS_RESULT_DISCONNECT) is factored into mt7996_pci_mark_removed() and used by both handlers. The hif handler tolerates a not-yet-claimed hif (back-pointer still NULL if the error races the primary function's probe) by reporting the disconnect without dereferencing it.
…anic
into a clean disconnect, but the WiFi stays down until a manual power cycle.
Implement actual PCIe error recovery so the device can come back on its own.
error_detected() now returns PCI_ERS_RESULT_NEED_RESET (unless the channel
has permanently failed), asking the PCIe core to perform a secondary bus
reset - which asserts PERST# to the endpoint and retrains the link - and
then call slot_reset(). The crash logs show this bus reset already succeeds
('Root Port link has been reset', 'ready 0ms after bus reset'); recovery
previously failed only because the driver registered no callbacks.
slot_reset() restores config space (clobbered by the bus reset), optionally
hard power-cycles the card through a board reset-gpios line - needed when
PERST# cannot clear an MCU wedged by a supply brownout, a no-op when the
GPIO is absent - and confirms the chip answers MMIO (MT_HW_REV) before
clearing MT76_REMOVED and driving the existing full chip reset
(mt7996_mac_full_reset() reloads firmware and rebuilds DMA/NAPI/token
state). If the chip does not answer, it returns DISCONNECT, leaving the safe
WiFi-down state of the earlier patches.
pci_save_state() is added at the end of probe so slot_reset() has valid
state to restore. The hif2 function gets a minimal slot_reset that restores
its own config space; the primary function drives the shared chip reinit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please check this series because it was developed with Anthropic Fable 5.
This has fixed the crash I was getting because of the PCIe surprise-down (mostly power related in my case, from SFP+ switching). Not sure about the last patch, since apparently there is a GPIO line that's not connected (https://forum.banana-pi.org/t/mt7996-wifi-error-11-why-rmmod-wont-fix-it-bpi-r4-pro-8x/27523).
My two BE14s are quite stable and they recover without a need for a restart usually so don't know how to reproduce the crash these patches fix reliably. So please check. Have tested these for the past few days and the wifi has been stable and had no crashes, even while swapping the SFP+ live.