Bugzilla – Bug 1215802
boot freeze with kernel-default=5.14.21-150500.55.28.1 and amdgpu
Last modified: 2023-12-12 16:30:48 UTC
After having installed version 5.14.21-150500.55.28.1 of the kernel release candidate for SLE 15 SP5 / openSUSE 15.5 on my desktop, the system hangs in early boot at: "Reaching target Initrd Root Device." When booting with loglevel=7, the last messages are: "amdgpu 0000:03:00.0: [drm: jpeg_v4_0_early_init [amdgpu]] JPEG decode is enabled in VM mode" This worked with version 5.14.21-150500.55.12.1 and latest release 5.14.21-150500.55.19.1. The release candidate is currently available at https://download.suse.de/ibs/SUSE:/Maintenance:/30736/SUSE_Updates_openSUSE-SLE_15.5/ The AMD GPU is a Radeon RX 7800 XT.
Created attachment 869810 [details] boot messages with default loglevel
Created attachment 869811 [details] boot messages with default loglevel=7
blocks regression fix update
Could not reproduce the freeze on a Ryzen 5500U laptop, or an AMD E2-9000e RADEON R2 or a Ryzen+POLARIS10 using Leap 15.5 and the kernel update from https://download.suse.de/ibs/SUSE:/Maintenance:/30736/SUSE_Updates_openSUSE-SLE_15.5/
Takashi, or someone else from graphics/kernel can you check if there is soemthing problematic here?
Could you try to boot with drm.debug=0x1e boot option? Also, just for narrowing down the regression range, try to boot with the retracted *-55.22.* kernel. It's still found in IBS.
Created attachment 870041 [details] boot options
Created attachment 870042 [details] boot output with drm.debug=0x1e and loglevel=7 I have attached two screen shots above with boot options and boot output
Created attachment 870043 [details] flood of ACPI error messages on next boot After resetting the system on hang (Alt-Sysrq-b), the next boot with the old, known-to-work kernel will fail as well, but for another reason: flood of ACPI message on boot as captured in the screenshot. Only power-cycling the system brings it back to normal.
The retracted *-55.22.* kernel has the same problem.
OK, then another simple check: could you boot with nomodeset boot option and verify whether it boots up or not? If it still hangs, it's not about the amdgpu driver. And, if nomodeset boot option works, try to copy amdgpu.ko.zst from the working Leap 15.5 kernel to the broken kernel. e.g. % mkdir -p /lib/modules/5.14.21-150500.55.28-default/updates % cp /lib/modules/5.14.21-150500.55.19-default/kernel/drivers/gpu/drm/amd/amdgpu.ko.zst /lib/modules/5.14.21-150500.55.28-default/updates/ % depmod -a 5.14.21-150500.55.28-default % dracut -f --kver 5.14.21-150500.55.28-default This will use the old amdgpu module on top of the new kernel.
It(In reply to Takashi Iwai from comment #11) > OK, then another simple check: could you boot with nomodeset boot option and > verify whether it boots up or not? If it still hangs, it's not about the > amdgpu driver. > Booting with nomodeset works. > And, if nomodeset boot option works, try to copy amdgpu.ko.zst from the > working Leap 15.5 kernel to the broken kernel. e.g. > > % mkdir -p /lib/modules/5.14.21-150500.55.28-default/updates > % cp > /lib/modules/5.14.21-150500.55.19-default/kernel/drivers/gpu/drm/amd/amdgpu. > ko.zst /lib/modules/5.14.21-150500.55.28-default/updates/ > % depmod -a 5.14.21-150500.55.28-default > % dracut -f --kver 5.14.21-150500.55.28-default > > This will use the old amdgpu module on top of the new kernel. I can confirm that this fixed my setup.
So far, so good. Could you give the boot log with drm.debug=0x1e from the corrected Leap 15.5 *-28 kernel? You might need to increase the size via log_buf_len=xxx option.
Created attachment 870084 [details] log of successful boot with 55.28
Created attachment 870085 [details] dmesg of successful boot with 55.28
In short, it worked on old Radeon 400 series (like the Polaris 10, see https://de.wikipedia.org/wiki/AMD-Radeon-400-Serie), but seems to fail on the Radeon RX 7000 series (like the RX 7800 XT, see https://de.wikipedia.org/wiki/AMD-Radeon-RX-7000-Series). Takashi, can you tell from the evidence collected which generations of AMD hardware would be affected ? Is there a risk that NVIDIA devices are impacted by the changes in the kernel / driver stack as well ?
As replacing only amdgpu module worked, it's a problem limited to AMD graphics. Unlikely involved with Nvidia. The nest step is to identify the culprit patch. There are 46 changes in amdgpu driver between *-19 and *-22. The best would be a la git-bisect. Unfortunately I caught a flu and will be off for the next days, so the further will be delayed.
Hi, Navi32 (7800 XT) is not supported in the SP5 kernel. Only Navi31 and Navi33 should work. Are you sure the AMDGPU driver got loaded in the previously working kernels or did you just get a firmware framebuffer (simpledrm)?
Never mind my question. You have already provided the logs for me to see what is going on. It seems 7800 XT for some reason works on the older kernel but I would say it's by accident. From my own testing I could never get navi32 working on SP5. I don't think we can get navi32 stable enough to claim support in SP5 so perhaps the better solution here is to blacklist it. AMD is using the IP DISCOVERY method for these boards which makes the driver load if enough IP blocks are detected. Not sure how they intend to handle incomplete support. Intel have a special flag for this. Perhaps AMD should start using one as well.
Created attachment 870115 [details] dmesg and modinfo of amdgpu driver when running 5.14.21-150500.55.19 I check that when running the previous released kernel update 5.14.21-150500.55.19-default the amdgpu driver is loaded and active. Although I have reported this against SLE 15 SP5, all my testing has been done on openSUSE 15.5 assuming that they share the same kernel binary (including the amdgpu driver) ...
Thanks for clarifying. Yes, SLE15-SP5 and Leap 15.5 share the same kernel so it shouldn't matter which one you use. When I boot Leap 15.5 with my 7800 XT the AMDGPU driver fails to load due to missing firmware files. Have you installed any additional firmware files that are not provided by the kernel-firmware-amdgpu package?
When looking closer, it seems my nightly test runs have been using an old version of the kernel-firmware-amdgpu package. Not sure why yet. Updating to the latest kernel-firmware-amdgpu version for 15.5 makes the driver load. So you probably didn't do anything wrong. I need to fix my kiwi descriptions. But it doesn't change the fact that 7800 XT is not supported on 15.5. The best thing I can suggest is that you use the tumbleweed kernel. We need to find a way to blacklist navi32 on 15.5.
(In reply to Patrik Jakobsson from comment #22) > When looking closer, it seems my nightly test runs have been using an old > version of the kernel-firmware-amdgpu package. Not sure why yet. Updating to > the latest kernel-firmware-amdgpu version for 15.5 makes the driver load. > > So you probably didn't do anything wrong. I need to fix my kiwi descriptions. > Yes, I just had installed kernel-firmware-amdgpu from the offical update channels: gargleblaster:~ # zypper search --installed-only -s kernel-firmware-amdgpu Loading repository data... Reading installed packages... S | Name | Type | Version | Arch | Repository --+------------------------+---------+-----------------------+--------+------------------------------------------------------------- i | kernel-firmware-amdgpu | package | 20230724-150500.3.6.1 | noarch | update-sle (15.5) i | kernel-firmware-amdgpu | package | 20230724-150500.3.6.1 | noarch | Update repository with updates from SUSE Linux Enterprise 15 > But it doesn't change the fact that 7800 XT is not supported on 15.5. The > best thing I can suggest is that you use the tumbleweed kernel. > Mhm, after having fixed my setup with the instructions from Takashi (see https://bugzilla.suse.com/show_bug.cgi?id=1215802#c11), I have used the desktop daily and have not had any issue with the amdgpu driver on Navi32. So, even if not supported, it seems to kinda worked ...
Ah great news! So it's just combination with older kernel-firmware.
(In reply to Heiko Rommel from comment #23) > So, even if not supported, it seems to kinda worked ... Yes, perhaps blacklisting the card is a bit aggressive. Maybe we can just say it's experimental. If it works, it works. I did some more testing and tried the 5.14.21-150500.55.28 kernel myself. Everything worked fine. No issues or warnings at all. So I cannot bisect this for you. I'll take a closer look at the patches that might have introduced the issue.
(In reply to Lubos Kocman from comment #24) > Ah great news! So it's just combination with older kernel-firmware. No the issue is still there. I was just surprised that the driver loaded for Heiko when it was not loading for me.
(In reply to Patrik Jakobsson from comment #26) > (In reply to Lubos Kocman from comment #24) > > Ah great news! So it's just combination with older kernel-firmware. > > No the issue is still there. I was just surprised that the driver loaded for > Heiko when it was not loading for me. Wasn't it just the missing Navi32 firmware for SP5-GA (due to the late delivery from AMD)? It's included in the later firmware update.
(In reply to Takashi Iwai from comment #27) > (In reply to Patrik Jakobsson from comment #26) > > (In reply to Lubos Kocman from comment #24) > > > Ah great news! So it's just combination with older kernel-firmware. > > > > No the issue is still there. I was just surprised that the driver loaded for > > Heiko when it was not loading for me. > > Wasn't it just the missing Navi32 firmware for SP5-GA (due to the late > delivery from AMD)? It's included in the later firmware update. My issue was that kiwi didn't build stuff from the update repo when using obs:// URI but worked when I switched it to HTTPS://. I still don't understand why. You can see in the daily IGT test results that navi32 (amd-7800xt) works fine now after I changed it. It never worked with the pre-release firmware so that's why I thought it had always been broken. Heiko is using an updated firmware so that must be a different problem.
(In reply to Patrik Jakobsson from comment #28) > (In reply to Takashi Iwai from comment #27) > > (In reply to Patrik Jakobsson from comment #26) > > > (In reply to Lubos Kocman from comment #24) > > > > Ah great news! So it's just combination with older kernel-firmware. > > > > > > No the issue is still there. I was just surprised that the driver loaded for > > > Heiko when it was not loading for me. > > > > Wasn't it just the missing Navi32 firmware for SP5-GA (due to the late > > delivery from AMD)? It's included in the later firmware update. > > My issue was that kiwi didn't build stuff from the update repo when using > obs:// URI but worked when I switched it to HTTPS://. I still don't > understand why. You can see in the daily IGT test results that navi32 > (amd-7800xt) works fine now after I changed it. It never worked with the > pre-release firmware so that's why I thought it had always been broken. OK. > Heiko is using an updated firmware so that must be a different problem. Yes, it's a clear regression and some of backports must have influenced. Heiko, could you verify that the latest SLE15-SP5 KOTD at OBS Kernel:SLE15-SP5 repo still shows the problem? If yes, for narrowing down the regression range a bit more, I built a few test kernels in OBS home:tiwai:bsc1215802-1, home:tiwai:bsc1215802-2, and home:tiwai:bsc1215802-3 repos, a la bisection (those are with drop of amdgpu patches). Please test with these kernels, too. Note that both KOTD and test kernels are only for testing, and you can uninstall immediately after testing.
Moving to a public product, as the impact will be mostly on Desktop / WS. Please be aware that 7900XTX series recently enabled AI/ML with PyTorch, so they can get quite popular https://www.tomshardware.com/news/amd-enables-rocm-and-pytorch-on-radeon-rx-7900-xtx
(In reply to Takashi Iwai from comment #29) > > Yes, it's a clear regression and some of backports must have influenced. > > Heiko, could you verify that the latest SLE15-SP5 KOTD at OBS > Kernel:SLE15-SP5 repo still shows the problem? > > If yes, for narrowing down the regression range a bit more, I built a few > test kernels in OBS home:tiwai:bsc1215802-1, home:tiwai:bsc1215802-2, and > home:tiwai:bsc1215802-3 repos, a la bisection (those are with drop of amdgpu > patches). Please test with these kernels, too. > > Note that both KOTD and test kernels are only for testing, and you can > uninstall immediately after testing. I have tested the following kernels: repository, version, testdate Kernel:/SLE15-SP5/pool, 5.14.21-150500.198.1.g73350a2, 2023-10-31 home:/tiwai:/bsc1215802-1/pool, 5.14.21-150500.1.1.g812bcdb, 2023-10-31 home:/tiwai:/bsc1215802-2/pool, 5.14.21-150500.1.1.gbc99036, 2023-10-31 home:/tiwai:/bsc1215802-3/pool, 5.14.21-150500.1.1.ga644019, 2023-10-31 Unfortunately, none of them worked, showing the same symptoms as initially reported. Please let me know if you are interested in the logs of any of those above. Just to double check, I had installed kernel-firmware-amdgpu-20230724-150500.3.6.1.noarch during all of those tests (latest published version).
Thanks. Then the possible causes are in the rest few other patches. I built a few more test kernels in OBS home:tiwai:bsc1215802-4, home:tiwai:bsc1215802-5, home:tiwai:bsc1215802-6 and home:tiwai:bsc1215802-7. Please check those. The *-4 kernel should work. If it doesn't, skip testing others.
I have tested the following kernels: result, repository, version, testdate WORKED, home:/tiwai:/bsc1215802-4/pool, 5.14.21-150500.1.1.g14157be, 2023-11-06 FAILED, home:/tiwai:/bsc1215802-5/pool, 5.14.21-150500.1.1.g0fb085e, 2023-11-06 WORKED, home:/tiwai:/bsc1215802-6/pool, 5.14.21-150500.1.1.g2061c39, 2023-11-06 FAILED, home:/tiwai:/bsc1215802-7/pool, 5.14.21-150500.1.1.g9fac727, 2023-11-06
(In reply to Heiko Rommel from comment #33) > I have tested the following kernels: > > result, repository, version, testdate > > WORKED, home:/tiwai:/bsc1215802-4/pool, 5.14.21-150500.1.1.g14157be, > 2023-11-06 > FAILED, home:/tiwai:/bsc1215802-5/pool, 5.14.21-150500.1.1.g0fb085e, > 2023-11-06 > WORKED, home:/tiwai:/bsc1215802-6/pool, 5.14.21-150500.1.1.g2061c39, > 2023-11-06 > FAILED, home:/tiwai:/bsc1215802-7/pool, 5.14.21-150500.1.1.g9fac727, > 2023-11-06 Thanks! It indicated that the likely culprit is patches.suse/drm-amdgpu-add-vram-reservation-based-on-vram_usageb.patch that is a backport of the upstream commit 4864f2ee9ee2acf4a1009b58fbc62f17fa086d4e I'll revert it and another relevant patch patches.suse/drm-amdgpu-Remove-unnecessary-domain-argument.patch
On OBS home:tiwai:bsc1215802=8, another test kernel is being built. It's with the revert of those two patches on top of the latest SLE15-SP5 git branch. Please confirm that it's still working later, once after the package build finishes. Meanwhile the reverts are in my PR to SLE15-SP5 branch.
WORKED, home:/tiwai:/bsc1215802-8/pool, 5.14.21-150500.1.g7bd1588, 2023-11-06
Thanks for confirmation. The revert is pushed to my SLE15-SP5/for-next.
Nice work in finding the regression. It's not clear to me why the offending patch didn't trigger a problem on my 7800 XT. The only reason I can think of is that we have different firmware versions (mine is pre production). I'll check if there is an update for my board.
The upstream devs suggested a possibly missing patch for the combination of the found culprit commit. I built another test kernel with all those applied again in OBS home:tiwai:bsc1215802-revive repo. Could you try it and report back if this works for you?
WORKED, home:/tiwai:/bsc1215802-revive/pool, 5.14.21-150500.1.1.g4bd84ea, 2023-11-21
Thanks! Then let's re-introduce the upstream patches. Pushed to SLE15-SP5/for-next now.
SUSE-SU-2023:4732-1: An update that solves 15 vulnerabilities, contains three features and has 39 security fixes can now be installed. Category: security (important) Bug References: 1207948, 1210447, 1212649, 1214286, 1214700, 1214840, 1214976, 1215095, 1215123, 1215124, 1215292, 1215420, 1215458, 1215710, 1215802, 1215931, 1216058, 1216105, 1216259, 1216527, 1216584, 1216621, 1216687, 1216693, 1216759, 1216761, 1216788, 1216844, 1216861, 1216909, 1216959, 1216965, 1216976, 1217036, 1217068, 1217086, 1217095, 1217124, 1217140, 1217147, 1217195, 1217196, 1217200, 1217205, 1217332, 1217366, 1217511, 1217515, 1217598, 1217599, 1217609, 1217687, 1217731, 1217780 CVE References: CVE-2023-2006, CVE-2023-25775, CVE-2023-3777, CVE-2023-39197, CVE-2023-39198, CVE-2023-4244, CVE-2023-45863, CVE-2023-45871, CVE-2023-46813, CVE-2023-46862, CVE-2023-5158, CVE-2023-5633, CVE-2023-5717, CVE-2023-6039, CVE-2023-6176 Jira References: PED-3184, PED-5021, PED-7237 Sources used: openSUSE Leap 15.5 (src): kernel-source-rt-5.14.21-150500.13.27.2, kernel-syms-rt-5.14.21-150500.13.27.1, kernel-livepatch-SLE15-SP5-RT_Update_8-1-150500.11.3.2 SUSE Linux Enterprise Live Patching 15-SP5 (src): kernel-livepatch-SLE15-SP5-RT_Update_8-1-150500.11.3.2 SUSE Real Time Module 15-SP5 (src): kernel-source-rt-5.14.21-150500.13.27.2, kernel-syms-rt-5.14.21-150500.13.27.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2023:4734-1: An update that solves 13 vulnerabilities, contains three features and has 38 security fixes can now be installed. Category: security (important) Bug References: 1084909, 1207948, 1210447, 1214286, 1214700, 1214840, 1214976, 1215123, 1215124, 1215292, 1215420, 1215458, 1215710, 1215802, 1215931, 1216058, 1216105, 1216259, 1216527, 1216584, 1216687, 1216693, 1216759, 1216788, 1216844, 1216861, 1216909, 1216959, 1216965, 1216976, 1217036, 1217068, 1217086, 1217095, 1217124, 1217140, 1217147, 1217195, 1217196, 1217200, 1217205, 1217332, 1217366, 1217511, 1217515, 1217598, 1217599, 1217609, 1217687, 1217731, 1217780 CVE References: CVE-2023-2006, CVE-2023-25775, CVE-2023-39197, CVE-2023-39198, CVE-2023-4244, CVE-2023-45863, CVE-2023-45871, CVE-2023-46862, CVE-2023-5158, CVE-2023-5633, CVE-2023-5717, CVE-2023-6039, CVE-2023-6176 Jira References: PED-3184, PED-5021, PED-7237 Sources used: openSUSE Leap 15.5 (src): kernel-syms-azure-5.14.21-150500.33.26.1, kernel-source-azure-5.14.21-150500.33.26.1 Public Cloud Module 15-SP5 (src): kernel-syms-azure-5.14.21-150500.33.26.1, kernel-source-azure-5.14.21-150500.33.26.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
SUSE-SU-2023:4730-1: An update that solves 13 vulnerabilities, contains three features and has 38 security fixes can now be installed. Category: security (important) Bug References: 1084909, 1207948, 1210447, 1214286, 1214700, 1214840, 1214976, 1215123, 1215124, 1215292, 1215420, 1215458, 1215710, 1215802, 1215931, 1216058, 1216105, 1216259, 1216527, 1216584, 1216687, 1216693, 1216759, 1216788, 1216844, 1216861, 1216909, 1216959, 1216965, 1216976, 1217036, 1217068, 1217086, 1217095, 1217124, 1217140, 1217147, 1217195, 1217196, 1217200, 1217205, 1217332, 1217366, 1217511, 1217515, 1217598, 1217599, 1217609, 1217687, 1217731, 1217780 CVE References: CVE-2023-2006, CVE-2023-25775, CVE-2023-39197, CVE-2023-39198, CVE-2023-4244, CVE-2023-45863, CVE-2023-45871, CVE-2023-46862, CVE-2023-5158, CVE-2023-5633, CVE-2023-5717, CVE-2023-6039, CVE-2023-6176 Jira References: PED-3184, PED-5021, PED-7237 Sources used: openSUSE Leap 15.5 (src): kernel-syms-5.14.21-150500.55.39.1, kernel-source-5.14.21-150500.55.39.1, kernel-livepatch-SLE15-SP5_Update_8-1-150500.11.3.1, kernel-obs-build-5.14.21-150500.55.39.1, kernel-obs-qa-5.14.21-150500.55.39.1, kernel-default-base-5.14.21-150500.55.39.1.150500.6.17.1 SUSE Linux Enterprise Micro 5.5 (src): kernel-default-base-5.14.21-150500.55.39.1.150500.6.17.1 Basesystem Module 15-SP5 (src): kernel-source-5.14.21-150500.55.39.1, kernel-default-base-5.14.21-150500.55.39.1.150500.6.17.1 Development Tools Module 15-SP5 (src): kernel-source-5.14.21-150500.55.39.1, kernel-syms-5.14.21-150500.55.39.1, kernel-obs-build-5.14.21-150500.55.39.1 SUSE Linux Enterprise Live Patching 15-SP5 (src): kernel-livepatch-SLE15-SP5_Update_8-1-150500.11.3.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.