I’ve had two server oses here: alma linux and debian(currently). On both of them, they will hang when I shut them down from cockpit, and they hang at the end of the shutdown.
Also, it takes an hour to a day to have this issue start. if it’s restarted two times in a row quickly, it works perfectly fine for some reason.
What I’ve tried:
- setting “acpi=off” and “acpi=force” kernel parameters in grub
- removing my nvidia gpu(i was using nouveau drivers)
- changing distros
nothing worked. here are some things that both distros had in common with eachother:
- systemd
- cockpit
- libvirt & qemu
- docker
does anyone have advice? nothing i’ve seen online has worked. thank you for suggestions
Is it actual server hardware? I’ve seen some very weird things with real servers that take ages to reboot (I was assuming it was self checking or something). Are you sure its hung, and not just very slow to shutdown/reboot?
Is there any serial/monitor output before the hang?
Monitor output after shutting down:
I’ve given it 6 hours or so to shut down, so it’s almost 100% a hang not a slow shutdown
I had this issue: failed to finalise remaining DM devices. Which led me to here https://github.com/systemd/systemd/issues/15004 and Skinner927 mentions your issue in that thread
I’d try uninstalling nouveau completely and see if the issue persists for you
The
xserver-xorg-video-nouveaupackage was not installed, how else would I remove nouveau?that’s only the X11 “driver” for it. nouveau is built into the kernel, the way to “uninstall” it is to make it not get loaded, by blacklisting it
https://wiki.archlinux.org/title/Nouveau
but this does not seem to be the problem
Agreed,
lsmod | grep nouveaureturns nothing, so I’m not concerned about nouveau or nvidia being the issue here.I don’t have an Nvidia GPU so I don’t have any experience with it but a quick search brought me to Nvidias website and the instructions seem to line up with users answers on other forums.
Disable it here https://docs.nvidia.com/ai-enterprise/deployment/vmware/latest/nouveau.html or apparently installing Nvidias proprietary drivers automatically blacklists Nouveau.
lsmod | grep nouveaureturns nothing, so I assume removing my gpu automatically stopped it from being loaded. that sorta rules out nouveau as an issue assuming it’s not present somewhere else.Direct link to skinners comment: https://github.com/systemd/systemd/issues/15004#issuecomment-2264687287
seems its a nvidia issue, i also have that issue, the gpu locks and i need to reboot while the VM with the nvidia passthrough freezes. i need a full reboot from baremetal machine to stop gpu using all his power stuck, don’t let it be for hours being on or you will kill your hardware
I have removed my gpu and the issue is still present.
yes sorry I read you after writting it, if you remove the GPU the log message is the same but without the GPU lockup line?
do you know that use device mapper? what kind of device is /dev/dm-1 ?
“dmsetup info” might help
sudo dmsetup inforeturns:Name: raven--vg-root State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 254, 0 Number of targets: 1 UUID: LVM-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Name: raven--vg-swap_1 State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 2 Event number: 0 Major, minor: 254, 1 Number of targets: 1 UUID: LVM-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxdid you make these yourself? if not, could you cdo an
ls -l /dev/mapper? it shows which name corresponds to which dm deviceSays reboot, are you issuing a reboot or a shutdown poweroff? Entering sleep state 5 shout be power off right?
I click the reboot button on cockpit, which issues a
shutdown --rebootcommand as root. I agree that sleep state S5 is powered off. From the acpi docs:This likely means my system is failing to reach that s5/g2 state.
If you ssh login directly and issue same command, not In cockpit interface, does it react the same?
no, sorry for not specifying. it’s scrapped together from old consumer components.