Unable to ssh to server after upgrade

[vjoomens]

Hi,

After many failures with previous versions of the tool I finally managed to perform a pretty flawless upgrade with 1.4.5 of centos2alma. There are 2 major issues and 1 minor:

Major:

1. I'm unable to login, either via console or via ssh (client_loop: send disconnect: Broken pipe), looks like the user manager is stopped:
   ip-172-30-2-95 login: root
   Password:
   Last login: Sat Jan 18 2 Stopping User runtime directory /run/user/0...
   [ OK ] Stopped User runtime directory /run/user/0.
   [ OK ] Removed slice User Slice of UID 0.
   [ OK ] Stopped Serial Getty on ttyS0.
   [ OK ] Started Serial Getty on ttyS0.
   [FAILED] Failed to start Seafile hub.
   See 'systemctl status [email protected]' for details.
   [FAILED] Failed to start Seafile hub.
   See 'systemctl status [email protected]' for details.

AlmaLinux 8.10 (Cerulean Leopard)
Kernel 4.18.0-553.34.1.el8_10.x86_64 on an x86_64

ip-172-30-2-95 login:

2. There's a problem with the apache config:
   httpd: Syntax error on line 353 of /etc/httpd/conf/httpd.conf: Syntax error on line 13 of /etc/httpd/conf.d/00_mod_security.conf: No matches for the wildcard '00*exclude.conf' in '/etc/httpd/modsecurity.d', failing (use IncludeOptional if required)

Minor:

1. The Plesk Premium Email installation got corrupted, but that was easily fixed via plesk admin.

Please help me with the 2 major issues, that would bring this upgrade to a positive end.

NOTE on issue 1, I could still ssh the server after the completion of stage 1 (the reboot).

Kind regards,

Victor Oomens

[vjoomens]

Anything on this yet? I'm anxious to do the upgrade, since we're so close now...

[SandakovMM]

Hello @vjoomens,
I apologize for the delay.

I'm unable to login, either via console or via ssh (client_loop: send disconnect: Broken pipe),

Concerning problem 1, we have not encountered similar issues with the conversion process before, so we will need to conduct some debugging.
The log you provided only indicates a problem with the Seafile service. This is peculiar, as Seafile is essentially a file syncing tool, and I am unsure how it could affect authentication. Nevertheless, could you please provide the output of "systemctl status [email protected]"?
As you are also unable to connect via the console, I suspect there may be a PAM misconfiguration. Is it possible to retrieve the /var/log/secure file from the server?

NOTE on issue 1, I could still ssh the server after the completion of stage 1 (the reboot)

Could you please clarify? Are you able to SSH before the first reboot, or can you only connect after the AlmaLinux 8 installation and the first reboot? Or are you unable to connect after the second reboot?

There's a problem with the apache config:

Regarding the apache configuration issue. The file mentioned in the logs is from the old method of installing atomic ruleset. You can safely remove it, and during the next daily maintenance, the ruleset should be reinstalled using the new method.

[vjoomens]

@SandakovMM thank you for taking the time to look at my issues with the tool.

As I write this, I'm doing a fresh test-upgrade on a copy of my Plesk server, still running Centos 7.

Here are my steps:

• Prepare the server for upgrade
• Starting the upgrade: sudo ./centos2alma
• Upgrade starts and server reboots
• During the reboot, the upgrade continues (checking console)
• When the upgrade in the console stops, I get a prompt
• At the prompt, I type reboot
• After the reboot, I can login with SSH

This time, the upgrade seems to have finished. The server is on almalinux 8 and plesk is up and running, although with errors.

There seems to be an issue with the upgrade step using the temp OS. In all cases I found the step not doing the automatic reboot. I had to manually reboot the server and do a ./centos2alma resume. After that, I was unable to SSH the server.

I will try again and see if I have some more information for you.

[vjoomens]

@SandakovMM ,

The upgrade just stops here, see screenshot and latest output log. Unable to login with SSH. I have to do a manual reboot @ the console. After reboot, I can SSH to the server. The upgrade process seems to continue (this time no centos2alma --resume required). During this process I'm still able to SSH the server.

Upgrade process now rebooted the server again. Now after the server restarted, the SSH login is broken:

Connection to ec2-108-129-154-166.eu-west-1.compute.amazonaws.com closed by remote host.
Connection to ec2-108-129-154-166.eu-west-1.compute.amazonaws.com closed.
client_loop: send disconnect: Broken pipe

Login with root @ console also does not work:

ip-172-30-2-210 login: root
Password: 
Last login: Wed Feb 12 13:1          Stopping User runtime directory /run/user/0...
[  OK  ] Stopped User runtime directory /run/user/0.
[  OK  ] Removed slice User Slice of UID 0.
[  OK  ] Stopped Serial Getty on ttyS0.
[  OK  ] Started Serial Getty on ttyS0.

AlmaLinux 8.10 (Cerulean Leopard)
Kernel 4.18.0-553.40.1.el8_10.x86_64 on an x86_64

ip-172-30-2-210 login: 

This is as far as I get. I found some more info in the Plesk log browser:

===============================================================================
Message from the Plesk dist-upgrader tool:
The server is being converted to AlmaLinux 8. Please wait. During the conversion the
server may reboot itself a few times.
To see the current conversion status, run the '/home/centos/centos2alma --status' command.
To monitor the conversion progress in real time, run the '/home/centos/centos2alma --monitor' command.

Last login: Wed Feb 12 11:49:58 2025 from 77-61-136-221.biz.kpn.net

===============================================================================
Message from the Plesk dist-upgrader tool:
The server is being converted to AlmaLinux 8. Please wait. During the conversion the
server may reboot itself a few times.
To see the current conversion status, run the '/home/centos/centos2alma --status' command.
To monitor the conversion progress in real time, run the '/home/centos/centos2alma --monitor' command.

This server is powered by Plesk.
To get the login link use the 'sudo plesk login | grep -v ec2.internal' command.
Use the 'sudo plesk' command to manage the server from the command line.

[centos@ip-172-30-2-210 ~]$ sudo ./centos2alma --status
Conversion process in progress:
( stage Do convert / action adopting repositories ) 01:45 / 13:29 [centos@ip-172-30-2-210 ~]$

almalinux-upgrade-console-output-2025-02-12.txt

reboot-log.txt

[vjoomens]

@SandakovMM

UPDATE, I've rebooted the server with kernel parameter selinux=0, and now all is fine! So hopefully this gives you a pointer on how to fix the upgrade script.

===============================================================================
Message from the Plesk dist-upgrader tool:
During the conversion, some of customized .repo files were updated. You can find the old
files with the .rpmsave extension. Below is a list of the changed files:
/etc/yum.repos.d/pgdg-redhat-all.repo
The roundcube configuration customizations have been relocated to the file '/usr/share/psa-roundcube/config/config.local.php'. This file should be included in the '/usr/share/psa-roundcube/config/config.inc.php'. If this inclusion is missing, please update Plesk to the latest version.
The dovecot configuration '/etc/dovecot/dovecot.conf' has been restored from original distro. Modern configuration was placed in '/usr/local/psa/var/centos2alma/dovecot.conf.conversion.bak'.
The logrotate configuration for rsyslog has been updated. The old configuration has been saved as /usr/local/psa/var/centos2alma/syslog.logrotate.bak
The server has been upgraded to AlmaLinux 8.
You can remove this message from the /etc/motd file.

Last login: Wed Feb 12 13:20:03 2025 from 92.70.157.160

===============================================================================
Message from the Plesk dist-upgrader tool:
During the conversion, some of customized .repo files were updated. You can find the old
files with the .rpmsave extension. Below is a list of the changed files:
/etc/yum.repos.d/pgdg-redhat-all.repo
The roundcube configuration customizations have been relocated to the file '/usr/share/psa-roundcube/config/config.local.php'. This file should be included in the '/usr/share/psa-roundcube/config/config.inc.php'. If this inclusion is missing, please update Plesk to the latest version.
The dovecot configuration '/etc/dovecot/dovecot.conf' has been restored from original distro. Modern configuration was placed in '/usr/local/psa/var/centos2alma/dovecot.conf.conversion.bak'.
The logrotate configuration for rsyslog has been updated. The old configuration has been saved as /usr/local/psa/var/centos2alma/syslog.logrotate.bak
The server has been upgraded to AlmaLinux 8.
You can remove this message from the /etc/motd file.

This server is powered by Plesk.
To get the login link use the 'sudo plesk login | grep -v ec2.internal' command.
Use the 'sudo plesk' command to manage the server from the command line.

[SandakovMM]

There seems to be an issue with the upgrade step using the temp OS. In all cases I found the step not doing the automatic reboot. I had to manually reboot the server and do a ./centos2alma resume

You are absolutely correct; it appears there are some problems within the temporary operating system. Normally, you should not manually restart the instance from it. I can see the error message from leapp PID: 1208 leapp.workflow.RPMUpgrade.dnf_upgrade_transaction: Cannot calculate, check, test, or perform the upgrade transaction. which means dnf returned a non-zero error code. However, I cannot see any specific error from dnf itself.

It is likely causing the issues you mentioned, as the selinux and atomic ruleset related packages were not upgraded during the distribution upgrade process.

I can assume some package on your system is triggering the problem. Usually, dnf inside the temporary OS has issues with encoding (for example non unicode symbols in author name), which you might find earlier in the console log. I believe, the best way to investigate the problem further is to ask an AlmaLinux elevate developer for help.

[vjoomens]

@SandakovMM thank you again for your reply. I've reported the issue to the Elevate community.

I've done another upgrade, this time without package failure. Would you please be so kind to take another look at this upgrade log? Still no automatic reboot after this upgrade process. But everything seems to be fine, except for the selinux issue. I've changed selinux to permissive to check the configuration issues. See attachment.

Can you make something out of this?

Kind regards

leapp-upgrade.log

audit-denied.log