Making SSH Nodes work both on Linux and Windows based systems

[dmatej]

TBD: Update description. Please try to review "per commit" - it is a lot of changes, tested on Kubuntu host and Windows Server 2025 in VirtualBox. I know it works, however I could make some mistakes.

When I implemented these tests I had to face quite a lot of issues even on Linux. When I finish, I will write some blog about it, it is another painful episode. However I have to make notes and it will be useful also for users and reviewers. In short:

Assumptions of the test

• The private key doesn't have any passphrase
• DAS uses create-node-ssh --install to "clone" GlassFish installation to another machine.
• Test uses Temurin docker images as a base.

Node side: Environment

• jar is not on PATH everywhere
  • however unzip might be already there - fix: add some autodetection code.

Node side: SSHD Configuration

• SFTP has to be explicitly enabled on some systems
• public key authentication can be disabled. Same applies to other types.
• User interaction can be forbidden.
• Port can be blocked by firewall or antivirus, etc.
• Port can be different.
• ...

DAS side: Environment

• When using scripts, no TTY might be available. Scripts can fail or wait for user input forever.
  • Always set timeout when such risk exists
  • For ssh command is useful sshpass command. Test uses it, however GlassFish uses Jsch library instead.

DAS side: SSH Command Configuration

• Applies to the test and user experiments, not to GlassFish
• Empty passphrase - be careful, when you set it via execInContainer: "''" means apostrophes, not empty passphrase.
• User interaction can be forbidden just like on the server (sshd)
• StrictHostKeyChecking=accept-new should not be used on production; however would be useful if we would not enforce user to run the ssh command on his own and manually accept the server's key. That is TODO.

DAS side: JSCH SSH Implementation

• "raw" and "resolved" password and passphrase. I doubt the implementation is correct.
• logging - fixes will be pushed later
  • race condition - fix: use Jsch logger name.
  • impossible to guess which logger level I should set, overengineered antipattern of "set logger for Jsch once for asadmin, then for kernel, then for cluster, then just jsch, ..." - then you set 4 logger levels and it still doesn't print anything. - fix: always use Jsch logger name.

[dmatej]

Local status: Large refactoring targeting File vs Path vs String vs SFTP vs Unix vs Windows. Local manual testing of DAS on Kubuntu and Node on Windows Server 2025 - was able to "replicate" glassfish installation, unpack, create node and instance. Instance did not start yet as I did not install Java to Windows yet :-)

TODO

• Proper session management (AutoCloseable, separate session and SSHLauncher)
• Data object with some autodetection of Node's operating system and features
• SFTP rmdir does not work on windows
• Documentation, example, ...

[dmatej]

TODO:

• Broken encoding in SSH stdout (hardcoded UTF8 vs Windows-1250)
• Missing autodeploy/bundles (I am often removing those empty directories with .empty file inside, this time it seems Felix cannot autocreate it)
• restart-domain now does not replace WALL_CLOCK_START system property, then GF computes incorrect startup time including CLI (now rather "time since CLI started this process"; on restart-domain the server restarts itself after it was asked to do so by restart-domain).
• SSH adapter should throw the new SSHException, which would always collect exec output. We are losing remote logs sometimes, details about errors. I have fixed it here and there where I needed it, but this should be systematic.