Port forwarding not working when using Podman against a remote backend

[rm3l]

/kind bug
/area odo-on-podman

What versions of software are you using?

Operating System:
Windows 11 with no Podman Machine setup. I wanted to leverage a Podman backend on another machine.
Not sure this issue could apply to other operating systems.
(Issue discovered while working on #6721)

Output of odo version:

odo v3.13.0 (0d1cf6e36)

Server: https://work-laptop:36443
Kubernetes: v1.27.4+k3s1
Podman Client: 4.6.1

How did you run odo exactly?

This expects a Podman service to be running on some other machine or VM. See https://www.redhat.com/sysadmin/podman-clients-macos-windows

1. Add a default system connection to that remote Podman service, e.g.:

podman system connection add work-laptop \
  --default \
  --identity ~/.ssh/id_rsa \
  ssh://[email protected]

2. Check that the Podman works correctly with that connection, e.g.:

$ podman version

Client:       Podman Engine
Version:      4.6.1
...

Server:       Podman Engine
Version:      4.6.1
...

3. Start a Dev Session on Podman:

$ odo init --name my-node-app --devfile nodejs --starter nodejs-starter
$ odo dev --platform podman

  __
 /  \__     Developing using the "my-node-app" Devfile
 \__/  \    Platform: podman
 /  \__/    odo version: v3.13.0
 \__/

↪ Running on podman in Dev mode
I0825 17:35:12.163048   79214 starterserver.go:123] API Server started at localhost:20000/api/v1
 ✓  Deploying pod [11s]
 ✓  Syncing files into the container [37s]
 ✓  Building your application in container (command: install) [5s]
 •  Executing the application (command: run)  ...
 ✓  Waiting for the application to be ready [3s]
 -  Forwarding from 127.0.0.1:20001 -> 3000
[...]

4. In a separate tab, try to connect to the forwarded port (20001 in the example above):

$ curl http://localhost:20001
curl: (7) Failed to connect to localhost port 20001 after 0 ms: Couldn't connect to server

Actual behavior

Not sure if that is expected when using remote connections like this with Podman, but the dev forwarded ports do not seem to be open on the local machine., but only on the machine running the Podman backend.

PS C:\Users\Armel S> netstat -aon | grep 2000

Active Connections

  Proto  Local Address          Foreign Address        State           PID
  TCP    0.0.0.0:20000          0.0.0.0:0              LISTENING       18608
  TCP    [::]:20000             [::]:0                 LISTENING       18608

Expected behavior

odo says - Forwarding from 127.0.0.1:20001 -> 3000, so I would expect to be able to reach 127.0.0.1:20001.

Workaround

Manually opening an SSH tunnel using the same 20001 port before calling curl does the trick.

Any logs, error output, etc?

[rm3l]

As explained in containers/podman#20027 (comment), this is expected from the Podman remote client, and there is nothing much odo can do about that (other than recommending users to manually open an SSH tunnel after the Dev session is started).
I'll update our troubleshooting guide with a section about that.

EDIT: Added to Troubleshooting page: https://odo.dev/docs/troubleshooting#i-am-using-the-remote-podman-client-and-i-am-unable-to-reach-the-ports-forwarded-by-odo