Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Traefik for Edge Agent service #24

Open
robdyke opened this issue Nov 18, 2020 · 34 comments
Open

Use Traefik for Edge Agent service #24

robdyke opened this issue Nov 18, 2020 · 34 comments

Comments

@robdyke
Copy link

robdyke commented Nov 18, 2020

Portainer communicates with the edge agent over port 8000 (as per Edge Agent Guide)

The example Traefik docker-compose does not expose :8000.

As Traefik can proxy any TCP traffic, let's use it?

@xe-nvdk
Copy link
Contributor

xe-nvdk commented Nov 19, 2020

Hi. No need to expose 8000. When you set up the Edge agent, you need to change the Portainer server URL to match the Edge entry point.

@robdyke
Copy link
Author

robdyke commented Nov 19, 2020

Hummmm. So I set https://domain.tld in the config by the edge agent complained that it couldn't reach ws://endpoint on :8000 although it could reach the https:// on :443

@xe-nvdk
Copy link
Contributor

xe-nvdk commented Nov 19, 2020

When you add an edge agent, you need to change the Portainer server's URL to point edge.yourdomain.com. With the current configuration, with any request that came for that URL, Traefik will take and redirect to the port 800 in the container.

Take note that you need to specify one URL for portainer UI that works in port 9000 and another URL for edge.

Anyway, your proposal is valid, so, please, write in a new file to have both alternatives available.

Thank you again for your contribution is very appreciated.
Ignacio

@robdyke
Copy link
Author

robdyke commented Nov 19, 2020 via email

@baskinsy
Copy link

I probably have the same issue or missing something. I have deployed portainer with compose and I have a resolvable second URL (edge.mydomain.com) but edge agents are unable to connect with the following ERROR:

2020/12/22 15:04:25 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]
2020/12/22 15:04:30 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]
2020/12/22 15:04:35 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]
2020/12/22 15:04:40 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]

Edge URL is set when creating the edge endpoint as https://edge.mydomain.com and it is listening but the edge agent refuses to associate. My understanding was that proxying 8000 port with traefik on the separate URL is enough.

@robdyke
Copy link
Author

robdyke commented Dec 22, 2020 via email

@baskinsy
Copy link

The only log on edge side is the above, on portainer side logs are not displaying anything at all. I made a clean test again to verify that and the only log I can find is the above ERROR on egde agent side....

@baskinsy
Copy link

baskinsy commented Dec 22, 2020

I think the tokens are broken somehow. I get "Incorrect padding" when i try to decode the token copied from portainer here https://www.base64code.com/decode

@robdyke
Copy link
Author

robdyke commented Dec 22, 2020

@baskinsy I had those errors when using the provided docker-compose. I opened :8000 direct to portainer and the agent connected. Try the compose file that proxies agent through Traefik

@baskinsy
Copy link

Yes I read your proposal to proxy also 8000 but then is the communication secured? I would try it but that means the provided docker-compose does not work for edge agents as it seems.

@robdyke
Copy link
Author

robdyke commented Dec 22, 2020

@baskinsy the ws:// on :8000 is (as I understand) encrypted

@baskinsy
Copy link

baskinsy commented Dec 22, 2020

Found the issue and was confirmed by a staff member on slack. The KEY generation is broken in my case due to I'm using a four level domain for URL (edge.staging.mydomain.com).

portainer/portainer#4647

@robdyke
Copy link
Author

robdyke commented Dec 22, 2020

Glad yr sorted. Opened an issue upsteam? I often use four.level.doma.is

@baskinsy
Copy link

Yes seems the KEY cannot be decoded when a four level URL is used. When IP is used the decode works. I'll redeploy tomorrow with a third level URL on edge traefic vhost and report back if it works.

@hSinding
Copy link

@baskinsy Did you get it to work with a third level URL?
When i'm trying to decode my KEY i'm getting Incorrect padding aswell, but with an IP it works.

xxxx.mydomain.com

@baskinsy
Copy link

@hSinding Yes I had success on decoding a KEY with a third level domain but I have still issues to connect the edge agent, it is correctly added and registered but cannot be browsed. Although the KEY issue can be circumvent with a third level domain, at least for me.

@reinoldus
Copy link

reinoldus commented Mar 17, 2021

Same issue here, edge just won't connect:

2021/03/17 13:09:08 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: Get https://edge1.domain.com/api/endpoints/5/status: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]

But it seems to be a traefik misconfiguration because when I call the api endpoint manually I get 404

@charnesp
Copy link

With a small hack I was able to make it work through Traefik :

Portainer is setup in Traefik with :

      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend-portainer.rule=Host(`portainer.mydomain.com`)"
      - "traefik.http.services.frontend-portainer.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend-portainer.service=frontend-portainer"
      - "traefik.http.routers.frontend-portainer.tls.certresolver=lets-encrypt"

      # Edge
      - traefik.http.routers.edge-portainer.rule=Host(`edge.mydomain.com`)
      - traefik.http.services.edge-portainer.loadbalancer.server.port=8000
      - traefik.http.routers.edge-portainer.service=edge-portainer
      - traefik.http.routers.edge-portainer.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.regex=^http://(.*)
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.replacement=https://$${1}
      - traefik.http.routers.http.middlewares=edge-portainer-redirect
      - traefik.http.services.http.loadbalancer.server.port=80

You first need to decode the EDGE_KEY (using https://www.base64decode.org for example) in order to obtain the fingerprint and the endpoint ID.

You should obtain something like: https://edge.mydomain.com|edge.mydomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

3 being the endpoint ID, aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00 being the server fingerprint.

I modified it like that : https://portainer.mydomain.com|https://edge.mydomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

and used https://www.base64encode.org (with URL-safe encoding enabled) to regenerate the key.
Then simply use this generated EDGE_KEY on the agent and it should work without exposing any 8000port

@alackmann
Copy link

@charnesp this was such a handy tip. Many thanks. Portainer-peeps - it'd be fantastic to be able to generate an EDGE_KEY from the UI that supports this configuration. At very least an additional (optional) field for the edge URL would help.

@BerkeleyTrue
Copy link

@charnesp Thanks for the help.

I'd like to add I had to add an extra step. With the new key, it complained about a deprecated MD5 fingerprint so I had to update it with a SHA256 fingerprint. No idea why.

image

This really needs to be an option in the UI, or at least update the docs so it's not point edge as a separate url that doesn't work without the 8000 port open.

@mfruhner
Copy link

Hello, I am currently trying to achieve the same: Manage an Edge Agent in a VM via Portainer behind Traefik running in WSL.

As i described here, I try to browse an Edge Agent after successful association. The request to open the tunnel connection however goes to 127.0.0.1 and the connection fails.. Any advice?

@SAOPP
Copy link

SAOPP commented Sep 7, 2022

I'm had made it before like was described at this post #24 (comment)

But agent communication is possible to solve with dns:port and entrypoint also, just needed added your port of edge into traefik ports:

...
      - 8000:8000
...

And entrypoints:

...
      - --entrypoints.edge.address=:8000   
...   

And add second route for this service at portainer's side:

...
    labels:
      - traefik.enable=true
      #portainer route
      - traefik.http.services.portainer.loadbalancer.server.port=9000
      - traefik.http.routers.portainer.rule=Host(`portainer.domain.ltd`)
      - traefik.http.routers.portainer.entrypoints=websecure
      - traefik.http.routers.portainer.service=portainer
      - traefik.http.routers.portainer.tls=true
      - traefik.http.routers.portainer.tls.certresolver=yourresolver
      #edge route
      - traefik.http.services.edge.loadbalancer.server.port=8000
      - traefik.http.routers.edge.rule=Host(`portainer.domain.ltd`)
      - traefik.http.routers.edge.entrypoints=edge
      - traefik.http.routers.edge.service=edge
      - traefik.http.routers.edge.tls=true
      - traefik.http.routers.edge.tls.certresolver=yourresolver
...

So, that expected, we are have portainer at portainer.domain.ltd and edge at portainer.domain.ltd:8000

Now, to access your edge, you just need to add the port to the dns name.

@tarmacx
Copy link

tarmacx commented Oct 19, 2022

In my case had to disable the tls and certresolver to make it work.
Are the data still encrypted by usinf ws instead of wss and disabling tls ?

@agoodshort
Copy link

agoodshort commented Oct 31, 2022

Thanks @SAOPP your steps here helped me, but same as @tarmacx I had to disable tls as I was facing the below error:

2022/10/31 19:53:49 client: Connecting to ws://portainer.mydomain.com:8000
2022/10/31 19:53:49 client: Connection error: websocket: bad handshake
2022/10/31 19:53:49 client: Give up

Now, to access your edge, you just need to add the port to the dns name.

I think I didn't understand this part. Where do you add the port?


By the way, it seems that we are getting really close to the correct config template. We should edit the template on the official portainer docs

@SAOPP
Copy link

SAOPP commented Nov 1, 2022

I think I didn't understand this part. Where do you add the port?

Hi! I mean portainer.mydomain.com:8000 port ova here.

@Jigsaw5279
Copy link

With a small hack I was able to make it work through Traefik :

Portainer is setup in Traefik with :

      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend-portainer.rule=Host(`portainer.mydomain.com`)"
      - "traefik.http.services.frontend-portainer.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend-portainer.service=frontend-portainer"
      - "traefik.http.routers.frontend-portainer.tls.certresolver=lets-encrypt"

      # Edge
      - traefik.http.routers.edge-portainer.rule=Host(`edge.mydomain.com`)
      - traefik.http.services.edge-portainer.loadbalancer.server.port=8000
      - traefik.http.routers.edge-portainer.service=edge-portainer
      - traefik.http.routers.edge-portainer.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.regex=^http://(.*)
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.replacement=https://$${1}
      - traefik.http.routers.http.middlewares=edge-portainer-redirect
      - traefik.http.services.http.loadbalancer.server.port=80

You first need to decode the EDGE_KEY (using https://www.base64decode.org for example) in order to obtain the fingerprint and the endpoint ID.

You should obtain something like: https://edge.mydomain.com|edge.mydomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

3 being the endpoint ID, aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00 being the server fingerprint.

I modified it like that : https://portainer.mydomain.com|https://edge.mydomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

and used https://www.base64encode.org (with URL-safe encoding enabled) to regenerate the key. Then simply use this generated EDGE_KEY on the agent and it should work without exposing any 8000port

I tried setting things up like this but I'm getting bad handshake on the agent machine. I can't see what I did wrong there, any hints?

@charnesp
Copy link

charnesp commented Dec 6, 2022

Hi,

Maybe an incorrectly generated key.

You can use the following script for generating a corrected key :

#!/usr/bin/python3
from base64 import urlsafe_b64encode, urlsafe_b64decode
import argparse

INITIAL_EDGE_URL = "portainer.mydomain.com:8000"
FINAL_EDGE_URL = "https://edge.mydomain.com"

def replace_edge_address_edge_key(edge_key, initial_edge_url = INITIAL_EDGE_URL, final_edge_url = FINAL_EDGE_URL ):
    base64_bytes = edge_key.encode('utf-8')

    message_bytes = urlsafe_b64decode(base64_bytes + b'=' * (-len(base64_bytes) % 4))
    decoded_initial_key = message_bytes.decode('utf-8')
    decoded_corrected_key = decoded_initial_key.replace(initial_edge_url, final_edge_url)
    base64_bytes = decoded_corrected_key.encode('utf-8')
    message_bytes = urlsafe_b64encode(base64_bytes)
    corrected_key = message_bytes.decode('utf-8').replace('=',"")

    return corrected_key, decoded_initial_key

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('edge_key', action='store', type=str,
                        help='Initial encoded edge key')

    parser.add_argument('--initial_edge_url', action='store', type=str,
                         default=INITIAL_EDGE_URL,
                         help='Initial edge url (default: %s)' % INITIAL_EDGE_URL)

    parser.add_argument('--final_edge_url', action='store', type=str,
                         default=FINAL_EDGE_URL,
                         help='Final edge url (default: %s)' % FINAL_EDGE_URL)

    args = parser.parse_args()

    edge_key=args.edge_key
    initial_edge_url=args.initial_edge_url
    final_edge_url=args.final_edge_url

    corrected_key, decoded_initial_key = replace_edge_address_edge_key(edge_key,initial_edge_url = initial_edge_url, final_edge_url = final_edge_url )
    print("Initial decoded key: {} \nCorrected encoded key: {}".format(decoded_initial_key, corrected_key))


if __name__ == '__main__':
    main()

@Jigsaw5279
Copy link

Nope, that didn't do the trick

@charnesp
Copy link

charnesp commented Dec 6, 2022

Did you disabled TLS for your environment, or setup SSL certificate for Portainer? As Portainer is behind Traefik, which handles the HTTPS connections, you should use plain HTTP protocol.

@Jigsaw5279
Copy link

I've found the issue.. I had mistyped the service name in the traefik label, so the request wouldn't ever reach portainer.
"Bad handshake" is a really misleading error though

@SAOPP
Copy link

SAOPP commented Dec 6, 2022

@Jigsaw5279 Please share the details for those who are likely to experience this in the future.

@TimoVerbrugghe
Copy link

TimoVerbrugghe commented Jan 10, 2023

@Jigsaw5279 / @SAOPP / @tarmacx anybody that was able to solve the bad handshake issue?

2023/01/10 10:07:01 client: Connecting to ws://portainer.mydomain.com:8000
2023/01/10 10:07:01 client: Connection error: websocket: bad handshake
2023/01/10 10:07:01 client: Give up

I followed the steps from @SAOPP:

  • Exposed port 8000 on my traefik container & port forwarded from my router
  • Added edge as additional entrypoint
  • Added an additional route to my portainer docker container (doublechecked the service labels)

Traefik uses letsencrypt certs. I added the edge agent with these settings:
Screenshot 2023-01-10 at 11 25 58

It does work when I remove the tls and certresolver labels without changing any other labels, so I'm quite sure it's not a mistyping of service names in the traefik labels...

EDIT: Tried both with force https on/off, no changes. I have to remove the tls and certresolver labels to make this work...

@radarsu
Copy link

radarsu commented Jan 31, 2023

For long time I had similar problem, but error was a bit different (something with ws connection give up). Then I base64 decoded EDGE_KEY and replaced second parameter (edge server address) with portainer.example.com:443, because I have found no other way to change default 8000 port which I didn't want to use.

Error has changed to bad handshake and then commenting out labels fixed the issue. Although that means certificates won't be automatically renewed at least it works. So I'm posting full configuration (in TypeScript instead of yaml) as it might help someone.

Portainer + Portainer Edge

services[portainerConfig.serviceName] = {
    container_name: portainerConfig.serviceName,
    image: `portainer/portainer-ee:2.16.2-alpine`,

    command: `-H unix:///var/run/docker.sock`,
    labels: [
        `traefik.enable=true`,

        // portainer-http
        `traefik.http.routers.${portainerConfig.serviceName}-http.entrypoints=web`,
        `traefik.http.routers.${portainerConfig.serviceName}-http.rule=Host(\`${portainerConfig.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}-http.middlewares=${traefikConfig.serviceName}-redirect-to-https`,
        `traefik.http.routers.${portainerConfig.serviceName}-http.service=${portainerConfig.serviceName}`,

        // portainer
        `traefik.http.routers.${portainerConfig.serviceName}.entrypoints=websecure`,
        `traefik.http.routers.${portainerConfig.serviceName}.rule=Host(\`${portainerConfig.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}.service=${portainerConfig.serviceName}`,
        `traefik.http.routers.${portainerConfig.serviceName}.tls.certresolver=acmeresolver`,
        `traefik.http.routers.${portainerConfig.serviceName}.tls.domains[0].main=${portainerConfig.domain}`,
        `traefik.http.routers.${portainerConfig.serviceName}.tls=true`,
        `traefik.http.services.${portainerConfig.serviceName}.loadbalancer.server.port=9000`,

        // portainer-edge-http
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.entrypoints=web`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.rule=Host(\`${portainerConfig.edge.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.middlewares=${traefikConfig.serviceName}-redirect-to-https`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.service=${portainerConfig.serviceName}-edge`,

        // portainer-edge
        `traefik.http.routers.${portainerConfig.serviceName}-edge.entrypoints=websecure`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge.rule=Host(\`${portainerConfig.edge.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge.service=${portainerConfig.serviceName}-edge`,
        // Commenting those out fixed bad handshake issue.
        // `traefik.http.routers.${portainerConfig.serviceName}-edge.tls.certresolver=acmeresolver`,
        // `traefik.http.routers.${portainerConfig.serviceName}-edge.tls.domains[0].main=${portainerConfig.edge.domain}`,
        // `traefik.http.routers.${portainerConfig.serviceName}-edge.tls=true`,
        `traefik.http.services.${portainerConfig.serviceName}-edge.loadbalancer.server.port=8000`,
    ],
    networks: [`traefik-network`],
    restart: `unless-stopped`,
    volumes: [`/var/run/docker.sock:/var/run/docker.sock`, `${sharedConfig.docker.volumes.root}/${portainerConfig.serviceName}/data:/data`],
};

Portainer Edge Agent (different machine on the Internet)

services[portainerAgentConfig.serviceName] = {
    container_name: portainerAgentConfig.serviceName,
    image: `portainer/agent:2.16.2-alpine`,

    environment: {
        EDGE: `1`,
        EDGE_ID: `<ID>`,
        EDGE_INSECURE_POLL: `0`,
        EDGE_KEY: `<BASE64_DECODED_MODIFIED_AND_ENCODED_BACK_MANUALLY>`
    },
    labels: [`traefik.enable=false`],
    networks: [`traefik-network`],
    restart: `unless-stopped`,
    volumes: [
        `/var/run/docker.sock:/var/run/docker.sock`,
        `/var/lib/docker/volumes:/var/lib/docker/volumes`,
        `/:/host`,
        `${sharedConfig.docker.volumes.root}/${portainerAgentConfig.serviceName}/data:/data`,
    ],
};

@dmr138
Copy link

dmr138 commented Jun 23, 2023

With a small hack I was able to make it work through Traefik :

Portainer is setup in Traefik with :

      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend-portainer.rule=Host(`portainer.mydomain.com`)"
      - "traefik.http.services.frontend-portainer.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend-portainer.service=frontend-portainer"
      - "traefik.http.routers.frontend-portainer.tls.certresolver=lets-encrypt"

      # Edge
      - traefik.http.routers.edge-portainer.rule=Host(`edge.mydomain.com`)
      - traefik.http.services.edge-portainer.loadbalancer.server.port=8000
      - traefik.http.routers.edge-portainer.service=edge-portainer
      - traefik.http.routers.edge-portainer.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.regex=^http://(.*)
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.replacement=https://$${1}
      - traefik.http.routers.http.middlewares=edge-portainer-redirect
      - traefik.http.services.http.loadbalancer.server.port=80

You first need to decode the EDGE_KEY (using https://www.base64decode.org for example) in order to obtain the fingerprint and the endpoint ID.

You should obtain something like: https://edge.mydomain.com|edge.mydomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

3 being the endpoint ID, aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00 being the server fingerprint.

I modified it like that : https://portainer.mydomain.com|https://edge.mydomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

and used https://www.base64encode.org (with URL-safe encoding enabled) to regenerate the key. Then simply use this generated EDGE_KEY on the agent and it should work without exposing any 8000port

Thank you much this worked perfectly!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests