Minio console gives "Invalid login" if internal CA cert is not imported when using reverse proxying inside docker

[pitastrudl]

NOTE

I deployed Minio in Docker swarm and was battling getting the console work for several hours. Minio gave no indication of errors and nginx was only saying '401 unahtorized'. In the end what solved it was to include the CA certificate of our internal CA being used to access our services (including minio console) into the default path /root/.minio/certs/CAs/internal.crt.

Expected Behavior

With the correct proxying settings for nginx and minio config - console is supposed to open.

Current Behavior

With the correct proxying settings for nginx and minio config - console says 'invalid login' even with correct credentials. Minio container should warn if a certificate is missing.

Possible Solution

• Warn in Minio container if an error occurs that would cause 'invalid login'. I am not sure where it fails.
• Note in document https://min.io/docs/minio/linux/operations/network-encryption.html or here https://min.io/docs/minio/linux/integrations/setup-nginx-proxy-with-minio.html that TLS is needed if it's behind a proxy with TLS.

Steps to Reproduce (for bugs)

1. Setup a default minio deployment without a custom certificate in Docker
2. Setup nginx loadbalancer with TLS termination endpoint, reverse proxying console to Minio docker deployment
3. Access console via browser
4. Receive 'invalid login'

Context

Trying to setup minio to be used within services of the Docker swarm deployment. All services are exposed via HTTPs endpoints which Minio supports. Custom internal CA is used for issuing SSL certificates for secure communication. Additional documentation for this could be done.

I only managed to solve this after finding this issue ( minio/minio#19096 ) and translating the (google translate said Chinese) and seeing that they had issues with certificates.

• The issue was impossible to find due to not having any logs, noted here How to set different log levels for minio? minio#14213.
• Similar issue here but unsure what solved it for them Login error when using TLS minio#12883.

On an unrelated note, https://min.io/docs/minio/linux/integrations/setup-nginx-proxy-with-minio.html should note that the console does need access to the API as its defined in the environment variables. This is referenced in the README.md.

Regression

Unknown

Your Environment

• MinIO version used: RELEASE.2024-04-18T19-09-19Z
• Server setup and configuration: Nginx 1.25.3-alpine
• Operating System and version (uname -a): In Docker version 24.0.9

Nginx proxy conf (double $$ due to jinja2 escaping):

    set $$upstream_minio_console http://minio:9001;
    proxy_buffering off;
    location /minioconsole/ {
        rewrite ^/minioconsole/(.*)$$ /$$1 break;
        chunked_transfer_encoding off;
        proxy_pass $$upstream_minio_console;

        proxy_set_header Host $$http_host;
        proxy_set_header X-Real-IP $$remote_addr;
        proxy_set_header X-Forwarded-For $$proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $$scheme;
        proxy_set_header X-NginX-Proxy true;

        #real_ip_header X-Real-IP;

        proxy_connect_timeout 300;
        proxy_set_header Upgrade $$http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_http_version 1.1;
        # proxy_set_header Connection keep-alive;
        proxy_set_header X-Forwarded-Proto $$http_x_forwarded_proto;
        proxy_cache_bypass $$http_upgrade;
        proxy_set_header Origin 'null';

        proxy_request_buffering off;
    }

    set $$upstream_minio_api http://minio:9000;
    location /{

        proxy_set_header Host $$http_host;
        proxy_set_header X-Real-IP $$remote_addr;
        proxy_set_header X-Forwarded-For $$proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $$scheme;
        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "null";
        chunked_transfer_encoding off;
        proxy_pass $$upstream_minio_api;

    }

Docker command: command: 'server /data --console-address ":9001"'

Minio environmental settings:

  MINIO_ROOT_USER: examplekey
  MINIO_ROOT_PASSWORD: examplekeypass
  MINIO_SERVER_URL: 'https://example.local/'
  MINIO_BROWSER_REDIRECT_URL: 'https://example.local/minioconsole/'

[arodriguezd]

I have the same problem when implementing it with Nginx Proxy with SSL.
In the documentation I only see that they use http and not https.
https://min.io/docs/minio/linux/integrations/setup-nginx-proxy-with-minio.html

[pitastrudl]

Hmm, I actually see Im using http to proxypass to minio, wonder if that had any affect. It seems in the docs they use https for frontend and https for proxypass. Sadly cant test untill a week from now. What exactly were you referring to?

[pjuarezd]

Hmm, I actually see Im using http to proxypass to minio, wonder if that had any affect. It seems in the docs they use https for frontend and https for proxypass. Sadly cant test untill a week from now. What exactly were you referring to?

Hello, noticed you have http in example nginx config for upstream servers, this should be https if sending TLS traffic from load balacer to minio Console or MinIO API servers.

set $$upstream_minio_console http://minio:9001;
set $$upstream_minio_api http://minio:9000;

[pjuarezd]

With the correct proxying settings for nginx and minio config - console says 'invalid login' even with correct credentials. Minio container should warn if a certificate is missing.

Mmm not sure it should be warned in the web browser client, think about it MinIO Console running in browser has no way to know the TLS Certificate is not truted from Nginx invoking MinIO Console. Browse is only going to receive a valid https reponse, with a non-satisfactory message. This specific case should be monitored on Nginx, and probably add the trusted TLS CA with ssl_trusted_certificate

Loading

sequenceDiagram
    Browser->>+Nginx: https (trusted certificate)
    Nginx->>-MinIO Console Server : https (non trusted certificate)

[pjuarezd]

Sending this ticket to ducumentation team for the outstanding items cc @ravindk89 @feorlen @djwfyi

[djwfyi]

I notice that you are using Docker while referencing the Linux docs.
I wonder if the TLS docs for containers better address your questions?
https://min.io/docs/minio/container/operations/network-encryption.html#enabling-tls

[pitastrudl]

Hmm, I actually see Im using http to proxypass to minio, wonder if that had any affect. It seems in the docs they use https for frontend and https for proxypass. Sadly cant test untill a week from now. What exactly were you referring to?

Hello, noticed you have http in example nginx config for upstream servers, this should be https if sending TLS traffic from load balacer to minio Console or MinIO API servers.

set $$upstream_minio_console http://minio:9001;
set $$upstream_minio_api http://minio:9000;

Yeah, noticed that after the issue post as well. Though would setting it to https change anything? I am guessing Minio would still need to have a SSL cert imported?

With the correct proxying settings for nginx and minio config - console says 'invalid login' even with correct credentials. Minio container should warn if a certificate is missing.

Mmm not sure it should be warned in the web browser client, think about it MinIO Console running in browser has no way to know the TLS Certificate is not truted from Nginx invoking MinIO Console. Browse is only going to receive a valid https reponse, with a non-satisfactory message. This specific case should be monitored on Nginx, and probably add the trusted TLS CA with ssl_trusted_certificate

Loading

sequenceDiagram
    Browser->>+Nginx: https (trusted certificate)
    Nginx->>-MinIO Console Server : https (non trusted certificate)

Agreed. The SSL certificate is imported inside nginx as other apps depend on it and would be obvious from my side that it would not be working. Nothing else to be done here.

I notice that you are using Docker while referencing the Linux docs. I wonder if the TLS docs for containers better address your questions? https://min.io/docs/minio/container/operations/network-encryption.html#enabling-tls

These docs are much more helpful but there was no way to for me to find the link to it when researching the reverse proxy and trying to make it work. I am aware of just reading the manual, but in the docs for reverse proxying it does not state that https is mandatory, explicitly, and that you'd need to import your CA cert if not trusted. Probably some kind of linkage between these two sites would be great and some more info about it if possible. Thank you for the link.

[garylavayou]

@pitastrudl @pjuarezd since I am notified this issue is closed without further progress, I would like to share some experience I just gained when solving this problem.

If you have used NGINX as a proxy, it would be better to configure SSL in nginx's virtual server, so that the HTTPS connection is terminated at nginx, and nginx will forward the request to the Minio console with plain HTTP (not put certificate files at Minio's --certs-dir=${HOME}/.minio/certs to run Minio server and console in plain HTTP).

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name s3.your.domain;

    ssl_certificate /etc/nginx/ssl/minio.crt;     # certificate for accessing minio's domain
    ssl_certificate_key /etc/nginx/ssl/minio.key; 
    #...
}

In this way, setting MINIO_SERVER_URL='' will address the problem without changing your nginx configuration. This is because MINIO_SERVER_URL is used by Minio console (:9001) to access Minio server's s3 storage service (:9000), not for the browser at the client side. Setting MINIO_SERVER_URL to empty will let Minio Console use the local IP address of the storage service to access the server (It's OK, since they are started from the same command).

Then the client that access minio console only needs to trust minio.crt or your CA's certificate that signed minio.crt to avoid browser's warning about "insecure access".

Similar discussion can be found at Cannot login to minio behind nginx proxy.

Considering various deployment scenarios,

• If you just have a single node deployment (for test purpose), and the access to Minio storage service from Minio console does not need load balancing, then we use the host local IP address is OK.
• For an enterprise level deployment, MINIO_SERVER_URL may need to be set to the reverse proxy's (e.g., nginx) domain to enable load balancing for Minio console's request to the storage service. In this case, the certificate used by the reverse proxy must be trusted by Minio server (as discussed in the next section).
• For a geo-distributed deployment, the owner typically has certificates signed by public CAs with public domain names. So, the certificate trust issue should not be a problem.

---

@pitastrudl Beside, from your post,

In the end what solved it was to include the CA certificate of our internal CA being used to access our services (including minio console) into the default path /root/.minio/certs/CAs/internal.crt.

I also learn that putting CA's certificate at ${HOME}/.minio/certs/CAs will make Minio Console service trust HTTPS connections with certificate that signed by the CA, which however, is not fully explained for the --certs-dir parameter for minio server in the CLI reference. Indeed, there are documents on TLS/CA configuration as we show later.

Before reading this, If I do not clear MINIO_SERVER_URL, I have to try to add the certificate to my host system's CA store, and bind mount the host's /etc/ssl/certs/ca-certificates.crt to the Minio service container (deployed with docker compose), or copy the file to a customized Minio container image. (This method should work in theory but not verified by me, as I prefer serving Minio with plain HTTP behind reverse proxy for a test environment).

After looking around, the official document indeed provides dedicated instructions to enable TLS for Minio services, under deployment of Linux, Docker, etc., which just like what @pitastrudl mentioned.

From the above document, we can see

From Self-signed, Internal, Private Certificates, and Public CAs with Intermediate Certificates: If the MinIO server does not have the necessary CAs, it may return warnings or errors related to TLS validation when connecting to other services.

What problem we confronted is that MinIO console will connect to MinIO s3 storage to accessing the internal system bucket. In this scenario, MinIO s3 storage is the so-called "other service" as quoted above, while Minio client should be regard as a client. So, we need to add its/CA's certificate to the Minio's trust store.

---

Furthermore, does Minio server really validate the TLS certificate of connecting client? My experience is that a client will validate the server's certificate (As we see in browser, if we browse a website without trusted certificate), but does the validation also occur at the server side? If I really need to add a client's certificate to the trust store, where to find it? For example, where is the certificate for a browser like Chrome/Edge, or an application that access S3 storage.

From Third-Party Certificate Authorities: If the MinIO Server cannot match an incoming client’s TLS certificate issuer against any of the available CAs, the server rejects the connection as invalid.

[ravindk89]

This is because MINIO_SERVER_URL is used by Minio console (:9001) to access Minio server's s3 storage service (:9000), not for the browser at the client side. Setting MINIO_SERVER_URL to empty will let Minio Console use the local IP address of the storage service to access the server (It's OK, since they are started from the same command).

So the reason this is closed out is because of this : #3319

We are hoping this should remove the need for MINIO_SERVER_URL entirely, which will in turn help us better understand exactly the ideal nginx config.

I will note - we are not (nor can we be) nginx experts nor provide that level of support. The tutorials we have are best effort for what works in local environments, not necessarily what will work in any or ever production environment. Community feedback usually helps to close the gap a little bit.

If I really need to add a client's certificate to the trust store, where to find it?

I believe it will depend on the certificate. Keep in mind via browser you hit the MinIO Console, which has its own way of talking to the server.

But a client using a third-party or self-signed certificate will, I believe, have some issues talking to MinIO if that 3P CA or self-signed cert key isn't included in the ~/.certs/CAs directory. You may need to use an SDK so you can specify a custom set of TLS certs to fully test this.