Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bonfire timeout SSL sessions when fetching the components #454

Open
Sgitario opened this issue Jan 3, 2025 · 12 comments
Open

Bonfire timeout SSL sessions when fetching the components #454

Sgitario opened this issue Jan 3, 2025 · 12 comments

Comments

@Sgitario
Copy link

Sgitario commented Jan 3, 2025 •
edited
Loading

I followed these instructions for a fresh installation of bonfire: https://github.com/RedHatInsights/bonfire?tab=readme-ov-file#installing-locally

And when doing a bonfire deployment using "bonfire deploy rhsm --no-remove-resources app:rhsm", it keeps failing with:

2025-01-07 08:13:16 [   DEBUG] [          MainThread] should_remove_dependencies evaluates to False
2025-01-07 08:13:16 [    INFO] [          MainThread] ignoring component curiosity-frontend, user opted to disable frontend deployments
2025-01-07 08:13:16 [   DEBUG] [          MainThread] parsing optionalDependencies for component 'curiosity-frontend' (a member of app group 'rhsm')
2025-01-07 08:13:16 [   DEBUG] [          MainThread] app 'rhsm' has component 'wiremock'
2025-01-07 08:13:16 [    INFO] [          MainThread] --> processing component wiremock
2025-01-07 08:13:16 [   DEBUG] [          MainThread] component: 'wiremock' fetching template using git ref 'master'
2025-01-07 08:13:16 [   DEBUG] [          MainThread] attempting ref 'master'
2025-01-07 08:13:16 [   DEBUG] [          MainThread] checking connection to 'api.github.com', port 443, timeout 5sec
2025-01-07 08:13:16 [   DEBUG] [          MainThread] using GITHUB_API_URL 'https://api.github.com' with GITHUB_TOKEN
2025-01-07 08:13:16 [   DEBUG] [          MainThread] Starting new HTTPS connection (1): api.github.com:443
2025-01-07 08:13:17 [   DEBUG] [          MainThread] https://api.github.com:443 "GET /repos/RedHatInsights/wiremock-consoledot/git/refs/heads/master HTTP/1.1" 404 None
2025-01-07 08:13:17 [    INFO] [          MainThread] failed to fetch git ref 'master' (http code: 404, response txt: {"message":"Not Found","documentation_url":"https://docs.github.com/rest/git/refs#get-all-references-in-a-namespace","status":"404"})
2025-01-07 08:13:17 [    INFO] [          MainThread] trying alternate: main
2025-01-07 08:13:17 [   DEBUG] [          MainThread] attempting ref 'main'
2025-01-07 08:13:17 [   DEBUG] [          MainThread] https://api.github.com:443 "GET /repos/RedHatInsights/wiremock-consoledot/git/refs/heads/main HTTP/1.1" 200 None
2025-01-07 08:13:17 [    INFO] [          MainThread] fetch succeeded for ref 'main'
2025-01-07 08:13:17 [   DEBUG] [          MainThread] Starting new HTTPS connection (1): raw.githubusercontent.com:443
2025-01-07 08:14:18 [   ERROR] [          MainThread] failed to fetch template file for wiremock
2025-01-07 08:14:18 [   DEBUG] [          MainThread] Traceback (most recent call last):
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connection.py", line 741, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/util/ssl_.py", line 460, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/util/ssl_.py", line 504, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/usr/lib64/python3.12/ssl.py", line 1319, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/bonfire/processor.py", line 658, in _get_component_items
    commit, template_content = rf.fetch()
                               ^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/bonfire/utils.py", line 200, in fetch
    result = self._fetch_github()
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/bonfire/utils.py", line 350, in _fetch_github
    response = self._session.get(url, headers=self._gh_auth_headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))

2025-01-07 08:14:18 [   ERROR] [          MainThread] hit fatal error: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))
2025-01-07 08:14:18 [    INFO] [          MainThread] releasing namespace 'ephemeral-qvdwh7'
2025-01-07 08:14:19 [    INFO] [          MainThread] processing namespace reservation
2025-01-07 08:14:19 [ WARNING] [          MainThread] converted template's deprecated apiVersion 'v1' to 'template.openshift.io/v1'
2025-01-07 08:14:19 [    INFO] [          MainThread] running (pid 13178): oc apply -f - 
2025-01-07 08:14:20 [    INFO] [           pid-13178]  |stdout| namespacereservation.cloud.redhat.com/bonfire-reservation-765ccfb1 configured
2025-01-07 08:14:20 [    INFO] [          MainThread] releasing reservation 'bonfire-reservation-765ccfb1' namespace 'ephemeral-qvdwh7'
2025-01-07 08:14:20 [    INFO] [          MainThread] deploy failed: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))
ERROR: deploy failed: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))

bonfire version: 6.0.2

Using the bonfire version 6.0.1 works fine.
@bsquizz
Copy link
Collaborator

bsquizz commented Jan 3, 2025

Can you clarify -- is the token in the file named config.yaml or in the file named env?

It should be in the file named env, for example:

echo 'GITHUB_TOKEN=<your api token>' >> ~/.config/bonfire/env

@Sgitario
Copy link
Author

Sgitario commented Jan 4, 2025

Can you clarify -- is the token in the file named config.yaml or in the file named env?

It should be in the file named env, for example:

echo 'GITHUB_TOKEN=<your api token>' >> ~/.config/bonfire/env

It is in the file named "env". Description updated.

Is there any logging we can enable to better troubloshoot what is going on?

@bsquizz
Copy link
Collaborator

bsquizz commented Jan 6, 2025

Yes, you can run bonfire --debug <args> to get much more detailed log output.

It's true, I am able to reach https://raw.githubusercontent.com/RedHatInsights/curiosity-frontend/04c796e27790839aafbad7da44e402843a9407f8/deploy/frontend.yaml even from an un-authenticated browser, which makes me wonder if something is wrong with your token.

Has your github token possibly expired? Is it missing a needed permission? If you remove the github token configuration from the ~/.config/bonfire/env, does this command start to work?

@Sgitario
Copy link
Author

Sgitario commented Jan 7, 2025

Yes, you can run bonfire --debug <args> to get much more detailed log output.

It's true, I am able to reach https://raw.githubusercontent.com/RedHatInsights/curiosity-frontend/04c796e27790839aafbad7da44e402843a9407f8/deploy/frontend.yaml even from an un-authenticated browser, which makes me wonder if something is wrong with your token.

Has your github token possibly expired? Is it missing a needed permission? If you remove the github token configuration from the ~/.config/bonfire/env, does this command start to work?

The token that I'm using is new and I added all the permissions.

Using the --debug option, it confirms that the token is taken from ~/.config/bonfire/env as expected. Yet, I got the same error. I also tried to remove the token from the file, and I get the same error.

Moreover, I tried it with python 3.12 and 3.13 with the same failure.

@Sgitario
Copy link
Author

Sgitario commented Jan 7, 2025

Adding more logs:

2025-01-07 07:43:27 [    INFO] [          MainThread] local configuration found for apps: ['my_custom_app']
2025-01-07 07:43:27 [    INFO] [          MainThread] diff in apps config after merging local config into remote config:
--- 

+++ 

@@ -1561,6 +1561,11 @@

                                           'ref': '8d8f77e7e118915bc0e8aee9d95169111e20870b',
                                           'repo': 'RedHatInsights/module-update-router'}],
                           'name': 'module-update-router'},
+ 'my_custom_app': {'components': [{'host': 'github',
+                                   'name': 'my_app_component',
+                                   'path': 'deploy/template.yaml',
+                                   'repo': 'user/repo'}],
+                   'name': 'my_custom_app'},
  'notifications': {'components': [{'host': 'github',
                                    'name': 'notifications-backend',
                                    'parameters': {'BACKOFFICE_HOST': 'backoffice-proxy.apps.ext.spoke.preprod.us-west-2.aws.paas.redhat.com',
2025-01-07 07:43:27 [    INFO] [          MainThread] processing app 'rhsm'
2025-01-07 07:43:27 [    INFO] [          MainThread] --> processing component curiosity-frontend
2025-01-07 07:43:27 [ WARNING] [          MainThread] http response 404 for url https://raw.githubusercontent.com/RedHatInsights/curiosity-frontend/04c796e27790839aafbad7da44e402843a9407f8/deploy/frontend.yaml, checking for template in current working dir...
2025-01-07 07:43:28 [   ERROR] [          MainThread] failed to fetch template file for curiosity-frontend
2025-01-07 07:43:28 [   ERROR] [          MainThread] hit fatal error: [Errno 2] No such file or directory: '/home/jcarvaja/sources/RedHatInsights/rhsm-subscriptions/deploy/frontend.yaml'
2025-01-07 07:43:28 [    INFO] [          MainThread] releasing namespace 'ephemeral-zadfl2'

The "my_custom_app" is automatically added to the bonfire config with an application that does not exist. Could be that the requests session is being "polluted" because of this? I mean, it gets a 404 for a non existing resource, and when getting an existing resource using the same requests session, it also gets 404.

@Sgitario
Copy link
Author

Sgitario commented Jan 7, 2025

If I copy the frontend.yaml file to the local folder to move forward, then it randomly fails starting a new HTTPS connection in any of the components to be installed:

2025-01-07 08:13:16 [   DEBUG] [          MainThread] should_remove_dependencies evaluates to False
2025-01-07 08:13:16 [    INFO] [          MainThread] ignoring component curiosity-frontend, user opted to disable frontend deployments
2025-01-07 08:13:16 [   DEBUG] [          MainThread] parsing optionalDependencies for component 'curiosity-frontend' (a member of app group 'rhsm')
2025-01-07 08:13:16 [   DEBUG] [          MainThread] app 'rhsm' has component 'wiremock'
2025-01-07 08:13:16 [    INFO] [          MainThread] --> processing component wiremock
2025-01-07 08:13:16 [   DEBUG] [          MainThread] component: 'wiremock' fetching template using git ref 'master'
2025-01-07 08:13:16 [   DEBUG] [          MainThread] attempting ref 'master'
2025-01-07 08:13:16 [   DEBUG] [          MainThread] checking connection to 'api.github.com', port 443, timeout 5sec
2025-01-07 08:13:16 [   DEBUG] [          MainThread] using GITHUB_API_URL 'https://api.github.com' with GITHUB_TOKEN
2025-01-07 08:13:16 [   DEBUG] [          MainThread] Starting new HTTPS connection (1): api.github.com:443
2025-01-07 08:13:17 [   DEBUG] [          MainThread] https://api.github.com:443 "GET /repos/RedHatInsights/wiremock-consoledot/git/refs/heads/master HTTP/1.1" 404 None
2025-01-07 08:13:17 [    INFO] [          MainThread] failed to fetch git ref 'master' (http code: 404, response txt: {"message":"Not Found","documentation_url":"https://docs.github.com/rest/git/refs#get-all-references-in-a-namespace","status":"404"})
2025-01-07 08:13:17 [    INFO] [          MainThread] trying alternate: main
2025-01-07 08:13:17 [   DEBUG] [          MainThread] attempting ref 'main'
2025-01-07 08:13:17 [   DEBUG] [          MainThread] https://api.github.com:443 "GET /repos/RedHatInsights/wiremock-consoledot/git/refs/heads/main HTTP/1.1" 200 None
2025-01-07 08:13:17 [    INFO] [          MainThread] fetch succeeded for ref 'main'
2025-01-07 08:13:17 [   DEBUG] [          MainThread] Starting new HTTPS connection (1): raw.githubusercontent.com:443
2025-01-07 08:14:18 [   ERROR] [          MainThread] failed to fetch template file for wiremock
2025-01-07 08:14:18 [   DEBUG] [          MainThread] Traceback (most recent call last):
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connection.py", line 741, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connection.py", line 920, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/util/ssl_.py", line 460, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/util/ssl_.py", line 504, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/usr/lib64/python3.12/ssl.py", line 1319, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/bonfire/processor.py", line 658, in _get_component_items
    commit, template_content = rf.fetch()
                               ^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/bonfire/utils.py", line 200, in fetch
    result = self._fetch_github()
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/bonfire/utils.py", line 350, in _fetch_github
    response = self._session.get(url, headers=self._gh_auth_headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jcarvaja/bonfire_venv/lib64/python3.12/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))

2025-01-07 08:14:18 [   ERROR] [          MainThread] hit fatal error: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))
2025-01-07 08:14:18 [    INFO] [          MainThread] releasing namespace 'ephemeral-qvdwh7'
2025-01-07 08:14:19 [    INFO] [          MainThread] processing namespace reservation
2025-01-07 08:14:19 [ WARNING] [          MainThread] converted template's deprecated apiVersion 'v1' to 'template.openshift.io/v1'
2025-01-07 08:14:19 [    INFO] [          MainThread] running (pid 13178): oc apply -f - 
2025-01-07 08:14:20 [    INFO] [           pid-13178]  |stdout| namespacereservation.cloud.redhat.com/bonfire-reservation-765ccfb1 configured
2025-01-07 08:14:20 [    INFO] [          MainThread] releasing reservation 'bonfire-reservation-765ccfb1' namespace 'ephemeral-qvdwh7'
2025-01-07 08:14:20 [    INFO] [          MainThread] deploy failed: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))
ERROR: deploy failed: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1000)')))

@Sgitario
Copy link
Author

Sgitario commented Jan 7, 2025

Using bonfire version: 6.0.1 works fine, so I guess there is a regression using the latest binaries. Issue updated.

@Sgitario Sgitario changed the title Bonfire does not use the GITHUB_TOKEN set in ~/.config/bonfire/env Bonfire timeout SSL sessions when fetching the components Jan 7, 2025
@bsquizz
Copy link
Collaborator

bsquizz commented Jan 7, 2025

Can you provide the deployment args you are using so I can try to reproduce? Either here or via RH slack if there's any sensitive info. Thanks!

@Sgitario
Copy link
Author

Sgitario commented Jan 8, 2025

bonfire deploy rhsm --no-remove-resources app:rhsm

It was in the issue description: "bonfire deploy rhsm --no-remove-resources app:rhsm".
Repo: https://github.com/RedHatInsights/rhsm-subscriptions

@bsquizz
Copy link
Collaborator

bsquizz commented Jan 8, 2025

Woops sorry, totally missed that. OK so, I'm unable to reproduce 😬

I tried on a virtual environment with python 3.11.9 and also with 3.12.7

2025-01-08 16:22:50 [   DEBUG] [          MainThread] app 'rhsm' has component 'wiremock'
2025-01-08 16:22:50 [    INFO] [          MainThread] --> processing component wiremock
2025-01-08 16:22:50 [   DEBUG] [          MainThread] component: 'wiremock' fetching template using git ref 'master'
2025-01-08 16:22:50 [   DEBUG] [          MainThread] attempting ref 'master'
2025-01-08 16:22:50 [   DEBUG] [          MainThread] checking connection to 'api.github.com', port 443, timeout 5sec
2025-01-08 16:22:50 [   DEBUG] [          MainThread] using GITHUB_API_URL 'https://api.github.com' with GITHUB_TOKEN
2025-01-08 16:22:50 [   DEBUG] [          MainThread] Starting new HTTPS connection (1): api.github.com:443
2025-01-08 16:22:50 [   DEBUG] [          MainThread] https://api.github.com:443 "GET /repos/RedHatInsights/wiremock-consoledot/git/refs/heads/master HTTP/1.1" 404 None
2025-01-08 16:22:50 [    INFO] [          MainThread] failed to fetch git ref 'master' (http code: 404, response txt: {"message":"Not Found","documentation_url":"https://docs.github.com/rest/git/refs#get-all-references-in-a-namespace","status":"404"})
2025-01-08 16:22:50 [    INFO] [          MainThread] trying alternate: main
2025-01-08 16:22:50 [   DEBUG] [          MainThread] attempting ref 'main'
2025-01-08 16:22:51 [   DEBUG] [          MainThread] https://api.github.com:443 "GET /repos/RedHatInsights/wiremock-consoledot/git/refs/heads/main HTTP/1.1" 200 None
2025-01-08 16:22:51 [    INFO] [          MainThread] fetch succeeded for ref 'main'
2025-01-08 16:22:51 [   DEBUG] [          MainThread] Starting new HTTPS connection (1): raw.githubusercontent.com:443
2025-01-08 16:22:51 [   DEBUG] [          MainThread] https://raw.githubusercontent.com:443 "GET /RedHatInsights/wiremock-consoledot/8b81a334173712fa28bd60d23f5011fe1e60e4aa/deploy/clowdapp.yaml HTTP/1.1" 200 1194

Since this looks like an SSL connection error ... I have some questions:

  1. Is this still failing for you today on 6.0.2? Or was there just a transient error with raw.githubusercontent.com?
  2. Is HTTP_PROXY or HTTPS_PROXY set as an env var? (I am doubtful... since you said downgrading to 6.0.1 worked, but want to double check)
  3. Could anything be different between our environments w/ respect to package versions... especially with openSSL, urllib, requests, etc.? (see my env details below). If there's differences it may be helpful to remove the virtual environment and re-create it just to rule anything out there.

If all this is ruled out, there may be some more things to try. Based on some searches I've been doing, the cause behind this particular error seems to be a tough one to pin down sometimes.

❯ python --version
Python 3.12.7

❯ bonfire version
bonfire version 6.0.2

❯ python -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 3.2.2 4 Jun 2024

❯ pip freeze
anyio==4.8.0
anytree==2.12.1
app-common-python==0.2.7
babel==2.16.0
backoff==2.2.1
cached-property==2.0.1
cachetools==5.5.0
certifi==2024.12.14
chardet==5.2.0
charset-normalizer==3.4.1
click==8.1.8
crc-bonfire==6.0.2
distlib==0.3.9
durationpy==0.9
filelock==3.16.1
fsspec==2024.6.1
ghp-import==2.1.0
google-auth==2.37.0
gql==3.5.0
graphql-core==3.2.5
huggingface-hub==0.24.0
idna==3.10
junitparser==3.2.0
kubernetes==31.0.0
Markdown==3.3.7
markdown-inline-graphviz-extension==1.1.2
mdx-truly-sane-lists==1.3
mergedeep==1.3.4
mkdocs==1.6.1
mkdocs-get-deps==0.2.0
mkdocs-material==9.5.33
mkdocs-material-extensions==1.3.1
mkdocs-monorepo-plugin==1.1.0
mkdocs-redirects==1.2.1
mkdocs-techdocs-core==1.4.2
multidict==6.1.0
oauthlib==3.2.2
ocviapy==1.3.0
packaging==24.2
paginate==0.5.7
parsedatetime==2.6
pathspec==0.12.1
pem==23.1.0
pipenv==2024.4.0
plantuml-markdown==3.10.3
platformdirs==4.3.6
propcache==0.2.1
pyasn1==0.6.1
pyasn1_modules==0.4.1
pymdown-extensions==10.3.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-slugify==8.0.4
PyYAML==6.0.2
pyyaml_env_tag==0.1
requests==2.32.3
requests-oauthlib==2.0.0
requests-toolbelt==1.0.0
rsa==4.9
setuptools==75.8.0
sh==2.1.0
showcert==0.2.3
six==1.17.0
sniffio==1.3.1
tabulate==0.9.0
text-unidecode==1.3
typing_extensions==4.12.2
urllib3==2.3.0
virtualenv==20.28.0
wait-for==1.2.0
watchdog==5.0.3
websocket-client==1.8.0
wheel==0.45.1
yarl==1.18.3

@Sgitario
Copy link
Author

It started happening to me again with 6.0.1...

❯ python --version
Python 3.13.1

❯ bonfire version
bonfire version 6.0.1

❯ python -c 'import ssl; print(ssl.OPENSSL_VERSION)'
OpenSSL 3.2.2 4 Jun 2024

❯ pip freeze
anyio==4.8.0
anytree==2.12.1
app-common-python==0.2.7
backoff==2.2.1
cached-property==2.0.1
cachetools==5.5.0
certifi==2024.12.14
chardet==5.2.0
charset-normalizer==3.4.1
click==8.1.8
crc-bonfire==6.0.1
durationpy==0.9
google-auth==2.37.0
gql==3.5.0
graphql-core==3.2.5
idna==3.10
junitparser==3.2.0
kubernetes==31.0.0
multidict==6.1.0
oauthlib==3.2.2
ocviapy==1.3.0
packaging==24.2
parsedatetime==2.6
propcache==0.2.1
pyasn1==0.6.1
pyasn1_modules==0.4.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
PyYAML==6.0.2
requests==2.32.3
requests-oauthlib==2.0.0
requests-toolbelt==1.0.0
rsa==4.9
sh==2.1.0
six==1.17.0
sniffio==1.3.1
tabulate==0.9.0
urllib3==2.3.0
wait-for==1.2.0
websocket-client==1.8.0
yarl==1.18.3

This is indeed a random issue which makes it much more frustrating :(

@Sgitario
Copy link
Author

By the way, doing:

git config --global http.proxy ""
git config --global --unset http.proxy
git config --global --unset https.proxy

Seems to help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Sgitario @bsquizz