unable to query self signed https API with genericjsonapi in superset, while python test script with request successfully call the https API

[smetis]

Hi,
I was testing genericjsonapi adapter by running query to our intranet https API. The run on superset sqllab always return error below:

shillelagh error: HTTPSConnectionPool(host='somehost.com', port=443): Max retries exceeded with url: /someapp/api/v1.0/common/product/list (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))

I pointed the environment variable REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-bundle.trust.crt. Run a sample python script to query the same https API url and it was a success.

import requests
def make_https_request(url):
    try:
        response = requests.get(url)
        # Check if the request was successful (status code 200)
        if response.status_code == 200:
            print("Request successful!")
            print("Response:")
            print(response.text)  # This will print the response body
        else:
            print(f"Request failed with status code: {response.status_code}")
    except requests.exceptions.RequestException as e:
        print(f"An error occurred: {e}")

# Example usage:
url = "https://somehost.com/someapp/api/v1.0/common/product/list"
make_https_request(url)

Is shillelagh having different requirements on specifying self signed cert? or is there any way to set verify=false (bad practice but acceptable for my intranet use case).

rgds,
Eric

[cwegener]

I can't think of any reason why this shouldn't work in Superset .... unless containers?

[smetis]

@cwegener you mentioned "unless containers", is there anything specific that may have caused this error when running in container?

[cwegener]

I don't have experience with containers, as I don't use a lot of microservices

But what you describe is exactly how I would also set it up in a container by using the environment variable function in the container system.

HOWEVER:

IF your container uses Debian/Ubuntu, then your manual test snippet will ONLY be a valid test, if that test snippets runs in the SAME environment as Superset.

IF you run Debian/Ubuntu

AND IF you run the test snippet using the Operating System's requests package,

BUT THEN you don't run Superset using the Operating System's requests package also,

THEN your test results can be misleading.

BECAUSE: Debian/Ubuntu's requests package is NOT the same as the requests package from PyPI. You would need to identify what Operating System your container image is using and find out which patches are applied to the requests package that comes with the Operating System.

[smetis]

I actually not familiar with both containers and python, guess it is time to get my hand dirty. I will look into your suggestion on digging into container image OS linux flavor. If I have any finding, will post it back here for future reference. thanks!

[smetis]

Ends up it was caused by I setup container env incorrectly. The docker compose of superset has it's own env file which is not the default .env file. When I ran the test script, I specifically set the env by SSH into superset container hence it works on test script but this way the env actually not getting into container environment. I found the right way after inspecting docker-compose file and set the env in the right file, then it works as expected. Entirely my setup mistake, but a good learning. Thanks for your tips!