ConnectionError(HTTPConnectionPool(host='vpc-*************', port=80)

[benfrench1]

To whom this concerns,

I am encountering issues when trying to deploy this as a helm chart to my cluster. The logs on the elastalert pod are:

Traceback (most recent call last):
  File "/usr/local/bin/elastalert-create-index", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/elastalert/create_index.py", line 264, in main
    create_index_mappings(es_client=es, ea_index=index, recreate=args.recreate, old_ea_index=old_index)
  File "/usr/local/lib/python3.6/site-packages/elastalert/create_index.py", line 23, in create_index_mappings
    esversion = es_client.info()["version"]["number"]
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 259, in info
    return self.transport.perform_request("GET", "/", params=params)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 86, in perform_request
    raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPConnectionPool(host='vpc-*************', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31d87fe080>: Failed to establish a new connection: [Errno -3] Try again',))) caused by: ConnectionError(HTTPConnectionPool(host='vpc-*************', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f31d87fe080>: Failed to establish a new connection: [Errno -3] Try again',)))

When deploying I am setting the elastalert host and port, otherwise all values are default.

Any help and guidance on this issue would be much appreciated please 😄 !

Many thanks,

Ben

[jertel]

There's not much to go on from the log, other than obviously it can't connect to ES. I am curious though why you chose port 80. Typically ES is deployed with its REST API available on port 9200, and since port 80 is normally used for user-facing web pages that was a surprise to see. Certainly it will work if your ES cluster is really listening on port 80, so I'm not suggesting it's wrong, but rather atypical.

Also, if your ES cluster is running in the same Kubernetes cluster I would normally have the ES hostname set to something like elasticsearch.svc.cluster.local, assuming the ES chart was setup to use elasticsearch as the service name. If you are running Elasticsearch outside of the K8S cluster then ignore this comment.

[benfrench1]

Hi jertel,

Thank you for your swift response! I am migrating our elastalert from the stable location. This was previously configured with port 80 and was performing as expected. I did try using port 9200 but I get the same error unfortunately.
I am sure it is something silly and basic I have missed but I cant get my head around it.
Our Elasticsearch is hosted in AWS.

[jertel]

You can use kubectl to exec into the elastalert pod and attempt to connect to the ES host, to help prove or disprove that it's a networking issue between Kubernetes and the AWS ES host. Your command will look something like this, replacing the eshostname with your actual hostname:

kubectl -n elastalert exec -it elastalert-xxxxx wget http://vpc-eshostname:80 -qO -

This assumes:

1. You deployed the chart to the elastalert namespace (hence the -n elastalert above.
2. Your elastalert pod name looks something like elastalert-xxxxx. You can find it by using kubectl -n elastalert get pods

If the elastalert pod is crashing because of the connectivity failure you will have to act quickly to exec into the pod before the retries expire. Alternatively you could use another stable pod in the same K8s cluster to run the network connectivity check, since most likely all pods will have similar firewall/network constraints between your k8s cluster and the ES host.

This type of troubleshooting is really outside the scope of this project so I may not be able to help much further. But I agree, what ever the issue is it most likely is something simple.

[benfrench1]

I performed the following command:

kubectl -n logging exec -it elastalert-56bf4db44c-8zn5f wget http://vpc-******:80

This returned the following:

Connecting to vpc-******:80 (IP:80)
saving to 'index.html'
index.html           100% |*********************************************************************************************************************************************|   456  0:00:00 ETA
'index.html' saved

I then can see the file when exec'ing onto the pod in question. However, if I then perform the exact same command again "kubectl -n logging exec -it elastalert ..... etc" I now get

wget: bad address 'vpc-**********:80'
command terminated with exit code 1

🤔

[jertel]

Perhaps a Kubernetes DNS issue?

[benfrench1]

This helped push me in the right direction! Many thanks @jertel 😄