Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Promtimer should support alternate address (was: promtimer issue working with remote AWS instances) #50

Open
stevewatanabe opened this issue May 2, 2023 · 7 comments
Open

Promtimer should support alternate address (was: promtimer issue working with remote AWS instances) #50

stevewatanabe opened this issue May 2, 2023 · 7 comments

Comments

@stevewatanabe
Copy link
Member

I was encountering an issue when starting promtimer against a remote AWS instance. I put a printf in urllib/request.py in do_open

1347             try:
1348                 print(f"{req.get_method()} {req.selector} {req.data} {headers}")
1349                 h.request(req.get_method(), req.selector, req.data, headers,
1350                           encode_chunked=req.has_header('Transfer-encoding'))
1351             except OSError as err: # timeout error
1352                 raise URLError(err)
1353             r = h.getresponse()

and see this crash. Note the /pools/default/buckets is being done against the localhost (127.0.0.1) which is my mac. This is a single node, just created cluster so maybe that's a reason

steve.watanabe @ ~/promtimer/bin (master) $ ./promtimer --cluster ec2-52-23-193-116.compute-1.amazonaws.com:8091 --user Administrator --password asdasd
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Connection': 'close'}
GET /pools/default/buckets None {'Host': '127.0.0.1:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1349, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 941, in connect
    self.sock = self._create_connection(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/steve.watanabe/promtimer/bin/../promtimer/promtimer.py", line 336, in <module>
    main()
  File "/Users/steve.watanabe/promtimer/bin/../promtimer/promtimer.py", line 302, in main
    buckets = stats_sources[0].get_buckets()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/steve.watanabe/promtimer/promtimer/cbstats.py", line 412, in get_buckets
    response = util.execute_request('{}:{}'.format(self.host(), self.port()),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/steve.watanabe/promtimer/promtimer/util.py", line 139, in execute_request
    response = opener.open(request)
               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 519, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1378, in http_open
    return self.do_open(http.client.HTTPConnection, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1352, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 61] Connection refused>

I then add a second node to the cluster and try again. Note it uses a non-local IP address (172.31.15.151) but not the external one and so the connection times out.

steve.watanabe @ ~/promtimer/bin (master) $ ./promtimer --cluster ec2-52-23-193-116.compute-1.amazonaws.com:8091 --user Administrator --password asdasd
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
GET /pools/default/nodeServices None {'Host': 'ec2-52-23-193-116.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Connection': 'close'}
GET /pools/default/buckets None {'Host': '172.31.15.151:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1349, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 941, in connect
    self.sock = self._create_connection(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/usr/local/Cellar/[email protected]/3.11.2_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

I skimmed the promtimer code and see it is using /pools/default/nodeServices

Running on my node

      "thisNode": true,
      "hostname": "172.31.15.151"

      "hostname": "ec2-54-88-70-0.compute-1.amazonaws.com"

and

      "hostname": "172.31.15.151"

      "thisNode": true,
      "hostname": "ec2-54-88-70-0.compute-1.amazonaws.com"

Having entered this issue perhaps there's two issues: 1) remote promtimer doesn't work against a node that doesn't have a "hostname" in the results from /pools/default/nodeServices and 2) ns_server may not be returning the right hostname...though I'm not sure how ns_server would know the IP isn't the external IP.
@stevewatanabe
Copy link
Member Author

Thinking that it might work against the second node it too used the non-external hostname and thus timed out.

steve.watanabe @ ~/promtimer/bin (master) $ ./promtimer --cluster ec2-54-88-70-0.compute-1.amazonaws.com:8091 --user Administrator --password asdasd
GET /pools/default/nodeServices None {'Host': 'ec2-54-88-70-0.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}
GET /pools/default/nodeServices None {'Host': 'ec2-54-88-70-0.compute-1.amazonaws.com:8091', 'User-Agent': 'Python-urllib/3.11', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=', 'Connection': 'close'}
GET /pools/default/buckets None {'Host': '172.31.15.151:8091', 'User-Agent': 'Python-urllib/3.11', 'Connection': 'close'}

@dave-finlay
Copy link
Contributor

Hey Steve

pools/default/nodeServices only returns a hostname when the node has a hostname which is something other than cb.local(which is a logical name for either 127.0.0.1 or ::1 depending on the address family. The reasons for not including the hostname as 127.0.0.1 should probably be clear.)

When you bring up a standalone node and don't configure it with a hostname explicitly, it will pick up the name cb.local which is why you're not getting a hostname with 1 node, I guess. We can improve this behavior in Promtimer to use the cluster address instead of 127.0.0.1. This would be a small patch.

When you add a second node, you did so using the internal addresses - and so these are returned in the pools/default/nodeServices response. If you want this use case to work, you should use the external addresses when setting up the cluster and adding the nodes. Or you should use "alternate address" and we'll need to add support in Promtimer to use that. (This would be an improvement).

@stevewatanabe
Copy link
Member Author

Hi Dave,
I started from scratch and....
For the first node I installed bits and went to the UI http://ec2-52-23-193-116.compute-1.amazonaws.com:8091/ and "setup the cluster" with defaults. I then added "ec2-54-88-70-0.compute-1.amazonaws.com" as the second node and did a rebalance.

Both nodes show the Servers as
image

I'm not sure how to setup the first node and tell it to use ec2-52-23-193-116.compute-1.amazonaws.com as the hostname.

@dave-finlay
Copy link
Contributor

dave-finlay commented May 2, 2023 •
edited
Loading

I'm not sure how to setup the first node and tell it to use ec2-52-23-193-116.compute-1.amazonaws.com as the hostname.

Enter it in the hostname box here:
image

or of course you can use the CLI.

@stevewatanabe
Copy link
Member Author

That did the trick! I am able to start promtimer against either of the two nodes. Thanks for the help Dave!

@dave-finlay
Copy link
Contributor

Hey Steve - I've created a PR that will allow Promtimer to connect to a remote single node instance running on localhost: #51. Would you be able to give it a try? It doesn't solve the problem of having Promtimer be able to connect via alternate address - but it should work for single nodes, which can be convenient.

@stevewatanabe
Copy link
Member Author

Tested on AWS linux instance. Thanks for the quick fix.

@dave-finlay dave-finlay changed the title promtimer issue working with remote AWS instances Promtimer should support alternate address (was: promtimer issue working with remote AWS instances) May 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dave-finlay @stevewatanabe