local smappy looses connection and needs reconnect

[stefferber]

after 60 Min the local smappy object

ls = LocalSmappee(ip='smappee')

looses connection to the local gateway and needs to reconnect:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 133, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 87, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 78, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 516, in url open
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 308, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.4/http/client.py", line 1090, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.4/http/client.py", line 1086, in end headers
    self._send_output(message_body)
  File "/usr/lib/python3.4/http/client.py", line 924, in _send_output
    self.send(msg)
  File "/usr/lib/python3.4/http/client.py", line 859, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 154, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
    (self.host, self.timeout))
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7641a910>, 'Connection to smappee timed out. (connect timeout=5)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 559, in url open
    _pool=self, _stacktrace=stacktrace)
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 265, in increment
    raise MaxRetryError(_pool, url, error)
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='smappee', port=80): Max retries exceeded with url: /gateway/apipublic/instantaneous (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7641a910>, 'Connection to smappee timed out. (connect timeout=5)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "smappee-pi.py", line 90, in <module>
    instantaneous_consumption = ls.load_instantaneous()
  File "/usr/local/lib/python3.4/dist-packages/smappy/smappy.py", line 517, in load_instantaneous
    r = requests.post(url, data=data, headers=self.headers, timeout=5)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 94, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 411, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='smappee', port=80): Max retries exceeded with url: /gateway/apipublic/instantaneous (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7641a910>, 'Connection to smappee timed out. (connect timeout=5)'))

I can think of two approaches:

1. the LocalSmappee objects reconnects automatically, this requires to store the password
2. the LocalSmappee returns a failure and reconnect is handled by the user via ls.logon(password='XXX')

What do you think?

[JrtPec]

I can think of a few way to mitigate this:

• Change from using single requests to using a requests.Session object. This might hold on to data for a longer time, and also store cookies etc. if that is the case.
• Requiring the password upon instantiation and storing it + using a wrapper to handle logging in (if needed) and wrapping every request method. (cfr. the authentication wrapper in the other Smappee object)
• Store the time of last login and repeating the login in when more than 60 minutes have elapsed.

In the snippets posted in #16 I noticed that the javascript code checks the flag authenticated before every request. Could you dig around and see where that flag is set and how the javascript code determines it?

[stefferber]

Good idea to check authenticated flag. I will do this over the weekend.

[stefferber]

Looks like there is no "active" check on timeout. Here is what I have found in smappee monitor.html:
[...]

        $(document).ready(function(){
            plotScale = "24 hours";
            plotMethod=null;
            setAuthenticated(false);
            setAuthenticatedSupport(false);
        });


        function setAuthenticated(stat) {
            if (stat) {
                $("#logonMenu").css("color","#2B708F");
                authenticated=true;
            }
            else {
                $("#logonMenu").css("color","#DF5858");
                authenticated=false;
            }
        }

        function setAuthenticatedSupport(stat) {
            if (stat) {
                $("#logonMenu").css("color","#2B708F");
                authenticatedSupport=true;
            }
            else {
                $("#logonMenu").css("color","#DF5858");
                authenticatedSupport=false;
            }
        }


        function setHeader(title) {
            $("#headerTitle").empty();
            $("#headerTitle").append(title);
        }

        function setHeaderError(title) {
            $("#headerTitle").empty();
            $("#headerTitle").append("<div class=\"alert alert-error\">"+title+"</div>");
        }

        function setHeaderSuccess(title) {
            $("#headerTitle").empty();
            $("#headerTitle").append("<div class=\"alert alert-success\">"+title+"</div>");
        }

[...]

        function clearRefreshInterval() {
            if (refreshIntervalId != null) {
                clearInterval(refreshIntervalId);
                refreshIntervalId=null;
            }
        }

        function logonMenu() {
            clearRefreshInterval();
            console.log("logonMenu()");
            $('#contentArea').empty();
            $('#contentArea').load('logon.html');
        }

        function advancedPublicMenu() {
            if (!authenticated) setHeaderError("Error not authenticated. Use Logon first!");
            else {
                clearRefreshInterval();
                console.log("advancedMenu()");
                $('#contentArea').empty();
                $('#contentArea').load('advancedpublic.html');
            }
        }

[...]

[stefferber]

and this is the complete logon.html file

<!DOCTYPE html>
<!-- saved from url=(0033)http://192.168.178.36//logon.html -->
<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

    <title>Smappee monitor</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">


    <link rel="stylesheet" type="text/css" href="./logon_files/bootstrap.css">
    <link rel="stylesheet" type="text/css" href="./logon_files/emeter.css">

    <script src="./logon_files/jquery-1.11.0.min.js"></script>
    <link rel="stylesheet" type="text/css" href="./logon_files/bootstrap-responsive.css">


    <!-- HTML5 shim, for IE6-8 support of HTML5 elements -->
    <!--[if lt IE 9]>
      <script src="bootstrap/js/html5shiv.js"></script>
    <![endif]-->

    <!-- Fav and touch icons -->
    <link rel="apple-touch-icon-precomposed" sizes="144x144" href="http://192.168.178.36/ico/apple-touch-icon-144-precomposed.png">
    <link rel="apple-touch-icon-precomposed" sizes="114x114" href="http://192.168.178.36/ico/apple-touch-icon-114-precomposed.png">
    <link rel="apple-touch-icon-precomposed" sizes="72x72" href="http://192.168.178.36/ico/apple-touch-icon-72-precomposed.png">
    <link rel="apple-touch-icon-precomposed" href="http://192.168.178.36/ico/apple-touch-icon-57-precomposed.png">
    <link rel="shortcut icon" href="http://192.168.178.36/ico/favicon.png">

    <script>

        var baseURL = "/gateway/apipublic";

        $(document).ready(function(){
            setHeader("Logon to the monitor...");
        });


        function logoff() {
                    var url = baseURL + "/logoff";
            $.ajax(
                    {
                         async : true,
                         type:'POST',
                         contentType: 'application/json',
                         url: url,
                         data:$("#logonPassword").val(),
                         dataType: 'son',

                         success: function(data) {  
                            console.log("logoff success");
                            setHeaderSuccess(data.header);
                            setAuthenticated(false);
                            setAuthenticatedSupport(false);
                            $("#configMenu").prop('disabled', true);
                            $("#statisticsMenu").prop('disabled', true);
                            $("#advancedMenu").prop('disabled', true);
                            $("#report").prop('disabled', true);
                            $("#commandcontrol").prop('disabled', true);
                            $("#waveformdisplay").prop('disabled', true);

                            $("#logbrowser").hide();
                            $("#logbrowser").prop('disabled', true);
                         },

                         error: function(data) {   
                             console.log("logon error");
                             alert("Error, "+data.response);
                         }
                     }
                 ); 

        }

        function logon() {

            var url = baseURL + "/logon";
            $.ajax(
                    {
                         async : true,
                         type:'POST',
                         contentType: 'application/json',
                         url: url,
                         data: $("#logonPassword").val(),
                         dataType: 'son',

                         success: function(data) {  
                            console.log("logon success");

                            if (data.error) {
                                setAuthenticated(false);
                                setAuthenticatedSupport(false);
                                setHeaderError(data.error);
                                $("#configMenu").prop('disabled', true);
                                $("#statisticsMenu").prop('disabled', true);
                                $("#advancedMenu").prop('disabled', true);
                                $("#report").prop('disabled', true);
                                $("#commandcontrol").prop('disabled', true);
                                $("#waveformdisplay").prop('disabled', true);

                                $("#logbrowser").hide();
                                $("#logbrowser").prop('disabled', true);
                            }
                            else {
                                setAuthenticated(true);
                                setHeaderSuccess(data.header);
                                $("#configMenu").prop('disabled', false);
                                $("#statisticsMenu").prop('disabled', false);
                                $("#advancedMenu").prop('disabled', false);
                                $("#report").prop('disabled', false);
                                $("#commandcontrol").prop('disabled', false);
                                $("#waveformdisplay").prop('disabled', false);

                                if (data.authenticatedSupport) {
                                    setAuthenticatedSupport(true);
                                    $("#logbrowser").show();
                                    $("#logbrowser").prop('disabled', false);
                                }
                                else {
                                    setAuthenticatedSupport(false);
                                    $("#logbrowser").hide();
                                    $("#logbrowser").prop('disabled', true);
                                }

                            }

                         },

                         error: function(data) {   
                             console.log("logon error");
                             alert("Error, "+data.response);
                         }
                     }
                 ); 
        }

    </script>

  </head>

    <body>

        <div id="logonDiv" class="myDiv">
        <fieldset>
        <span class="help-block">Logon</span>
        <div class="input-append">
          <input class="input-xlarge" id="logonPassword" type="password">
          <button class="btn btn-warning" type="button" id="logon" onclick="logon()">logon</button>
          <button class="btn btn-warning" type="button" id="logoff" onclick="logoff()">logoff</button>
        </div>
        </fieldset>

        </div>



</body></html>

[stefferber]

while configpublic.html sets the initial value for

            refreshIntervalId = setInterval(refreshInstantaneous, 5000); 

[JrtPec]

@stefferber I have implemented a session object that will be reused for all requests in a LocalSmappee instance. Maybe that does the trick. If not, I will implement an automatic re-logon after 60 minutes. Can you check?

[stefferber]

yes, will do. How do I get the new code? Just by updating smappy?

[stefferber]

OK. Looks very good up to now:
Installation of new version

$ sudo pip3 install --upgrade smappy
Password:
The directory '/Users/adminloc/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/adminloc/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting smappy
  Downloading smappy-0.2.10-py3-none-any.whl
Requirement already up-to-date: requests in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (from smappy)
Installing collected packages: smappy
  Found existing installation: smappy 0.2.7
    Uninstalling smappy-0.2.7:
      Successfully uninstalled smappy-0.2.7
Successfully installed smappy-0.2.10
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

and a first test 20h test on my MAC:

$ python3
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 26 2016, 10:47:25) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from smappy.smappy import Smappee, SimpleSmappee
>>> from smappy import LocalSmappee
>>> ls = LocalSmappee(ip='smappee')
>>> ls.logon(password='XXXX')
{'success': 'Logon successful!', 'header': 'Logon to the monitor portal successful...'}
>>> ls.active_power()
274.939
>>> from datetime import datetime, time delta
>>> from calendar import timegm
>>> datetime.now()
datetime.datetime(2016, 11, 12, 0, 39, 49, 637289)
>>> ls.active_power()
253.864
>>> datetime.now()
datetime.datetime(2016, 11, 12, 7, 40, 45, 101334)
>>> ls.active_power()
213.627
>>> datetime.now()
datetime.datetime(2016, 11, 12, 10, 42, 52, 757473)
>>> ls.active_power()
213.069
>>> datetime.now()
datetime.datetime(2016, 11, 12, 21, 7, 59, 352137)
>>> ls.active_power()
350.063

Now I will deploy on my Raspberry PI3 for permanent testing.

[stefferber]

on my raspberry pi 3 I get a time out error

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 393, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 389, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.4/http/client.py", line 1172, in get response
    response.begin()
  File "/usr/lib/python3.4/http/client.py", line 351, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.4/http/client.py", line 313, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.4/socket.py", line 371, in read into
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 640, in url open
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/util/retry.py", line 261, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 595, in url open
    chunked=chunked)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 395, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 315, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='smappee', port=80): Read timed out. (read timeout=5)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "smappee-pi.py", line 92, in <module>
    instantaneous_consumption = ls.active_power()
  File "/usr/local/lib/python3.4/dist-packages/smappy/smappy.py", line 551, in active_power
    inst = self.load_instantaneous()
  File "/usr/local/lib/python3.4/dist-packages/smappy/smappy.py", line 539, in load_instantaneous
    r = self._basic_post(url='instantaneous', data="loadInstantaneous")
  File "/usr/local/lib/python3.4/dist-packages/smappy/smappy.py", line 499, in _basic_post
    r = self.session.post(_url, data=data, headers=self.headers, timeout=5)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 522, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/adapters.py", line 499, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='smappee', port=80): Read timed out. (read timeout=5)

I will look a bit deeper into this...
Maybe it is similar to this python script stops

[JrtPec]

I have set all requests with a default timeout of 5 seconds, otherwise you risk that your pi gets stuck in an infinite loop if it can't establish a connection.

I could increase the timeout, but I suspect that in this case your raspberry pi just wasn't connected right.
HTTPConnectionPool(host='smappee', port=80)
host should be an ip-address... did you redact this information for this issue post or did you really pass it the string 'smappee' ?

[Trouble-51]

Smappee connection issue

I have no problem pinging the device both from with hassocks.io and IOS. However this error keeps coming up:

2019-12-19 22:02:47 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 408, in _async_add_entity
await entity.async_update_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 286, in async_update_ha_state
self._async_write_ha_state()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 320, in _async_write_ha_state
state = self.state
File "/usr/src/homeassistant/homeassistant/components/climate/init.py", line 165, in state
return self.hvac_mode
File "/usr/src/homeassistant/homeassistant/components/iaqualink/climate.py", line 69, in hvac_mode
state = AqualinkState(self.heater.state)
File "/usr/src/homeassistant/homeassistant/components/iaqualink/climate.py", line 132, in heater
return self.dev.system.devices[heater]
KeyError: 'swc_heater'
2019-12-19 22:02:48 ERROR (SyncWorker_18) [homeassistant.components.smappee] Local Smappee device authentication failed (HTTPConnectionPool(host='198.168.1.62', port=80): Max retries exceeded with url: /gateway/apipublic/logon (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fc8e83b9150>, 'Connection to 198.168.1.62 timed out. (connect timeout=5)')))
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 164, in _new_conn
% (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fc8e83b9150>, 'Connection to 198.168.1.62 timed out. (connect timeout=5)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='198.168.1.62', port=80): Max retries exceeded with url: /gateway/apipublic/logon (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fc8e83b9150>, 'Connection to 198.168.1.62 timed out. (connect timeout=5)'))