Debugging missing data in ems-esp with help of km200 with use of API and Syslog

[tp1de]

Since I am using ems-esp and km200 in parallel, I ask myself what would be the most efficient / elegant way to debug missing fields in ems-esp while triggering changes on km200.

My idea would be: Within a javascript code writing changes to km200 and analyzing the respective telegrams by "filtering the right ones" out of syslog.

What do you think or what would be the easiest way to identify the right telegram types and content.

[proddy]

I would agree using syslog is the easiest way. With debug on and MQTT off (because there's a lot of messages). Then a script in python, JS or shell (using curl) to invoke the commands in batches every n seconds. What we could do is add a new API command that sets a marker in the syslog so you can see which commands are called when. Would that help?

It'll be good to get EMS-ESP closer to the KM200 and complete the missing library.

[tp1de]

What we could do is add a new API command that sets a marker in the syslog so you can see which commands are called when

Would it be possible to enable/disable syslog by API command?
I would then integrate the syslog server into javascript and then

1. enable syslog by API
2. change respective field by API via km200 Rest API (km200 gateway then sends ems message to device boiler/thermostat)
3. disable syslog by API
4. analyze syslog data -Which filters on syslog entries should be used? Do you have a proposal?

It'll be good to get EMS-ESP closer to the KM200 and complete the missing library.

I am ready to support.

[proddy]

yes that would work. I'll create a new API endpoint called 'system/syslog_level' which takes a json body where 'value' has the syslog level. So OFF is off and DEBUG is on in this case. Do you need some help with the scripting?

[tp1de]

Do you need some help with the scripting?

Thanks, not for the moment since I have all code already.
when switching from off to debug do you expect any time delay?

[tp1de]

@proddy just a question related to syslog messages set as hexadecimal bytes:

Is crc send as well as last byte? If yes: How is crc byte calculated?

First testings and resulting telegrams from gateway (48) to boiler (08). Value is send as hex in byte 5:

dhwCircuits.dhw1.temperatureLevels.high value:50 ---> 48 08 EA 06 32 19
dhwCircuits.dhw1.temperatureLevels.high value:52 ---> 48 08 EA 06 34 1F

dhwCircuits.dhw1.temperatureLevels.low value:45 ---> 48 08 EA 12 2D 2E
dhwCircuits.dhw1.temperatureLevels.low value:43 ---> 48 08 EA 12 2B 28

[proddy]

Is crc send as well as last byte? If yes: How is crc byte calculated?

If the option "Output EMS telegrams as hexadecimal bytes" is enabled then the last byte will be the CRC. I use a lookup table for speed in the EMS-ESP code (see telegram.cpp line 50).

[proddy]

First testings and resulting telegrams from gateway (48) to boiler (08). Value is send as hex in byte 5:
dhwCircuits.dhw1.temperatureLevels.high value:50 ---> 48 08 EA 06 32 19
dhwCircuits.dhw1.temperatureLevels.high value:52 ---> 48 08 EA 06 34 1F
dhwCircuits.dhw1.temperatureLevels.low value:45 ---> 48 08 EA 12 2D 2E
dhwCircuits.dhw1.temperatureLevels.low value:43 ---> 48 08 EA 12 2B 28

EMS-ESP uses EA (the telegram type ID) with offset 6 to write the wwSelTemp value on EMS+. So matching dhwCircuits.dhw1.temperatureLevels.high. Do we need a new command like wwSelTempLow for the low value at offset 12?

[tp1de]

EMS-ESP uses EA (the telegram type ID) with offset 6 to write the wwSelTemp value on EMS+. So matching dhwCircuits.dhw1.temperatureLevels.high

• YES for writing values - This is working like I mentioned
• NO for actual value set since it is reset to 60°C afterwards - Is one of the broadcasting messages overwriting value with wrong content?

[tp1de]

Do we need a new command like wwSelTempLow for the low value at offset 12?

I recommend to discuss this in a seperate thread later. I still struggle with your name conventions:
Within RC310 / km200 these are predefined temperature levels and selected temp (settemp) comes out of timeswitch programs and/or holiday modes. So names should be rather wwtemplevel_high and wwtemplevel_low then wwseltemp because selected temperature (wwsettemp) depends on time / holiday.

I will prepare an overview related to RC310 and EMS+ but it might take some time.

[tp1de]

I tried to test within a JS program but it turned out to be unnecessary complex.
I build a node-red workflow to test with filters for telegrams. This works fine and is quite easy.
So I am ready to do some tests related to km200 fields and the respective telegrams send.

I would like to test sending telegrams by myelf rather then just watching telegrams send by the km200 gateway.
So I have to find a JS way to generate the crc.

[proddy]

this code snippet will calculate the CR, if you can re-write in JS:

// Calculate CRC for buffer
uint8_t ems_crc( char * buffer, int len ){
 uint8_t i,crc = 0x0;
 uint8_t d;
 for(i=0;i<len-2;i++){
   d = 0;
   if ( crc & 0x80 ){
     crc ^= 12;
     d = 1;
   }
   crc  = (crc << 1)&0xfe;
   crc |= d;
   crc ^= buffer[i];
 }
 return crc;
}

[proddy]

I recommend to discuss this in a seperate thread later. I still struggle with your name conventions:
Within RC310 / km200 these are predefined temperature levels and selected temp (settemp) comes out of timeswitch programs and/or holiday modes. So names should be rather wwtemplevel_high and wwtemplevel_low then wwseltemp because selected temperature (wwsettemp) depends on time / holiday.

yes this needs looking at with Michael when he's back. Also why the value is been re-written each time.

Do you still need an API call to set the SysLog log level? Or add a marker?

[tp1de]

Do you still need an API call to set the SysLog log level? Or add a marker?

I think midterm it makes sense to be able to set SysLog level by API (marker does not make a lot of sense), but wait until we have discussed the other topics.

In the meatime I use node-red with the syslog-node in node-red and I am filtering the messages in node-red (e.g. from gateway to boiler) and then analyzing just the filtered messages. Whenever I change a km200 field value within my ioBroker adapter just one single telegram is send to boiler / thermostat. So it is easy to analyze within node-red for the moment.

Also why the value is been re-written each time.

Do you know in which broadcasting telegrams the wwseltemp is included?

[proddy]

wwseltemp is from telegram 0xE9 (offset 10 of the data block). There is also wwsettemp in 0xE9 at offset 0. It's confusing what the difference is.

[tp1de]

wwseltemp is from telegram 0xE9 (offset 10 of the data block).

Offset 10 of E9 is always 3C (60°C) and never change when I change the templevel_high. Just Offset 0 is changed but I guess that this would be rather actual settemp than temp_level_high. Since I do not find any value representing temp_level low in E9 I believe that E9 does not contain the data for temperature levels.

[tp1de]

@proddy @MichaelDvP
I think node-red syslog should work. Can this be re-established? It is ideal for fast prototyping.

In the meantime I installed v3.2.2b1 without node red and reading the syslog within javascript. But now I am confused because syslog content/format changed from previous versions.
Could you please explain the changes and content as per now?

[proddy]

I don't know or use node so can't help you there I'm afraid. It may be easier to capture some syslog on 3.2.1 and then compare it to 3.2.2b1. Without seeing real examples of what is not working its difficult to assist.

[tp1de]

Last changes of @MichaelDvP made syslog working again with node-red. Thanks a lot. 🥇

BTW: Are there already API commands for syslog available?

[proddy]

BTW: Are there already API commands for syslog available?

yes.

[tp1de]

Thanks but these are console commands.
What whould be the command to be send by rest-api to go to syslog-level "All" and switch on hex output?
...and what be the rest-api commands to switch syslog on / off?

[proddy]

@MichaelDvP looks like we need to start updating the wiki. I hate writing doc but i think its needed

[MichaelDvP]

In principal: Yes.
But in this case it does not help, first paragraph of api-section says that commands can be called from console, mqtt and api. Also a simple check of emsesp.local/api/system/commands would have helped.

[proddy]

I meant adding to the table here https://emsesp.github.io/docs/#/APIv3?id=system-endpoints with an explanation of the payload

[tp1de]

@proddy @MichaelDvP

I created a setup with syslog-server in a javascript with filter functions and then used my ioBroker adapter to change all writeable fields in km200 and ems-esp and discover the telegrams send on the ems-bus by analyzing the telegrams on syslog afterwards.

Let me share the result:
telegrams.xlsx
empty means no telegram send or htpp write error (even if marked as writeable)

Does this help?

[MichaelDvP]

Thanks,
i have expected more from km200, but a few values we can add to emsesp:
/heatingCircuits/hcx/fastHeatupFactor, as single byte uint
/dhwCircuits/dhw1/charge as bool
I think both writeable?

for the /hmx/hcMode, hmx/dhwMode and /heatingCircuits/hc2/switchProgramMode it is unclear why value and data differs.
and the switchPrograms has much values, but small data, that does not fit (or i do not understand it).
Only for hmx/startStop data and value match and can be read and set. Is it usefull without modes?

[tp1de]

I think both writeable?

Yes since I can test only fields marked writable - so all km200 fields are marked writeable by rest-api.
The switchprogram values are encoded in a special way I think. I will do some tests within the next days and report later.

[MichaelDvP]

The switchtime-coding is 2 bytes per switchtime, first byte upper 3 bits: day of week, lower 3 bits: mode, second byte time in steps of 15 min. SwitchProgramA, Data 03 14 01 58 FF FF FF FF FF FF FF FF is first: day 0, mode 3, 300 min, second: day 0, mode 1, 1320 min, all other times disabled. (btw. ems RC35 uses 10 min per step)

Why are the value/data for modes like /system/holidayModes/hm2/hcMode data 4, value 3 different? And what is the meaning of mode 3 or 4? What values are possible?

[tp1de]

The holiday modes hm1 .. hm5 are defined within rest-api:

1. assignedTo:

      "id": "/system/holidayModes/hm2/assignedTo",
      "type": "arrayData",
      "writeable": 1,
      "recordable": 0,
      "values": [],
      "allowedValues": [
        "hc1",
        "hc2",
        "hc3",
        "hc4",
        "dhw1",
        "dhw2"
      ],

2. dhwMode:

      "id": "/system/holidayModes/hm2/dhwMode",
      "type": "stringValue",
      "writeable": 1,
      "recordable": 0,
      "value": "OFF",
      "allowedValues": [
        "OFF",
        "TD_OFF"
      ],

3.hcMode:

      "id": "/system/holidayModes/hm2/hcMode",
      "type": "stringValue",
      "writeable": 1,
      "recordable": 0,
      "value": "ECO",
      "allowedValues": [
        "AUTO_SAT",
        "FIX_TEMP",
        "OFF",
        "ECO"
      ],

AUTO_SAT is defined as same as Saturday .... whatever this is for during holidays

4. startStop:

      "id": "/system/holidayModes/hm2/startStop",
      "type": "stringValue",
      "writeable": 1,
      "recordable": 0,
      "value": "2009-01-01/2009-01-01",
      "valIs": "value"

[tp1de]

@MichaelDvP @proddy
I recognized that for the datapoints /system/holidayModes/hm1...4/assignedTo the km200 sends 2 telegrams:
First one for hc1-hc4 and second one for dhw1 and dhw2.
I adjusted my analysis Excel table and added a colum for allowedValues / enum

telegrams.xlsx

[proddy]

closing.