Add support for Shelly Pro3EM using Modbus TCP

[Jalle19]

The https://github.com/victronenergy/dbus-shelly implementation has several shortcomings, among others:

• it only updates ever so often (whenever the device itself broadcasts a NotifyStatus to all connected clients)
• it's flaky, sometimes device information is missing (Shelly Pro 3EM product ID, firmware version etc. missing dbus-shelly#9)
• all phases are for some reason not shown in the remote console or VRM (Shelly PRO 3EM values not showing up in "Pages" View and VRM dbus-shelly#4)

I realized that there is already great boilerplate and plumbing available for Modbus devices, so why not use that since the new Shelly Pro3EM supports Modbus TCP.

Compared to dbus-shelly, this:

• updates every second (this is as fast as the Shelly internally updates itself, reading register more frequently won't produce more granular readings)
• device information is registered correctly every time
• device shows up in VRM
• readings for all phases are shown on the main page

Some caveats:

• Shelly Pro3EM uses input registers for everything, not holding registers
• Only the triphase profile is supported
• Firmware version, device name, generic information, none of that is available via Modbus (at least for now)
• Energy readings (forward, reverse) are not available from the device itself
• Device probing is very rudimentary and may need further improvements
• Device name gets BEU added to the end for some unknown reason. Might be that we need a custom method for reading strings from these devices. This is apparently correct, see e.g. https://device.report/allterco-robotics/spem-003cebeu
• Error codes/alarms not yet supported

Would be great to get some feedback on this!

Screenshots:

[Jalle19]

Implemented energy readings

[Jalle19]

Doesn't seem to make any sense to implement /ErrorCode, it's hard to test as well since you need to introduce a serious fault on purpose

[Jalle19]

@mansr anything I can do to get this merged?

[Jalle19]

pinging @mpvader as well

[xsven80x]

Hi Jalle19,

I would like to try your script, unfortunately I have no idea how to install the script.

Do you perhaps have a few pointers for me here? What do I have to do to install your script on my Venus OS? Are configurations necessary?

Or did I just miss these tips?

Thank you in advance!!

greeting
Sven

[amadirnjul]

This would be great to test, but please add som details about how to implement this

[Jalle19]

I'll have to rebase this branch onto the latest changes, I can write a short guide after that

[Jalle19]

@xsven80x @amadirnjul here are instructions on how to test this:

Updated 2024-08-27

1. Make sure you're on VenusOS 3.41 (the newest version as of this writing)
2. SSH to your VenusOS device and run the follow commands:

mount -o remount,rw /
cd /opt/victronenergy/dbus-modbus-client
cp device.py device.py.bak
cp dbus-modbus-client.py dbus-modbus-client.py.bak
cd /tmp
wget https://codeload.github.com/Jalle19/dbus-modbus-client/zip/refs/heads/shelly-pro3em -O shelly-pro3em.zip
unzip shelly-pro3em.zip
cd dbus-modbus-client-shelly-pro3em
cp shelly.py device.py dbus-modbus-client.py /opt/victronenergy/dbus-modbus-client

3. Reboot your device
4. Go to Settings -> Modbus TCP/UDP devices -> Saved devices -> Add, enter the IP address of your Shelly Pro3EM.
5. That's it, the device should now be visible in the device list and working as expected

[amadirnjul]

That’s awesome. Can this also be used for the Shelly Plus 1PM Mini, as well?

[Jalle19]

@amadirnjul no, since it doesn't support Modbus

[amadirnjul]

I have two Shelly's installed in my sailboat. I thought it was the Plus 1PM mini and the Pro 3EM, but the last one is the Pro 1PM. This does also not support Modbus as far as I know. I better make the electrician to swap it out or I also might dig into the Pro 1 PM support for the Venus. Anyway, thanks alot

[Jalle19]

It would be fairly trivial to make a Modbus TCP proxy for various Shelly's that would expose them as a supported Shelly energy meter (currently only the Pro3EM since it's the only device with Modbus support).

Would be nice if someone who maintains this repository could comment here on the status of this pull request, @mansr ?

[Jalle19]

Rebased on top of the latest master (used by VenusOS 3.30).

I'm really disappointed with Victron Energy for not responding at all to this pull request. @mansr keeps releasing new versions while completely ignoring all pull requests here. Rebasing after every release is painstaking and error-prone, especially when the lower levels of the project are in constant motion.

I would really appreciate some feedback on this change.

[Jalle19]

Connection seems wonkier than with VenusOS 3.14, not really sure what's going on.
Turns out the issue was that I had two instances polling the same meter - the meter can barely keep up with a single consumer

[mpvader]

Hi, Matthijs here from Victron. The decision to includes this or not lies with me. I don’t have time to look into this the coming 2 weeks but will look into it thereafter.

One concern I have is that by making things work out of the box, customers also expect it to work out of the box and turn to us if it doesn’t.

and while shelly support is nice, we already support lots of energy meters and more and more types doesn’t bring extra functionality for users. However I also see the advantages..

Talk to you later! Matthijs

[Jalle19]

@mpvader thanks for getting back to me!

I appreciate your concerns about maintainability, however, most of the implementation is the same regardless of energy meter - it's just the particular registers that differ. Up until quite recently there was no genuine Victron-branded energy meter on the market, so to me it seemed logical to add support for new third-party meters.

Now that there's an official (and most likely superior, technically) product these third-party meters could perhaps me supported in the sense that the software supports them, but there is no guarantee that it will work.

Either way, Shelly products are quite popular and I believe this addition would greatly benefit the community.

Looking forward to further feedback from you.

[Jalle19]

Been using this in a three-phase ESS setup for over a week now without any issues. @mpvader have you had time to consider this PR?

[Marv2190]

Would be nice to have this added.

[Kudos50]

Working fine for me for 3-phase ESS since early april for 1x grid and 2x solar inverter. Thanks for that. Would be great if Martijn can add support.

[Jalle19]

It seems like @mansr has incorporated some of the changes introduced here (support for input registers), so there's some hope that the intent is to merge this at some point 🙏

[Fahrzeug-cpu]

@xsven80x @amadirnjul here are instructions on how to test this:

Updated 2024-04-05

1. Make sure you're on VenusOS 3.30 (the newest version as of this writing)

2. SSH to your VenusOS device and run the follow commands:

mount -o remount,rw /
cd /opt/victronenergy/dbus-modbus-client
cp device.py device.py.bak
cp dbus-modbus-client.py dbus-modbus-client.py.bak
cd /tmp
wget https://codeload.github.com/Jalle19/dbus-modbus-client/zip/refs/heads/shelly-pro3em -O shelly-pro3em.zip
unzip shelly-pro3em.zip
cd dbus-modbus-client-shelly-pro3em
cp shelly.py device.py dbus-modbus-client.py /opt/victronenergy/dbus-modbus-client

3. Reboot your device

4. Go to Settings -> Modbus TCP/UDP devices -> Saved devices -> Add, enter the IP address of your Shelly Pro3EM.

5. That's it, the device should now be visible in the device list and working as expected

How can i uninstall this settings ? Please explain

[Jalle19]

I don't think you can remove saved Modbus TCP devices unless they become unreachable/unconnectable. When they are unreachable you're offered to delete it in the main menu.

[Fahrzeug-cpu]

Can you explane me With Files i Must dealte ? Greetings Luca Am 08.07.2024 um 18:15 schrieb Sam Stenvall ***@***.***>: I don't think you can remove saved Modbus TCP devices unless they become unreachable/unconnectable. When they are unreachable you're offered to delete it in the main menu. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[Jalle19]

This is not a support forum I'm afraid. You'll need to replace the files affectes with the .bak versions you created if you followed the instructions. You can also update VenusOS, that will remove any modifications.

[Kudos50]

Think I found something. I am using the beta's for EVCS (2x) on Venus combined with your modifications. It seems my 2 chargers are no longer detected normally. Only one shows up in an auto-scan Discovered devices whereas the other is now listed under saved devices next to my 3 Shelly's.
Removing the saved device and executing a new scan does not change a thing. They both work. So good for now.

[Jalle19]

That's what happens when you apply older changes to a newer version of VenusOS. I haven't updated this code since version 3.30.

[crisi-solar]

This support worked perfectly on Raspy 4 until Venus firmware 3.40, but unfortunately not anymore since the new 3.41.
So back to 3.40.

or do I have to re-run the installation script after every Venus update?

[Jalle19]

Yes, you need to re-apply the patches after every update. I haven't tested this with VenusOS 3.40 or newer yet so I can't guarantee that everything still works.

[Jalle19]

The latest version of Venus OS (3.41) seems to still use v1.51 of this particular component, so my changes should still apply cleanly. I'll update to 3.41 myself today at some point.

[Jalle19]

Verified that the changes and instructions in #4 (comment) still work against v3.41.

[Jalle19]

@mpvader any updates on the plan for merging this?

[rkapsi]

To whom it may concern I've just created a PR that adds support for Venus OS 3.50+ (current beta).

#7

[FloMeyer]

I get the following error:

With Venus OS v3.42.

@4000000066ec20ca1319fde4 *** starting dbus-modbus-client ***
@4000000066ec20cb24c7f654 Traceback (most recent call last):
@4000000066ec20cb24c9c72c File "/opt/victronenergy/dbus-modbus-client/dbus-modbus-client.py", line 25, in
@4000000066ec20cb24d3f4a4 import carlo_gavazzi
@4000000066ec20cb24d4bbdc File "/opt/victronenergy/dbus-modbus-client/carlo_gavazzi.py", line 123, in
@4000000066ec20cb24e10464 probe.add_handler(probe.ModelRegister(Reg_u16(0x000b), models,
@4000000066ec20cb24e1cf84 File "/opt/victronenergy/dbus-modbus-client/probe.py", line 105, in init
@4000000066ec20cb24ec993c self.access = {m['handler'].default_access for m in models.values()}
@4000000066ec20cb24ed58a4 File "/opt/victronenergy/dbus-modbus-client/probe.py", line 105, in
@4000000066ec20cb24f8031c self.access = {m['handler'].default_access for m in models.values()}
@4000000066ec20cb24f908d4 AttributeError: type object 'EM24_Meter' has no attribute 'default_access'
@4000000066ec20cb2dd04f44 *** starting dbus-modbus-client ***