Testing of new firmware with new partition scheme and better upgrade code

[gskjold]

Got a new firmware ready for testing. This one does not have much new features, but is a big deal still!

To regain some much needed available space for firmware, we have decided to change the partitioning of esp32 based chips. This is easy for new installations, as it is defined by the partitions.bin file. But to be able to preserve data on existing installations and also remove the need to do a full flash, we have integrated a re-partitioning feature in the code.

NOTE that when installing this firmware, your device may reboot several times to complete the re-partitioning.

After the flash space have been re-partitioned, there will be much less available space on the file system. We have previously used this for temporary storage when uploading a new firmware through the GUI. Since this is no longer possible with the smaller file system, it will now write directly to the new flash space and activate the new firmware when it is done.

And finally, a much requested feature for our Pow-U device: You can now OTA upgrade without using USB. With the new firmware update code will monitor the device voltage and adapt the download from the server accordingly to preserve power and avoid accidental reboots. It will also remember how far into the upgrade it is and recover the upgrade if a reboot occurs.

I have only tested this fully on our ESP32-S2 device. I have also done some successful tests on ESP32, ESP32-S3 and ESP8266. I do not have a proper ESP32-C3 available for testing, so I hope someone can confirm if it works on C3.

How to test this:

• First: Back up your config before testing this firmware!
• In all cases, check that you day and moth graph is populated before and after upgrade

Testing OTA:

• Install the latest version, v2.3.9
• Using the GUI, upload the test firmware from this post (Use the firmware.bin from the zip file, see https://github.com/UtilitechAS/amsreader-firmware/wiki/Flashing-via-web-GUI)
• Confirm that the installed version now is 1453bbb
• After a short time, a notice in the header will notify that a new version v.2.3.9 is available, since this is the latest on the server
• Click this and confirm that you want to "upgrade". Progress will be shown in the header
• When complete, you will be back to v2.3.9. This confirms that the OTA upgrade works

Testing firmware upload:

• Now, download v2.3.9 zip file from https://github.com/UtilitechAS/amsreader-firmware/releases/tag/v2.3.9
• Do a full flash of v2.3.9, including ALL the bin files (partition.bin is important. See https://github.com/UtilitechAS/amsreader-firmware/wiki/flashing)
• After reboot, upload the new firmware (1453bbb) via GUI
• Confirm that you are running the new version
• Go to the status page and upload firmware.bin for v2.3.9 via GUI
• Confirm that the firmware reboots back to v2.3.9. This confirms that firmware upgrade via GUI upload works

Please report back your results. Let us know what type of device or chipset you have tested this on.
Also, if you have a Pow-U or another BUS powered device, please test the OTA upgrade with BUS power only.

esp32s2.zip
esp32s3.zip
esp32solo.zip
esp32.zip
esp8266.zip
esp32c3.zip

[Noschvie]

Hello, above ESP32-C3 fixed the problem discussed in #835?

[gskjold]

After re-partitioning, it should work, yes. Not sure if you can install this and expect it to re-partition on the fly though, you might need to run a full flash with all the bin files to make it work. If you try, remember to back up your config first

[dbeinder]

Tested OTA on my C3 board on MBUS without external power.
Upgrade worked fine and graphs were not lost.

Visual bug: After the device reboots, the page does not refresh and the last percentage and the old FW hash remain, but you can tell from the uptime resetting that a reboot happened:

after refreshing the page, it shows 2.3.9:

[Noschvie]

Thanks David!

[ArnieO]

Thank you for the feedback, @dbeinder !

[toast1234]

OTA update. HomeAssistant MQTT. No external power. No issue before. No issue after.
Meter 1:
Device information
Chip: esp32s2 (160MHz)
Device: [Pow-P1]
Meter
Manufacturer: Aidon
Model: 6534
ID: unknown

Meter 2 (Sagemcom CS50001, ELLEVIO):
Device information
Chip: esp32s2 (160MHz)
Device: [Pow-P1]
Meter
Manufacturer: Unknown
Model: \253833635_A
ID: unknown

Not sure if you are interested in detecting the meter name/model
/ELL5\253833635_A
from
(D) Received valid DSMR at 0 (D) NOT Ready to update (internal clock 19:13:56 UTC, meter clock: 19:13:47, list type 4, est: 0) (V) DSMR frame: (V) 2F 45 4C 4C 35 5C 32 35 33 38 33 33 36 33 35 5F (V) 41 0D 0A 0D 0A 30 2D 30 3A 31 2E 30 2E 30 28 32 (V) 34 31 32 31 32 32 30 31 33 34 38 57 29 0D 0A 31 (V) 2D 30 3A 31 2E 38 2E 30 28 30 30 30 34 37 35 32 (V) 32 2E 32 37 38 2A 6B 57 68 29 0D 0A 31 2D 30 3A (V) 32 2E 38 2E 30 28 30 30 30 30 30 30 30 30 2E 30 (V) 30 31 2A 6B 57 68 29 0D 0A 31 2D 30 3A 33 2E 38 (V) 2E 30 28 30 30 30 30 30 30 35 39 2E 30 33 39 2A (V) 6B 76 61 72 68 29 0D 0A 31 2D 30 3A 34 2E 38 2E (V) 30 28 30 30 30 31 32 35 37 30 2E 39 37 36 2A 6B (V) 76 61 72 68 29 0D 0A 31 2D 30 3A 31 2E 37 2E 30 (V) 28 30 30 30 31 2E 32 33 31 2A 6B 57 29 0D 0A 31 (V) 2D 30 3A 32 2E 37 2E 30 28 30 30 30 30 2E 30 30 (V) 30 2A 6B 57 29 0D 0A 31 2D 30 3A 33 2E 37 2E 30 (V) 28 30 30 30 30 2E 30 30 30 2A 6B 76 61 72 29 0D (V) 0A 31 2D 30 3A 34 2E 37 2E 30 28 30 30 30 30 2E (V) 34 35 31 2A 6B 76 61 72 29 0D 0A 31 2D 30 3A 32 (V) 31 2E 37 2E 30 28 30 30 30 30 2E 33 37 36 2A 6B (V) 57 29 0D 0A 31 2D 30 3A 34 31 2E 37 2E 30 28 30 (V) 30 30 30 2E 38 32 34 2A 6B 57 29 0D 0A 31 2D 30 (V) 3A 36 31 2E 37 2E 30 28 30 30 30 30 2E 30 33 30 (V) 2A 6B 57 29 0D 0A 31 2D 30 3A 32 32 2E 37 2E 30 (V) 28 30 30 30 30 2E 30 30 30 2A 6B 57 29 0D 0A 31 (V) 2D 30 3A 34 32 2E 37 2E 30 28 30 30 30 30 2E 30 (V) 30 30 2A 6B 57 29 0D 0A 31 2D 30 3A 36 32 2E 37 (V) 2E 30 28 30 30 30 30 2E 30 30 30 2A 6B 57 29 0D (V) 0A 31 2D 30 3A 32 33 2E 37 2E 30 28 30 30 30 30 (V) 2E 30 30 30 2A 6B 76 61 72 29 0D 0A 31 2D 30 3A (V) 34 33 2E 37 2E 30 28 30 30 30 30 2E 30 30 30 2A (V) 6B 76 61 72 29 0D 0A 31 2D 30 3A 36 33 2E 37 2E (V) 30 28 30 30 30 30 2E 30 30 30 2A 6B 76 61 72 29 (V) 0D 0A 31 2D 30 3A 32 34 2E 37 2E 30 28 30 30 30 (V) 30 2E 30 37 32 2A 6B 76 61 72 29 0D 0A 31 2D 30 (V) 3A 34 34 2E 37 2E 30 28 30 30 30 30 2E 33 34 39 (V) 2A 6B 76 61 72 29 0D 0A 31 2D 30 3A 36 34 2E 37 (V) 2E 30 28 30 30 30 30 2E 30 32 39 2A 6B 76 61 72 (V) 29 0D 0A 31 2D 30 3A 33 32 2E 37 2E 30 28 32 33 (V) 32 2E 38 2A 56 29 0D 0A 31 2D 30 3A 35 32 2E 37 (V) 2E 30 28 32 33 33 2E 35 2A 56 29 0D 0A 31 2D 30 (V) 3A 37 32 2E 37 2E 30 28 32 33 38 2E 35 2A 56 29 (V) 0D 0A 31 2D 30 3A 33 31 2E 37 2E 30 28 30 30 31 (V) 2E 38 2A 41 29 0D 0A 31 2D 30 3A 35 31 2E 37 2E (V) 30 28 30 30 34 2E 30 2A 41 29 0D 0A 31 2D 30 3A (V) 37 31 2E 37 2E 30 28 30 30 30 2E 31 2A 41 29 0D (V) 0A 21 34 37 36 30 0D 0A (D) Received valid DSMR at 0 (D) NOT Ready to update (internal clock 19:13:57 UTC, meter clock: 19:13:48, list type 4, est: 0)

[ArnieO]

Thank you for the feedback!

[tronde-ams]

Tested OK.
I have no HAN on this, so only tested FW upgrade / downgrade as described.

Last step going back from 1453bbb to 2.3.9 was extremely slow, about 10 minutes.

My card is this.

I used ESPWEBTOOL on Chromium / linux for flashing with this setting.

[Noschvie]

Using an ESP32-C3 I got the update of esp32c3-2.3.9-1453bbb installed via OTA starting from 2.2.28. The OTA update to 2.3.9 works too. But the HAN connection is lost as a result (details at #892).