esptool.py: use upstream version, switch to python3

[benpicco]

Contribution description

Instead of maintaining our own fork of esptool.py, switch to the upstream version. (Also includes support for ESP32-S2 now)

But patch it to use Python3.

Testing procedure

esp* can still be flashed.

Issues/PRs references

I would suddenly get this error espressif/esptool#350 after upgrading to Ubuntu 20.04
Using Python3 fixed it.

[benpicco]

gen_esp32part.py can also be found in esp-idf/components/partition_table/gen_esp32part.py so we shouldn't have to keep a copy of that either.

[gschorcht]

(Also includes support for ESP32-S2 now)

I don't think that we will be able to support ESP32-S because it requires a much newer ESP-IDF version, I guess 4.1. We still use version 3.1 and it would not be easy to port the current RIOT implementation for ESP32 to this new ESP-IDF version. The reason is, that the RIOT port for ESP32 can't use the FreeRTOS based ESP-IDF. Rather it is uses a mixture of vendor code extracted from the ESP-IDF and changed vendor code to make it working with RIOT. I guess that a change from 3.1 to 4.1 would require to reimplement large parts of the RIOT port for ESP32 😟

I have tried such a version upgrade for the ESP8266. I have been working on it for 4 months and it is still not finished. So for now I would prefer to leave the ESP32 port as it is and continue using ESP-IDF version 3.1. In this respect, support for the ESP32-S is not a problem.

[gschorcht]

gen_esp32part.py can also be found in esp-idf/components/partition_table/gen_esp32part.py so we shouldn't have to keep a copy of that either.

gen_esp32part.py is required for preparing the image for flashing. But flashing should be possible without having ESP-IDF installed. That's why we have our own copies of esptool.py and gen_esp32part.py.

[benpicco]

I don't think that we will be able to support ESP32-S because it requires a much newer ESP-IDF version

No pressure 😉
I just thought that would be a nice bonus.

In the first version of this PR I just replaced #!/usr/bin/env python with #!/usr/bin/env python3 because of the error I got (missing Python 2 libraries) and all the other tools in RIOT use Python 3.

I then started to wonder why we keep a copy of esptool.py in the RIOT repo, as this encourages modifying it, which makes upgrades harder.

So I replaced it with a pkg like it's done with other tools that have an upstream.
Upstream still didn't switch to Python3, so I added the patch, but now to the pkg.

[gschorcht]

Sorry for the delay. I was completely busy in last three months and starting resume my work on RIOT now.

Our esptool.py version was extracted from the ESP8266_RTOS_SDK. The reason was that the upstream version can't be used to generate ESP8266 images using the ESP8266_RTOS_SDK (firmware V3).

While our esptool.py allows to specify generate an ESP8266 image for firmware V3

RIOT/dist/tools/esptool/esptool.py

Lines 2343 to 2350 in ef4835a

	if args.chip == 'esp32':
	image = ESP32FirmwareImage()
	elif args.version == '1': # ESP8266
	image = ESP8266ROMFirmwareImage()
	elif args.version == '2': # ESP8266
	image = ESP8266V2FirmwareImage()
	else:
	image = ESP8266V3FirmwareImage()

the upstream repository doesn't https://github.com/espressif/esptool/blob/master/esptool.py#L2837-L2852.

That is, the upstream version will not work for ESP8266 😟

[benpicco]

Huh that's weird - I would have expected the upstream to version to support everything.
Could it be that they folded both v2 and v3 into ESP8266V2FirmwareImage()?

[gschorcht]

Huh that's weird - I would have expected the upstream to version to support everything.

Indeed. That is what someone would expect, especially if the same guys who are developing esptool.py are involved in the development of the ESP8266_RTOS_SDK. But as I said, the current master of ESP8266_RTOS_SDK uses still its own version of esptool.py which know a firmware version 3.

Could it be that they folded both v2 and v3 into ESP8266V2FirmwareImage()?

I don't think so. I have tried to use the upstream version of esptool.py. But with --version 2 here

RIOT/cpu/esp8266/Makefile.include

Line 17 in 028c0d4

FLASH_OPTS = --version 3

the bootloader says:

E (48) esp_image: image at 0x10000 has invalid magic byte
E (48) boot: Factory app partition is not bootable
E (48) boot: No bootable app partitions in the partition table
user code done

That is, the bootloader expects another magic byte. Using --version 3 as option for the right firmware version was determined from the ESP8266_RTOS_SDK make system.

Finally, we should keep our own version as long as ESP8266_RTOS_SDK also uses its own version. Therefore we should close this PR in favor of PR #14392.