Is CONFIG_ESP32_BT_RESERVE_DRAM deprecated?

[Jannik-M91]

Hi!

I have just started working on an ESP32 based project based on the main branch. After some trial and error I got a sample project running but as soon as I enable Bluetooth support the board does not boot anymore. The issue is that I do not have sufficient RAM for ESP heap allocation, so the following assertion fails:

zephyr/soc/espressif/common/esp_heap_runtime.c

Line 36 in b80a058

assert(ESP_HEAP_RUNTIME_MAX_SIZE > 65535);

I started looking through the linker scripts and found the following two files which are mainly used to setup the memory configuration:

https://github.com/zephyrproject-rtos/zephyr/blob/b80a058e9043087f2a114310df48eab4638598b6/soc/espressif/esp32/memory.h
https://github.com/zephyrproject-rtos/zephyr/blob/b80a058e9043087f2a114310df48eab4638598b6/soc/espressif/esp32/default.ld

After some digging I realized that a big chunk of my DRAM gets blocked by CONFIG_ESP32_BT_RESERVE_DRAM which is set to 54 kB by default when CONFIG_BT is set:

zephyr/soc/espressif/esp32/Kconfig

Lines 18 to 21 in b80a058

	config ESP32_BT_RESERVE_DRAM
	hex "Bluetooth controller reserved RAM region"
	default 0xdb5c if BT
	default 0

zephyr/soc/espressif/esp32/default.ld

Lines 20 to 22 in b80a058

	procpu_dram_end = SRAM2_DRAM_END;
	procpu_dram_org = SRAM2_DRAM_USER_START + CONFIG_ESP32_BT_RESERVE_DRAM;
	procpu_dram_len = SRAM2_DRAM_USER_SIZE - CONFIG_ESP32_BT_RESERVE_DRAM;

I have found no further references to CONFIG_ESP32_BT_RESERVE_DRAM, so I assume that reserved memory is never used (I am not very good at reading linker scripts though, so I may have missed something). For me it looks like the memory needed for Bluetooth and WiFi is now allocated on the heap by esp_heap_runtime.c:

https://github.com/zephyrproject-rtos/zephyr/blob/b80a058e9043087f2a114310df48eab4638598b6/soc/espressif/common/esp_heap_runtime.c

This is referenced by both the ESP Bluetooth and WiFi drivers:

https://github.com/zephyrproject-rtos/hal_espressif/blob/59d49a284136fb718cdd6e7b6a56d484f746d215/zephyr/esp32/src/bt/esp_bt_adapter.c
https://github.com/zephyrproject-rtos/hal_espressif/blob/59d49a284136fb718cdd6e7b6a56d484f746d215/zephyr/esp32/src/wifi/esp_wifi_adapter.c

Questions

• Is it safe to set CONFIG_ESP32_BT_RESERVE_DRAM=0 (when using Bluetooth)?
• Should CONFIG_ESP32_BT_RESERVE_DRAM be removed from the ESP32 Kconfig?

[Jannik-M91]

I have been digging some more and it seems like CONFIG_ESP_HEAP_MEM_POOL_REGION_1_SIZE also does nothing. The resulting linker scripts (build/zephyr/linker.cmd) look exactly the same no matter which value I set.

The Espressif HAL does allocate a memory section:

https://github.com/zephyrproject-rtos/hal_espressif/blob/59d49a284136fb718cdd6e7b6a56d484f746d215/zephyr/port/heap/heap_caps_zephyr.c#L15-L23

But I see no references to this in the linker script.

Also, what is the SRAM1_RESERVED_SIZE used for?

zephyr/soc/espressif/esp32/memory.h

Lines 12 to 19 in b80a058

	/* SRAM1 (128kB) instruction/data memory */
	#define SRAM1_IRAM_START (SRAM0_IRAM_START + SRAM0_SIZE)
	#define SRAM1_DRAM_START DT_REG_ADDR(DT_NODELABEL(sram1))
	#define SRAM1_SIZE DT_REG_SIZE(DT_NODELABEL(sram1))
	#define SRAM1_DRAM_END (SRAM1_DRAM_START + SRAM1_SIZE)
	#define SRAM1_RESERVED_SIZE 0x8000
	#define SRAM1_DRAM_USER_START (SRAM1_DRAM_START + SRAM1_RESERVED_SIZE)
	#define SRAM1_USER_SIZE (0x40000000 - SRAM1_DRAM_USER_START)

It reduces the IRAM and reserves some user RAM (Data RAM?). In the linker script two variables are created for that:

zephyr/soc/espressif/esp32/default.ld

Lines 24 to 25 in b80a058

	user_dram_2_seg_org = SRAM1_DRAM_USER_START;
	user_dram_2_seg_len = SRAM1_USER_SIZE;

But they are never used. So I assume currently this memory is wasted?

[arifbalik]

I have this exact problem.

When I run west build -b esp32_devkitc_wroom/esp32/procpu samples/hello_world

I get;

Memory region         Used Size  Region Size  %age Used
           FLASH:      143108 B    4194048 B      3.41%
     iram0_0_seg:       33024 B       224 KB     14.40%
     dram0_0_seg:       16256 B       192 KB      8.27%
     irom0_0_seg:       12036 B      4092 KB      0.29%
     drom0_0_seg:         64 KB      4092 KB      1.56%
    rtc_iram_seg:          0 GB         8 KB      0.00%
    rtc_slow_seg:          0 GB         8 KB      0.00%
    rtc_data_seg:          0 GB         8 KB      0.00%
        IDT_LIST:          0 GB         8 KB      0.00%

where iram_0_0_seg region size should be 192KB but it is not. And also dram0_0_seg size is wrong. Where is the SRAM2 in this case? Its not even showing up here.

[Jannik-M91]

I have done some more testing and unfortunately the memory reserved by ESP heap allocator is needed in addition to the reserved 54kB at the start of SRAM2. On top of that come 8kB which are reserved for ROM functions so the usable size of SRAM2 for DRAM is only ~138kB when Bluetooth is enabled. Of that another 64kB are needed by the ESP heap allocator when BT and WiFi are used, so there are only 74kB usable by the OS which is extremely tight.

The DRAM could expand into SRAM1 so in theory another 128kB could be added (the addresse ranges of SRAM2 and SRAM1 are continuous so from a SW standpoint they could be treated as one memory section), BUT there is another ROM reserved memory section at the start of SRAM1 which must not be used by the OS (I tried, it crashes my application). In the Espressif SDK this is handled by the ESP heap allocator which knows those reserved regions and does not allocate memory there, but this is not implemented in the Zephyr driver.

If found this article very helpful: https://developer.espressif.com/blog/esp32-programmers-memory-model/#dram-organisation

For DRAM this means we are stuck with whatever remains of SRAM2. The only quick solution I found was moving the memory used by the ESP heap allocator into SRAM1 so I can at least save those 64kB which are required for the BT and WiFi heap for my application. The section can be placed behind the ROM reserved area in SRAM1 so there is no problem there.

The rest of SRAM1 can then be used as IRAM. While we are speaking of IRAM: The first 32kB of SRAM0 are reserved for Cache of the PRO CPU, the following 32kB are used for Cache of the APP CPU. This reduces the available IRAM to 160 or 128kB. Additionally when MCUboot is used some of the IRAM used by MCUboot cannot be reclaimed, otherwise MCUboot would override itself while copying the application code. Usually that code should get placed in the Cache section of the APP CPU, so it is "hidden". When only the PRO CPU is used it has to be considered though.

I have re-written the memory.h and default.ld files for the ESP32 to implement what I have described above. Also some small changes are needed for the ESP heap allocator to use SRAM1 instead of SRAM2.

Edit: Something I did not understand at fist: SRAM1 can be addressed via both the instruction and data memory bus. The trick is that on the instruction bus it is addressed in reverse oder. That means on the data bus SRAM2 can grow into SRAM1 from one side while on the instruction bus SRAM0 can grow into SRAM1 from the other side so in theory it can be split as required by the application into IRAM and DRAM. The only problem is the reserved memory region at the start of SRAM1 (when viewed on the data bus). As long as the Zephyr memory allocator does not respect those reserved regions, we are stuck with SRAM2 for DRAM.

Edit 2: Here is how I understand the memory map now:

/*
 * Instruction Memory Bus (SRAM1 addresses are reversed):
 *
 * ********* ------------------------------------
 * | SRAM1 | Reserved for ROM functions
 * |       | ------------------------------------
 * |       | ESP HEAP
 * |       | ------------------------------------
 * |       | IRAM (high addresses)
 * *********
 * | SRAM0 | IRAM (low addresses)
 * |       | ------------------------------------
 * |       | 2nd Stage Bootloader IRAM (optional)
 * |       | External Memory Cache
 * ********* ------------------------------------
 *
 * Data Memory Bus (SRAM1 addresses are not reversed):
 *
 * *********
 * | SRAM1 | IRAM (high addreses)
 * |       | ------------------------------------
 * |       | ESP HEAP
 * |       | ------------------------------------
 * |       | Reserved for ROM functions
 * ********* ------------------------------------
 * | SRAM2 | DRAM
 * |       | ------------------------------------
 * |       | Reserved for Bluetooth (optional)
 * |       | Reserved for ROM functions
 * ********* ------------------------------------
 */

[arifbalik]

Do you have your changes somewhere in your fork?

[Jannik-M91]

I have just pushed it here:

https://github.com/Proton-Camera/zephyr/tree/esp32
https://github.com/Proton-Camera/hal_espressif

Note that I have only used the "Pro" CPU so far, not sure if using the "App" CPU still works with my changes.

[arifbalik]

Perfect, thanks very much, this fits perfectly for my use case. I'll give it a try.

It seems like this part of the codebase needs some rework. I'll see if there is any ongoing work and open an issue if there isn't.