DWC2 DMA support #2576

HiFiPhile · 2024-04-05T20:32:34Z

Describe the PR
Add Internal DMA support to DWC2.

Status

Model	Core Rev	Enumeration	`cdc_ual_ports`	`audio_4_channel_mic`	`video_capture_2ch`
STM32F407 FS w/wo DMA	2.81a	✔️	✔️	✔️	✔️
STM32F429 FS w/wo DMA	2.81a	✔️	✔️	✔️	✔️
STM32F723 FS + HS w/wo DMA	3.30a	✔️	✔️	✔️	✔️
STM32U5 HS w/wo DMA	4.11a	✔️	✔️	✔️	✔️

Model	Core Rev	Enumeration	`cdc_msc_freertos`	`audio_4_channel_mic_freertos`	`hid_composite_freertos`
ESP32-S3 FS w/wo DMA	4.00a	✔️	✔️	✔️	✔️

Benchmark

STM32F723E-DISCO
IAR 9.50 High-Balanced
I-Cache enabled

Method used in #920, all buffers set to 2048b.

	Throughput	Total CPU Usage
DMA OFF	26.7 MB/s	55%
DMA ON	29.5 MB/s	26%

src/portable/synopsys/dwc2/dcd_dwc2.c

HiFiPhile · 2024-04-25T22:03:17Z

src/portable/synopsys/dwc2/dcd_dwc2.c

@@ -67,18 +67,6 @@
 // Debug level for DWC2
 #define DWC2_DEBUG    2

-#ifndef dcache_clean
-#define dcache_clean(_addr, _size)


dcache code should be removed and add MPU suggestion to make usb ram non cacheable:

Not all internal buffers are aligned to cache line size and has size of multiple cache line size, speculative read or cache eviction of adjacent variables could lead to racing with DMA.

Frequent cache clean invalidate hurt performance

In most cases Cortex-M7 recommend use DTCM where cache is not used

not entirely sure with this case, as H743 and probably most of stm32 m7 h7 USB DMA controller don't have access to DTCM

At least for stm32f7xx, those all seem to have the AHB-S port of the M7 connected directly to the bus matrix, so it is accessible by the USB DMA in this case. Though, that is only for DTCM, ITCM is still inaccessible by anything besides the CPU.

right, I haven't check f7, maybe this is specific to h7 only

HiFiPhile · 2024-04-25T22:04:34Z

src/portable/synopsys/dwc2/dcd_dwc2.c

+static inline bool dma_enabled(uint8_t rhport)
+{
+  // DMA doesn't support fifo transfer
+#ifdef TUD_AUDIO_PREFER_RING_BUFFER


Don't want to depend on class driver but maybe it's the simplest way.

CFG_TUD_DWC2_DMA must be enabled at compile time, audio driver/user should adapt accordingly.

HiFiPhile · 2024-04-25T22:08:57Z

src/portable/synopsys/dwc2/dcd_dwc2.c

@@ -351,6 +382,12 @@ static void bus_reset(uint8_t rhport) {
  // Setup the control endpoint 0
  _allocated_fifo_words_tx = 16;

+  // DMA needs extra space for processing
+  if(dma_enabled(rhport)) {
+    uint16_t reserved = _dwc2_controller[rhport].ep_fifo_size / 4- dwc2->ghwcfg3_bm.total_fifo_size;


Kernel use dwc2->ghwcfg3_bm.total_fifo_size it should be reliable.
We can even remove _dwc2_controller[rhport].ep_fifo_size and use this field instead, except GD32VF103 looks strange.

yeah, this is the max epinfo where scatt/gather dma is used. We can use gdfifocfg to dynamic change the epinfo base to reduce the DMA region for simple DMA mode.

…_dma

roma-jam · 2024-09-18T10:52:19Z

Hi @HiFiPhile ,
May I ask how did you measure the total CPU load?

hathach · 2024-09-18T14:35:59Z

thank you very much @HiFiPhile again for another great PR. sorry for super late response, I am doing more test with this, seems like it have some issue with my h743 eval which uses external HS PHY. That is ok, I am doing more testing and reading hopefully we could fix and merge this soon.

HiFiPhile · 2024-09-18T15:44:11Z

seems like it have some issue with my h743 eval which uses external HS PHY

Thanks for testing, it's the combination I'm missing.

May I ask how did you measure the total CPU load?

@roma-jam In short I'm using GPIO toggle in ISR and in event queue processing, then take the averaged sum of 10 cycles.

hathach · 2024-09-24T10:01:18Z

found the issue, h743 usb dma cannot access DTCM, change linker to use SRAM1 and it is all good. for this reason, it is also required to have CFG_TUD_MEM_SECTION for dma as well. I will do more clean up and we can merge this real soon.

…d explanation for EPInfo address and GDFIFO. some function rename update h743 linker to use SRAM1 since USB DMA cannot access DTCM ram update xmc4500 to use uuid for testing

src/portable/synopsys/dwc2/dcd_dwc2.c

  }

  return true;
 }

+static void dfifo_init(uint8_t rhport) {


hathach

really long review with incorrect poking. It turns out H743 USB DMA cannot access DTCM ram. Though that force me to do lots of reading and have a better understand of the dwc2 register for EPInfo for DMA. Which store 1 words for normal DMA, and 4 words for scatter/gather. We can dynamically configure gdififocfg with epinfo base to save footprint. Since currently we does not support scatter/gather, we only need 2xep_count.

While troubleshooting, I also rewrite to simplify the fifo allocation scheme and update documentation as well since we don't need to close/re-open iso anymore.

Since DMA cannot dynamically switched on/off per endpoint, I have made ths an compiled timed option, which is default to off CFG_TUD_DWC2_DMA. The reason is, in the future with non-dma mode, we can actually get rid of local ep_buffer in the driver e.g cd (tu_fifo -->dwc2 dfifo). Which I have been thinkging but haven't done yet. So yeah DMA will require more memory for ep_buf (to live long enough for transfer to complete as well as word aligned like other ports). Other driver such audio should adapt with the new cfg option.

from Linux kernel driver
 * Unfortunately the choice to use DMA or not is global to the controller
 * and seems to be only settable when the controller is being put through
 * a core reset. This means we either need to fix the gadgets to take
 * account of DMA alignment, or add bounce buffers (yuerk).

PS: I have tested this with esp32s3, xmc4500, stm32 h743, f412, u5a5 and they works flawlessly. Thank you @HiFiPhile again for another great PR.

hathach · 2024-09-24T11:42:46Z

@HiFiPhile let me know if you spot any issue with changes that I made, otherwise we could merge this now :) (finally)

HiFiPhile · 2024-09-24T12:26:24Z

h743 usb dma cannot access DTCM

Oh it's a little pity for performance. I think we need to update the BSP with MPU config to make the region non-cacheable.

For gdfifocfg.EPINFOBASE (max is ghwcfg3.dfifo_depth) I'm the max is more like OTG_DFIFO_DEPTH ? Since ghwcfg3.dfifo_depth is already the value subtracted EP_LOC_CNT.

I'll test with my boards if everything works.

hathach · 2024-09-24T13:32:49Z

h743 usb dma cannot access DTCM

Oh it's a little pity for performance. I think we need to update the BSP with MPU config to make the region non-cacheable.

Maybe just leave it as it is if it isn't much of an issue. I prefer to have generic/simpler code to maintain :)

For gdfifocfg.EPINFOBASE (max is ghwcfg3.dfifo_depth) I'm the max is more like OTG_DFIFO_DEPTH ? Since ghwcfg3.dfifo_depth is already the value subtracted EP_LOC_CNT.

I'll test with my boards if everything works.

From my understanding, it is the max if we use scatter/gather DMA mode. I think the parameter is pre-calculated by synopsys ip configuration tool based on the number of endpoints. However, we can dynamic assign gdfifocfg if we only use the buffer/normal DMA mode.

HiFiPhile · 2024-09-24T19:34:00Z

Just tested on STM32F407, STM32F723 and STM32U5A5, didn't redo the full matrix but all major combinations work.

hathach · 2024-09-25T03:16:22Z

thank you @HiFiPhile for retesting, I think it is perfect to merge now. I will do more follow-up clean up to opt out code when DMA is used later on e.g remove the xfer fifo and tu_fifo const etc..

This comment was marked as resolved.

Sign in to view

HiFiPhile commented Apr 6, 2024

View reviewed changes

src/portable/synopsys/dwc2/dcd_dwc2.c Outdated Show resolved Hide resolved

HiFiPhile force-pushed the dwc2_dma branch from bf8b60b to ceebe23 Compare April 6, 2024 17:29

HiFiPhile mentioned this pull request Apr 7, 2024

tinyusb support dma feature #2191

Closed

1 task

HiFiPhile force-pushed the dwc2_dma branch 3 times, most recently from 9cafcdd to 97251e4 Compare April 23, 2024 21:01

HiFiPhile added 4 commits April 25, 2024 22:44

dwc2: add dma support.

cde7223

Check IN ep count limit.

394dc06

Fix DMA FIFO reservation.

298f7f2

Change DMA condition.

8765568

HiFiPhile force-pushed the dwc2_dma branch from 97251e4 to 8765568 Compare April 25, 2024 21:27

HiFiPhile commented Apr 25, 2024

View reviewed changes

HiFiPhile changed the title ~~WIP: DWC2 DMA support~~ DWC2 DMA support Apr 25, 2024

HiFiPhile marked this pull request as ready for review April 25, 2024 22:15

HiFiPhile mentioned this pull request May 1, 2024

Rewrite of NCM device driver #2227

Merged

This comment was marked as resolved.

Sign in to view

HiFiPhile mentioned this pull request May 10, 2024

CDC : Add xfer_fifo & DMA support #1064

Closed

HiFiPhile mentioned this pull request Jun 21, 2024

STM32L475 will not send ACK packets after data stage finished #1494

Open

HiFiPhile added 2 commits June 27, 2024 20:49

Merge branch 'master' of https://github.com/hathach/tinyusb into dwc2…

eefca22

…_dma

Fix GenID 3.10 issue on STM32L4.

6d4e2f6

hathach added the Prio Higher label Aug 28, 2024

Merge branch 'master' into dwc2_dma

7867464

tore-espressif mentioned this pull request Sep 9, 2024

Upstream ESP32-P4 support #2791

Closed

hathach self-requested a review September 18, 2024 10:59

hathach added 2 commits September 18, 2024 18:01

Merge branch 'refs/heads/master' into fork/HiFiPhile/dwc2_dma

9bfb3e7

skip pico2 dual test, it seems not stable

40b5517

hathach added 2 commits September 24, 2024 17:38

update dfifo allocation scheme to use top pointer, update document an…

86b4608

…d explanation for EPInfo address and GDFIFO. some function rename update h743 linker to use SRAM1 since USB DMA cannot access DTCM ram update xmc4500 to use uuid for testing

add CFG_TUD_DWC2_DMA, make it compile time option

a124438

github-advanced-security bot found potential problems Sep 24, 2024

View reviewed changes

src/portable/synopsys/dwc2/dcd_dwc2.c

}

return true;

}

static void dfifo_init(uint8_t rhport) {

Check notice

Code scanning / CodeQL

Unused static function Note

Static function dfifo_init is unreachable

hathach approved these changes Sep 24, 2024

View reviewed changes

more rename

6a15e78

hathach force-pushed the dwc2_dma branch from 814bdf7 to 6a15e78 Compare September 24, 2024 11:12

Add a note about data cache.

e483c6a

hathach merged commit c8ab65f into hathach:master Sep 25, 2024
97 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DWC2 DMA support #2576

DWC2 DMA support #2576

HiFiPhile commented Apr 5, 2024 •

edited

Loading

This comment was marked as resolved.

HiFiPhile Apr 25, 2024

hathach Sep 24, 2024 •

edited

Loading

leptun Sep 24, 2024

hathach Sep 24, 2024

HiFiPhile Apr 25, 2024

hathach Sep 24, 2024

HiFiPhile Apr 25, 2024 •

edited

Loading

hathach Sep 24, 2024

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

roma-jam commented Sep 18, 2024

hathach commented Sep 18, 2024

HiFiPhile commented Sep 18, 2024 •

edited

Loading

hathach commented Sep 24, 2024

hathach left a comment •

edited

Loading

hathach commented Sep 24, 2024

HiFiPhile commented Sep 24, 2024

hathach commented Sep 24, 2024

HiFiPhile commented Sep 24, 2024

hathach commented Sep 25, 2024

DWC2 DMA support #2576

DWC2 DMA support #2576

Conversation

HiFiPhile commented Apr 5, 2024 • edited Loading

Status

Benchmark

This comment was marked as resolved.

HiFiPhile Apr 25, 2024

Choose a reason for hiding this comment

hathach Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

leptun Sep 24, 2024

Choose a reason for hiding this comment

hathach Sep 24, 2024

Choose a reason for hiding this comment

HiFiPhile Apr 25, 2024

Choose a reason for hiding this comment

hathach Sep 24, 2024

Choose a reason for hiding this comment

HiFiPhile Apr 25, 2024 • edited Loading

Choose a reason for hiding this comment

hathach Sep 24, 2024

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

roma-jam commented Sep 18, 2024

hathach commented Sep 18, 2024

HiFiPhile commented Sep 18, 2024 • edited Loading

hathach commented Sep 24, 2024

hathach left a comment • edited Loading

Choose a reason for hiding this comment

hathach commented Sep 24, 2024

HiFiPhile commented Sep 24, 2024

hathach commented Sep 24, 2024

HiFiPhile commented Sep 24, 2024

hathach commented Sep 25, 2024

HiFiPhile commented Apr 5, 2024 •

edited

Loading

hathach Sep 24, 2024 •

edited

Loading

HiFiPhile Apr 25, 2024 •

edited

Loading

HiFiPhile commented Sep 18, 2024 •

edited

Loading

hathach left a comment •

edited

Loading