Segmented memory address bug caused by ram_mask_1_reg assertion in dma_if_pcie_rd.v

[C0L]

Thanks for this great project!

I am prototyping a system that uses DMA on an Alveo U250 FPGA, Vivado 2023.2, and Rev. 23 of the Ultrascale+ Integrated Block for PCIe express. My setup uses pcie_us_if + dma_if_pcie + dma_psdpram to perform r/w from a host which I believe is the suggested configuration. Issuing DMA reads less than 64 bytes works perfectly, and the data is written to the psdpram correctly. Issuing reads greater than 64 bytes works except for an incorrect first address generated by dma_if_pcie in its write of received data to the psdpram.

Here is the ILA of the first batch of writes from dma_if_pcie_rd to psdpram in completion of a 256 byte read. The first block of 64 bytes in this read is 01010101....., the second block of 64 is 02020202......, the third block of 64 is 0303030303......., the fourth block of 64 is 040404040...... destined for address 8192 of the psdpram, which I believe should be 0x40, of the even chunk of the segmented memory.

The address generated is incorrect, specifying 0x41 of the even chunks of the segmented memory. Oddly, the successive chunks are placed at the correct addresses. The ram segment address width is 11 bits, and the data is written at the following locations:
01010101.... -> 0x41 (even chunk)
02020202.... -> 0x40 (odd chunk)
03030303.... -> 0x41 (even chunk)
04040404.... -> 0x41 (odd chunk)
This results in data loss where the 64 bytes at address 8192 remain their default state of 0.

Digging into the code in dma_if_pcie_rd it seems that for some reason this block of code is being activated:

if (ram_mask_1_reg[i]) begin
    ram_wr_cmd_addr_pipe_next[i*RAM_SEG_ADDR_WIDTH +: RAM_SEG_ADDR_WIDTH] = addr_delay_reg[RAM_ADDR_WIDTH-1:RAM_ADDR_WIDTH-RAM_SEG_ADDR_WIDTH]+1;
end

This is explains where the +1 is coming from in the first address but there is no reason for the mask register to be asserted I believe. I have not been able to piece together exactly what the issue is yet, but by removing that block of code I get the following from the ILA:

The addresses are correct and things seem to be working. I am sure that removing that block of code breaks all sorts of other functionality (anything that is not aligned probably), but luckily what I am doing so far does not call for that. Would like to find a full solution if possible though!

Here are the full config parameters of my setup:

AXIS_PCIE_DATA_WIDTH = 512,
AXIS_PCIE_KEEP_WIDTH = 16,
AXIS_PCIE_RC_USER_WIDTH = 161,
AXIS_PCIE_RQ_USER_WIDTH = 137,
AXIS_PCIE_CQ_USER_WIDTH = 183,
AXIS_PCIE_CC_USER_WIDTH = 81,
RC_STRADDLE = 1,
RQ_STRADDLE = 1,
CQ_STRADDLE = 1,
CC_STRADDLE = 1,
RQ_SEQ_NUM_WIDTH = 6,
RQ_SEQ_NUM_ENABLE = 1,
IMM_ENABLE = 0,
IMM_WIDTH = 32,
PCIE_TAG_COUNT = 256,
READ_OP_TABLE_SIZE = 256,
READ_TX_LIMIT = 32,
READ_CPLH_FC_LIMIT = 512,
READ_CPLD_FC_LIMIT = 2048,
WRITE_OP_TABLE_SIZE = 32,
WRITE_TX_LIMIT = 32,
TLP_DATA_WIDTH = 512,
TLP_STRB_WIDTH = 16,
TLP_HDR_WIDTH = 128,
TLP_SEG_COUNT = 1,
TX_SEQ_NUM_COUNT = 2,
TX_SEQ_NUM_WIDTH = 5,
TX_SEQ_NUM_ENABLE = 1,
PF_COUNT = 1,
VF_COUNT = 0,
F_COUNT = PF_COUNT+VF_COUNT,
TLP_FORCE_64_BIT_ADDR = 0,
CHECK_BUS_NUMBER = 0,
RAM_SEL_WIDTH = 2,
RAM_ADDR_WIDTH = 18,
RAM_SEG_COUNT = 2,
RAM_SEG_DATA_WIDTH = 512,
RAM_SEG_BE_WIDTH = 64,
RAM_SEG_ADDR_WIDTH = 11,
AXI_DATA_WIDTH = 512,
AXI_STRB_WIDTH = 64,
AXI_ADDR_WIDTH = 26,
AXI_ID_WIDTH = 8,
PCIE_ADDR_WIDTH = 64,
LEN_WIDTH = 16,
TAG_WIDTH = 8

Update: There seems to be a similar or identical issue with the legacy ultracale+ design as well.