CNDP VPP Plugin

[maryamtahhan]

Ho Folks I've noticed that cndp_buf_alloc tries to allocate buffers from a numa co-located buffer pool

n_alloc = vlib_buffer_alloc_from_pool (vm, bi, nb_bufs, cd->pool);

However, the umem setup for each device isn't pool specific:

 cd->umem_addr = uword_to_pointer (vm->buffer_main->buffer_mem_start, void *);

And the same with the umem configuration setup

p_config->umem_addr =
         uword_to_pointer (vm->buffer_main->buffer_mem_start, void *);

I think either the allocation should be modified to not allocate buffers from a specific pool unless you are using a umem address that is being configured for that cd for that exact pool or the umems should be configured appropriately to match the right pools...

In the vlib_main_t struct you can access the available buffer pools from the buffer_main parameter: https://github.com/FDio/vpp/blob/39fdefdc90abeaa142e4056bd40ff3e89dc689ab/src/vlib/buffer.h.
so maybe setting the umem address to the right pool from the start is more appropriate using something like:

vlib_buffer_pool_t *bp;
     /* *INDENT-OFF* */
     vec_foreach (bp, vm->buffer_main->buffer_pools)
   {
     
       if(bp->numa_node ==cndp_get_numa ((char *)ifname)) {
                vlib_physmem_map_t *pm;
            pm = vlib_physmem_get_map (vm, bp->physmem_map_index);
            cd->umem_addr =  p_config->umem_addr = pm->base;
          }

   }

I think it's also worth investigating how many pools VPP allocates by default and what happens when the number of AF_XDP queues increases dramatically using only 1 of the available mempools... there may be performance impacts

[jeffreybshaw]

I think it's also worth investigating how many pools VPP allocates by default

By default, VPP allocates a buffer pool per NUMA node. See here.

and what happens when the number of AF_XDP queues increases dramatically using only 1 of the available mempools

I don't think we're only using a single mempool. Here's why...

We configure the umem using buffer_mem_start/size, which is eventually used for the XDP_UMEM_REG setsockopt(). This makes it so all the AF_XDP sockets we create can reference any buffer belonging to any pool, regardless of the pool the buffer was originally allocated, since the buffer_mem area is used for all buffer pools.

We do want to allocate buffers from a NUMA co-located pool, and hope we get lucky and the packets received on a device attached to a node are transmitted on a device also attached to the same node. In case we don't get lucky, the packet may need to be transmitted on a device attached to a different node. If the buffer was allocated from pool[0] and needs to transmit on a device that had a umem which is only valid for buffers from pool[1], I think the packet would need to be copied into a new buffer from pool[1].

Looking at lib/core/xskdev.:__get_mbuf_addr_tx_unaligned() we do

addr   = xskdev_buf_get_addr(xi, mb) - umem_addr - (uint64_t)xi->buf_mgmt.pool_header_sz;

Which means if we had a buffer from pool[0], the address would be like, 0x1000, but the umem start address for pool[1] might be like 0x20000, which would result in a very large addr.

Moral of the story is that packets cannot be reliably transmitted across AF_XDP sockets that do not use the same umem.

[maryamtahhan]

I don't think we're only using a single mempool. Here's why...

We configure the umem using buffer_mem_start/size, which is eventually used for the XDP_UMEM_REG setsockopt(). This makes it so all the AF_XDP sockets we create can reference any buffer belonging to any pool, regardless of the pool the buffer was originally allocated, since the buffer_mem area is used for all buffer pools.

If the AF_XDP queues all belong to the one device, or both devices sit on the same numa node you will be using a single mempool...

I just remember seeing some pretty weird interactions when using a pretty large number of queues (on a single device) and only one (numa co-located) mempool in the past.

Moral of the story is that packets cannot be reliably transmitted across AF_XDP sockets that do not use the same umem.

Ok, I can see that sharing using a large umem space in this instance can help avoid the copy between AF_XDP sockets attached to different netdevs...

Anyway - I will close this discussion topic as resolved. Thanks @jeffreybshaw for the explanation