Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Rdma device rename in container #28

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions pkg/utils/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ import (
"fmt"
"path"
"path/filepath"
"strconv"
"strings"

"github.com/vishvananda/netlink"
)
Expand Down Expand Up @@ -45,3 +47,51 @@ func GetVfPciDevFromMAC(mac string) (string, error) {
}
return dev, err
}

// Get RDMA device prefix and index. e.g for mlx5_3: prefix is mlx5 and index is 3
// Note: the index is not related to the kernel RDMA device index
func getRdmaDevNamePrefixIndex(rdmaDev string) (prefix string, idx uint64, err error) {
s := strings.Split(rdmaDev, `_`)
if len(s) != 2 {
return "", 0, fmt.Errorf("unexpeded RDMA device format: %s", rdmaDev)
}
prefix = s[0]
idx, err = strconv.ParseUint(s[1], 0, 32)
if err != nil {
err = fmt.Errorf("failed to parse RDMA device index: %s, %v", rdmaDev, err)
}
return prefix, idx, err
}

func getRdmaDevIndexFromName(rdmaDev string) (uint64, error) {
_, idx, err := getRdmaDevNamePrefixIndex(rdmaDev)
return idx, err
}

// Get the next RDMA device name for a given RDMA device prefix
func GetNextRdmaDeviceName(prefix string, currDevs []string) (string, error) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adrianchiris
I think it is better to use PCI device based naming scheme from
https://github.com/linux-rdma/rdma-core/blob/master/kernel-boot/rdma-persistent-naming.rules

And let rdma/hca operator to configure system for rdma device naming.

var nextDevIdx uint64
nextDevIdx = 0
if len(currDevs) != 0 {
for _, dev := range currDevs {
if !strings.HasPrefix(dev, prefix) {
continue
}
// extract index
idx, err := getRdmaDevIndexFromName(dev)
if err != nil {
return "", err
}
if idx > nextDevIdx {
nextDevIdx = idx + 1
}
}
}
return fmt.Sprintf("%s_%d", prefix, nextDevIdx), nil
}

// Get RDMA device driver prefix. e.g for mlx5_3 the prefix would be mlx5
func GetRdmaDevicePrefix(rdmaDev string) (string, error) {
prefix, _, err := getRdmaDevNamePrefixIndex(rdmaDev)
return prefix, err
}