Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Update VFIO to be bootc compliant #2150

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

feat: Update VFIO to be bootc compliant #2150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f5fb7c8
feat: include VFIO modules on image
HikariKnight Jan 21, 2025
4ee3cc6
feat(dx): update bluefin-dx-kvmfr-setup script for bootc
HikariKnight Jan 21, 2025
55c3b7a
feat(dx): update configure-vfio to not require initramfs regeneration
HikariKnight Jan 21, 2025
07c776c
feat: add ls-iommu to image
HikariKnight Jan 22, 2025
49e071d
chore: make a dir to extract ls-iommu into for better cleanup
HikariKnight Jan 22, 2025
2c86005
feat: add text mentioning ls-iommu in configure-vfio
HikariKnight Jan 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions build_files/dx/04-override-install-dx.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,12 @@ rm -rf /tmp/monaspace-font*
fc-cache -f /usr/share/fonts/monaspace
fc-cache --system-only --really-force --verbose

# ls-iommu helper tool for listing devices in iommu groups (PCI Passthrough)
DOWNLOAD_URL=$(curl https://api.github.com/repos/HikariKnight/ls-iommu/releases/latest | jq -r '.assets[] | select(.name| test(".*x86_64.tar.gz$")).browser_download_url')
curl --retry 3 -Lo /tmp/ls-iommu.tar.gz "$DOWNLOAD_URL"
mkdir /tmp/ls-iommu
tar --no-same-owner --no-same-permissions --no-overwrite-dir -xvzf /tmp/ls-iommu.tar.gz -C /tmp/ls-iommu
mv /tmp/ls-iommu/ls-iommu /usr/bin/
rm -rf /tmp/ls-iommu*

echo "::endgroup::"
21 changes: 9 additions & 12 deletions just/bluefin-system.just
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,11 @@ toggle-devmode:
source /usr/lib/ujust/ujust.sh
CURRENT_IMAGE=$(rpm-ostree status -b --json | jq -r '.deployments[0]."container-image-reference"')
if grep -q "/var/ublue-os/image" <<< $CURRENT_IMAGE ; then
cat <<EOF
bash -c "cat <<EOF
Before we can switch to the Bluefin Developer Experience
the current system needs an update. Please run 'ujust update'
and reboot your system when the update is finished
EOF
EOF"
exit
fi
if /bin/grep -q "\-dx" <<< $CURRENT_IMAGE ; then
Expand Down Expand Up @@ -132,22 +132,22 @@ configure-vfio ACTION="":
fi
OPTION={{ ACTION }}
if [ "$OPTION" == "help" ]; then
cat <<EOF
bash -c "cat <<EOF
Usage: ujust configure-vfio <option>
<option>: Specify the quick option to skip the prompt
Use 'vfio-on' to select Enable VFIO drivers
Use 'vfio-off' to select Disable VFIO drivers
Use 'kvmfr' to select Autocreate Looking-Glass shm
EOF
Use 'kvmfr' to enable kvmfr module for looking-glass
EOF"
exit 0
elif [ "$OPTION" == "" ]; then
cat <<EOF
bash -c "cat <<EOF
${bold}VFIO and kvmfr Configuration${normal}
This is only used for GPU passthrough of a secondary dGPU.
It will enable vfio and configure kvmfr for use with $(Urllink "https://looking-glass.io" "Looking Glass")
If you do not plan to use any of this then press ESC.
${bold}NOTE:${normal} Since this is a niche use case, support will be ${b}very limited${n}
EOF
EOF"
OPTION=$(Choose \
"Enable VFIO drivers" \
"Disable VFIO drivers" \
Expand All @@ -158,8 +158,6 @@ configure-vfio ACTION="":
echo "Enabling VFIO..."
CPU_VENDOR=$(grep "vendor_id" "/proc/cpuinfo" | uniq | awk -F": " '{ print $2 }')
VENDOR_KARG="unset"
echo 'add_drivers+=" vfio vfio_iommu_type1 vfio-pci "' | sudo tee /etc/dracut.conf.d/vfio.conf
rpm-ostree initramfs --enable
if [[ ${CPU_VENDOR} == "AuthenticAMD" ]]; then
VENDOR_KARG="amd_iommu=on"
elif [[ ${CPU_VENDOR} == "GenuineIntel" ]]; then
Expand All @@ -180,6 +178,8 @@ configure-vfio ACTION="":
echo "Please understand that since this is such a niche use case, support will be very limited!"
echo "To add your unused/second GPU device ids to the vfio driver by running"
echo 'rpm-ostree kargs --append-if-missing="vfio-pci.ids=xxxx:yyyy,xxxx:yyzz"'
echo 'you can find these using "ls-iommu -gr" or "ls-iommu -grr"'
echo 'for a dedicated gpu there will be at least 1 vga device and 1 audio device.'
echo "NOTE: Your second GPU will not be usable by the host after you do this!"
fi
elif [[ "${OPTION,,}" =~ (^disable[[:space:]]vfio|vfio-off) ]]; then
Expand All @@ -195,9 +195,6 @@ configure-vfio ACTION="":
echo "$VFIO_IDS"
VFIO_IDS_KARG="--delete-if-present=\"$VFIO_IDS\""
fi
echo "Removing dracut modules"
sudo rm /etc/dracut.conf.d/vfio.conf
rpm-ostree initramfs --enable
rpm-ostree kargs \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we still run kargs on a bootc-only system?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we use it in bazzite, bootc just does not like the initramfs rebuilding and layering from my understanding.
but i can remove lact from my install and double check to be 100% sure

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just checked, can be used with kargs, just not layers and initramfs

--delete-if-present="iommu=pt" \
--delete-if-present="iommu=on" \
Expand Down
1 change: 1 addition & 0 deletions system_files/dx/usr/lib/dracut/dracut.conf.d/80-vfio.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
force_drivers+=" vfio vfio_iommu_type1 vfio-pci "
37 changes: 23 additions & 14 deletions system_files/dx/usr/libexec/bluefin-dx-kvmfr-setup
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ source /usr/lib/ujust/ujust.sh
# Required disclaimer and where to report issues first
echo "$(Urllink "https://looking-glass.io/docs/rc/ivshmem_kvmfr/#libvirt" "This module") along with $(Urllink "https://looking-glass.io" "Looking Glass") is very experimental and not recommended for production use!"
echo "The ublue team packages the kvmfr module only because it has to be supplied with the system image while using an atomic desktop."
echo "If you do plan to use Looking Glass, please $(Urllink "https://universal-blue.discourse.group/docs?topic=956" "follow the guide here") on how to compile it for your system."
echo "If you do plan to use Looking Glass, please $(Urllink "https://docs.bazzite.gg/Advanced/looking-glass/#compiling-looking-glass-client" "follow the guide here") on how to compile it for your system."
echo "To use the kvmfr module after enabling it, just add and edit the xml for libvirt from the documentation in the first link."
echo "Since we package the kvmfr module please open kvmfr related issues you have on Bluefin and tag me"
echo "in the $(Urllink "https://discord.gg/WEu6BdFEtp" "Universal Blue Discord") or the $(Urllink "https://github.com/ublue-os/bluefin/issues" "Bluefin Github issue tracker")."
Expand All @@ -15,17 +15,22 @@ if [ "$CONFIRM" == "Cancel" ]; then
exit 0
fi

# Add kvmfr to dracut so that it's modprobe file can be used
echo ""
# Add kvmfr modprobe file following upstream documentation
echo "Setting up kvmfr module so it loads next boot"
sudo bash -c 'cat << KVMFR_DRACUT > /etc/dracut.conf.d/kvmfr.conf
install_items+=" /etc/modprobe.d/kvmfr.conf "
KVMFR_DRACUT'
if [ -f "/etc/modprobe.d/kvmfr.conf" ]; then
echo "Re-creating dummy kvmfr modprobe file"
sudo rm /etc/modprobe.d/kvmfr.conf
fi
sudo bash -c 'cat << KVMFR_MODPROBE > /etc/modprobe.d/kvmfr.conf
# This is a dummy file and changing it does nothing
# If you want to change the kvmfr static_size_mb
# Run "rpm-ostree kargs --replace=kvmfr.static_size_mb=oldvalue=newvalue"
# Default value set by us is 128 which is enough for 4k SDR
# Find the current value by running "rpm-ostree kargs"
KVMFR_MODPROBE'

# Add kvmfr modprobe file following upstream documentation
sudo bash -c "cat << KVMFR_MODPROBE > /etc/modprobe.d/kvmfr.conf
options kvmfr static_size_mb=128
KVMFR_MODPROBE"
# Add kvmfr static size karg
rpm-ostree kargs --append-if-missing="kvmfr.static_size_mb=128"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the modprobe override here seems like what we should do with bootc, but maybe it just works?

Copy link
Member Author

@HikariKnight HikariKnight Jan 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably can and the looking-glass people would prefer it over using rpm-ostree kargs which is unique to fedora atomic, i am just following the change kyle did in bazzite to add the vfio modules to the image. :)


# Add upstream udev rule for kvmfr, adjusted for fedora systems
echo "Adding udev rule for /dev/kvmfr0"
Expand Down Expand Up @@ -64,6 +69,10 @@ fi
if [ ! -d "$HOME/.config/selinux_te/pp" ]; then
mkdir -p "$HOME/.config/selinux_te/pp"
fi
if [ -f "$HOME/.config/selinux_te/kvmfr.te" ]; then
echo "Re-creating kvmfr selinux type enforcement rules"
rm $HOME/.config/selinux_te/kvmfr.te
fi
bash -c "cat << KVMFR_SELINUX > $HOME/.config/selinux_te/kvmfr.te
module kvmfr 1.0;
require {
Expand All @@ -73,6 +82,9 @@ require {
}
#============= svirt_t ==============
allow svirt_t device_t:chr_file { open read write map };

#============= virtqemud_t ==============
allow virtqemud_t device_t:chr_file { read write };
KVMFR_SELINUX"

# Tell user what type enforcement we made and how it looks like
Expand Down Expand Up @@ -100,9 +112,6 @@ echo "If you need to change it to a different value"
echo "you can do that in /etc/modprobe.d/kvmfr.conf"
echo "$(Urllink "https://looking-glass.io/docs/rc/ivshmem_kvmfr/#libvirt" "Please read official documentation for kvmfr for how to use it")"
echo ""
echo "Press OK to start the process of regenerating your initramfs, this will take a long time"
echo "and there is no good way to track progress for it, if anything is wrong it will error out."
echo "${b}NOTE: You can start using kvmfr right now without rebooting, but you will need to regenerate initramfs for it to auto load next boot.${n}"
echo "${b}NOTE: You can start using kvmfr right now without rebooting.${n}"

CONFIRM=$(Choose OK)
rpm-ostree initramfs --enable
Loading