Add macOS support to OpenZFS #12110
Open
Add macOS support to OpenZFS #12110
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lundman
force-pushed
the
macOS
branch
2 times, most recently
from
5e2d62d to
65a74e3
Compare
lundman
force-pushed
the
macOS
branch
2 times, most recently
from
4277da2 to
162912a
Compare
lundman
force-pushed
the
macOS
branch
6 times, most recently
from
ef3b4c8 to
e7b6e73
Compare
lundman
force-pushed
the
macOS
branch
3 times, most recently
from
c7316f5 to
ade3311
Compare
lundman
force-pushed
the
macOS
branch
12 times, most recently
from
84848e9 to
96f9ab0
Compare
lundman
force-pushed
the
macOS
branch
2 times, most recently
from
b401d59 to
7780bfd
Compare
|
This PR has been updated up to Sonoma (Apple clang version 15.0.0 (clang-1500.0.34.3)), x86_64 and arm64. |
lundman
force-pushed
the
macOS
branch
5 times, most recently
from
0d075ae to
bc3f497
Compare
|
@behlendorf @lundman @mcmilk I remember that we were waiting until the next release to look at this. Since 2.2 has released is there a chance of getting this reviewed for merging? I also remember someone brought up the ideas of
|
|
It would be nice when this PR gets into current master. I would help afterwards with the unified |
mcmilk
suggested changes
Oct 22, 2023
There was a problem hiding this comment.
We need to split this big commit into smaller parts.
At least ito these 4 commits:
part1: module/os/macos/* + include/os/macos/*
part2: module/*
part3: cmd/* + tests/*
part4: config/, scripts/, contrib/* and all other things
-> If you want, I can also create an example of that.
It shouldn't interfare the current CI, so that testing other PRs is working...
Maybe @behlendorf has also some ideas what needs to be done before merging.
|
Couple of bonus ones in there with 5 and 6. Let me know if you want them squashed into 1-4 somewhere. |
|
@lundman I tried compiling everything without the macos code (2-6) so if this PR ends up getting split into multiple PRs that technically counts as shared code |
|
Yep, all Makefiles are now read in all at once - something upstream did a couple of years back, and some of those Makefiles do various things, depending on .in files etc. The Makefile changes have to be last and in one go. |
yurikoles
reviewed
Nov 5, 2023
.github/workflows/macos-build.yaml
Outdated
| ./configure CPPFLAGS="-I/usr/local/opt/gettext/include -I/usr/local/opt/openssl/include" LDFLAGS="-L/usr/local/opt/gettext/lib/ -L/usr/local/opt/openssl/lib" | ||
| - name: build | ||
| run: | | ||
| make -j 2 |
There was a problem hiding this comment.
The default GitHub-hosted macOS runner has 3 vCPUs, but you are limiting make to only 2 jobs.
There was a problem hiding this comment.
This then?
make -j`nproc`Or
make -j$((`nproc`+1))There was a problem hiding this comment.
nproc is part of GNU Core Utilities, but macOS ships with CLI tools from BSD. A native macOS analogue is sysctl -n hw.ncpu, there is also a hw.physicalcpu and a hw.logicalcpu for hyper-threading.
There was a problem hiding this comment.
make -j `sysctl -n hw.ncpu`
make -j $((`sysctl -n hw.ncpu`+1))
|
I'm trying to get smaller PRs merged: |
part1: module/os/macos/* + include/os/macos/* This does then inlcude lib/*macos* as well, using grep so there are outliers, like manpage. Signed-off-by: Jorgen Lundman <[email protected]>
part2: module/* Signed-off-by: Jorgen Lundman <[email protected]>
part3: cmd/* + tests/* Signed-off-by: Jorgen Lundman <[email protected]>
part4: config/, scripts/, contrib/* Signed-off-by: Jorgen Lundman <[email protected]>
Bonus part5, the non-macos changes to include lib and the top level Makefiles Signed-off-by: Jorgen Lundman <[email protected]>
Signed-off-by: Jorgen Lundman <[email protected]> Set zp->z_blksz to ashift Slowdowns in zfs_log_write() can happen on large blocksize devices. Signed-off-by: Jorgen Lundman <[email protected]> MacOS: Build with n+1 cpus (#22) Signed-off-by: Andrew Innes <[email protected]> macOS: use portable paths for brewed deps (#23) Signed-off-by: Yurii Kolesnykov <[email protected]> Search for C++ and Obj C compilers (#21) Signed-off-by: Andrew Innes <[email protected]> cpuid uses a,b,c,d order. wikipedia paragraph referred to model string, which has a special case ordering. This fixes a bug where old intel would incorretly assume AES/qmul feature was available when it was not. (And vice-verse, but this would just skip asm versions and not panic) Signed-off-by: Jorgen Lundman <[email protected]> Redo cpuid feature detection We already have a proper cpuid feature API used in userland, but Linux took it out for KERNEL use, to plug into the Linux API. We do not have a kernel API to use, so we might as well use the userland one. This makes it nice and consistent, and more easy to read. Signed-off-by: Jorgen Lundman <[email protected]> Use built in cpuid() where possible Fixing a panic when running sysctl -a Signed-off-by: Jorgen Lundman <[email protected]> UpstreamÂ: remove sprintf usage Signed-off-by: Jorgen Lundman <[email protected]> Add IRC notifications Signed-off-by: Jorgen Lundman <[email protected]> xcode 16 compile fixes function argument type changed, adapt ASSERTs assembly bug in clang-16 work around Signed-off-by: Jorgen Lundman <[email protected]> Reduce userland CPU starvation from low-priority I/Os Once we have successfully read from disk we may do significant CPU-work on the data obtained, depending on features like record sizes, checksums, encryption, or compression. A successful write may cause significant CPU-work to be done for a subsequent zio. Because our vdev_disk layer and its use of IOKit is fundamentally asynchronous, on some media modern linearized scrubs and resilvers may "gang up" on bursts of interactive user I/Os. Moreover, all zfs kernel threads are higher priority than the vast majority of userland threads, therefore the latter can be starved of CPU especially for a scrubbing pool which has a vdev count conmparable to the CPU core count and where data was wrtten using expensive checksums like sha256. Practically all our IOKit I/Os are asynchronous, but significant work may be done on the taskq thread, possibly right to the entry into the vdev_disk_io_intr() callback function. We dispatch "background" I/Os into a lower thread-priority and lower thread-count taskq compared to other types of zio. In the callback function itself, for these low-priority I/Os we kpreempt() before before calling zio_delay_interrupt(). For writes, this may impose a system-load-dependent delay on notifying upper layers of zfs that IOKit has moved the buffer towards the physical device, generating backpressure on subsequent writes. For reads, this kpreempt() gives another thread in the system a chance to run before we do potentially heavy-CPU actions (such as checksumming or decyrption) on the data IOKit has obtained from the storage device. Signed-off-by: Jorgen Lundman <[email protected]> iconfig fixes Signed-off-by: Jorgen Lundman <[email protected]> abd_os changes Signed-off-by: Jorgen Lundman <[email protected]> New ASSERTs Signed-off-by: Jorgen Lundman <[email protected]> new KMEM flag Signed-off-by: Jorgen Lundman <[email protected]> issig() now takes no args Signed-off-by: Jorgen Lundman <[email protected]> sprintf is deprecated Signed-off-by: Jorgen Lundman <[email protected]> Add UIO_DIRECT Signed-off-by: Jorgen Lundman <[email protected]> Re-arrange z_blksz in zfs_znode_alloc() Signed-off-by: Jorgen Lundman <[email protected]> Set zp->z_blksz to ashift Slowdowns in zfs_log_write() can happen on large blocksize devices. Signed-off-by: Jorgen Lundman <[email protected]> altool is deprecated, use notarytool Signed-off-by: Jorgen Lundman <[email protected]> pkg-macos OS friendly name after macOS name change Signed-off-by: Jorgen Lundman <[email protected]> Update the pkg installer scripts Signed-off-by: Jorgen Lundman <[email protected]> zfs-tests: support newer macOS readOnly root Since BigSur days, "/" on macOS is a readOnly, sealed archive which means zfs-tests can not mount the expected /testpool (which all scripts assume are at root). We can work around this by leveraging /etc/synthetic.conf to create "virtual directories". They can be mounted on, but not modified. This way *most* pool names are accounted for in zfs-tests (more will come). We also have to make log_must(rm /TESTPOOL) be optional. Signed-off-by: Jorgen Lundman <[email protected]> Revert "Re-arrange z_blksz in zfs_znode_alloc()" This reverts commit 106e765. See next commit Signed-off-by: Jorgen Lundman <[email protected]> Change zfs_log_write() blocksize for performance "len" can be set to zero in the WR_INDIRECT case, if zp->z_blksz is 0. This leads to endless loop. Signed-off-by: Jorgen Lundman <[email protected]> Revert changes to zfs_znode_alloc for reals. Signed-off-by: Jorgen Lundman <[email protected]> Make sure blocksize is not 0 in zfs_log_write() Or the while(resid) loop will run forever. Signed-off-by: Jorgen Lundman <[email protected]> Negative numcpus make taskq stall Handle limited CPU VMs with 1 or 2 cores. The cpus variable would go to -1 and taskq would stall forever. Signed-off-by: Jorgen Lundman <[email protected]> Handle versions without IOSleepWithLeeway() Signed-off-by: Jorgen Lundman <[email protected]> Allow zfs_fallthrough to be defined Signed-off-by: Jorgen Lundman <[email protected]> Don't use typedefs before OSVersion workaround Signed-off-by: Jorgen Lundman <[email protected]> Handle SF_NOUNLINK on versions without it Signed-off-by: Jorgen Lundman <[email protected]> Update pkg_macos.sh to handle notarytool Signed-off-by: Jorgen Lundman <[email protected]> Compile fixes after rebase Signed-off-by: Jorgen Lundman <[email protected]> Export before removing file Due to a macOS quirk, we have to close the files in a filebased pool during some operations, which means the export -f task will fail to open the vdevs again, the pool will be suspended and zpool_export_004_pos will hang forever. By deleting the file after export, everything works as expected. Signed-off-by: Jorgen Lundman <[email protected]> Don't hold zfs_enter() in asyncput This can deadlock during unmount, as we are already holding WRITE lock. Signed-off-by: Jorgen Lundman <[email protected]> zpool freeze can hang in spa_evicting_os_wait It appears upstream can still unmount, and export the pool - to later re-import to unfreeze. However on macOS, export will hang in spa_evicting_os_wait(). It is unclear why we hang, but it could be we go through more syncs during unmount, and that spa_freeze_txg is set to txg + TXG_SIZE Now we clear (restoring UINT64_MAX) spa_freeze_txg in the unmount() call, ensuring unmount and export works. We might not have identical "zpool freeze" as upstream, but as the zpool sources say: * 'freeze' is a vile debugging abomination, so we treat * it as such. Signed-off-by: Jorgen Lundman <[email protected]> Do not install .in files with Makefile Signed-off-by: Jorgen Lundman <[email protected]> clone file can return ENOTSUP for fallback From Sonoma onwards, file_cmds-428 will handle ENOTSUP and fallback to copyfile. Prior to that, we favour EAGAIN as it will indicate to the users that it is a temporary failure. Signed-off-by: Jorgen Lundman <[email protected]> sprintf has been deprecated Signed-off-by: Jorgen Lundman <[email protected]> Move appveyor to .github/workflows/macos-build.yml Signed-off-by: Jorgen Lundman <[email protected]> make install would fail to mkdir first. Signed-off-by: Jorgen Lundman <[email protected]> Fix .yml to start zed, and zfs-tests Signed-off-by: Jorgen Lundman <[email protected]> fix abd, have taskq_wait_synced() wait for threads to be created taskq_wait_synced() did a VERIFY() on whether the taskq's threads were the requested number, but taskq_create() can ultimately return early because taskq_thread_create() is allowed to return when two desired threads are created. fix this race panic. also, taskq_wait_synced() may fail if if num_ecores is nonzero (on Apple Silicon), so create a flag that lets taskq_create_common() deal with the max_ncpus. Make boot_ncpus a variable that's MAX(1, (int)max_ncores - num_ecores). boot_ncpus is used in common code. Modify the alignments and quanta/import sizes of the abd kmem and vmem cache creations. Make DEBUG builds work with KMF_LITE | KMF_BUFCTL on the abd kmem caches. Signed-off-by: Sean Doran <[email protected]> Allow NOTARYTOOL path to be set Apple has decided that copying notarytool from new macOS to old macOS instead of supporting altool. But sadly "xcrun" will not run it as expected, so we allow NOTARYTOOL env var to be set to a path to use. Signed-off-by: Jorgen Lundman <[email protected]> Add copy_file_range() wrapper to fcopyfile(). Signed-off-by: Jorgen Lundman <[email protected]> Compile fixes since last rebase Signed-off-by: Jorgen Lundman <[email protected]> ZIO_TYPE_IOCTL renamed to ZIO_TYPE_FLUSH And also they renamed zfs_file_fallocate() to zfs_file_deallocate() somewhere along the way. Signed-off-by: Jorgen Lundman <[email protected]> Header fix Signed-off-by: Jorgen Lundman <[email protected]> zfs_racct prototype change Signed-off-by: Jorgen Lundman <[email protected]> zfs_log_write() now takes directio boolean Signed-off-by: Jorgen Lundman <[email protected]> zfs_znode_os.c also, change strcpy back to strlcpy yet again, thanks linux. Signed-off-by: Jorgen Lundman <[email protected]> isimd_stat_init simd_stat_fini Signed-off-by: Jorgen Lundman <[email protected]> Signed-off-by: Jorgen Lundman <[email protected]> Use SUBSTFILES for .in files Signed-off-by: Jorgen Lundman <[email protected]> Compile fixes for O_DIRECTIO Signed-off-by: Jorgen Lundman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Add macOS support.
Yes, one giant commit - but don't panic.
At first; the easy, independent, changes were taken out, and individually PRed against ZOL. Now we've reached a point where there is not so many of those opportunities left.
If you spot a change you would prefer to be PRed separately, please let me know.
What's in the giant commit?
Changes in tests/ have been left out, and will be in some future PR. Current zfs-tests pass-rate is about 50%. This is mostly due to wrong Unix tools used etc, as opposed to failure to perform ZFS tasks.
Interesting points of ... interests...
mount_zfs, some rename work added to macos/mount_zfs. cmd/os/linux/mount_zfs ?Description
This adds support for macOS 10.9 to macOS 12 including intel and arm64.
How Has This Been Tested?
macOS zfs-tester and numerous unfortunate users.
Types of changes
Checklist:
Signed-off-by.