zinject: add "probe" device injection type

[robn]

[Sponsors: Klara, Inc., Wasabi Technology, Inc.]

Motivation and Context

zinject can't create the scenario of a drive faulting after a failed write. The write failure can be injected, which triggers a probe, but probe IO always goes to the label regions, which are explicitly excluded from device injections. So the probe succeeds, and the write is retried, triggering another probe, over and over, forever.

This adds a new probe pseudo-IO-type for device injection, which allows a probe failure to be simulated.

Description

First, we extend the device injection type to allow more than just the "fundamental" IO types. zi_iotype was always large enough, there just wasn't a way to do more than the regular ZIO_TYPE_* values. Now it has its own values, ZINJECT_IOTYPE_* for injection IO types. For compatibility, the existing IO types are placed at the start so their numeric values match. all still matches all fundamental types, so this change should be UI and ABI-compatible.

Then, we add a new type, ZINJECT_IOTYPE_PROBE. This matches any and all ZIOs with the ZIO_FLAG_PROBE flag set, regardless of type or location on disk (including in the labels). The normal _READ and _WRITE injections won't match these, to preserve the existing behaviour.

Taken together, this we can simulate a device fault on write with:

$ zinject -d $<devcie> -e io -T write <pool>
$ zinject -d $<device> -e io -T probe <pool>

And indeed, this is exactly what the included test does.

How Has This Been Tested?

New test included.

All zinject-using tests have passed with this change in place.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

(Wow! Clean sweep!)

[tonyhutter]

@robn when you're satisfied with your changes, would you mind squashing these commits?

[robn]

@tonyhutter if/when #16950 lands, I'll rebase and then it will be just two logically-separate commits.

[amotin]

Have no objections once #16950 is merged.

[amotin]

zinject_probe test timed out on FreeBSD, possibly causing more errors after.

[robn]

zinject_probe test timed out on FreeBSD, possibly causing more errors after.

I'm guessing the max 10s wait for suspend is too short. Bumped it to max 30s, why not.

[amotin]

zinject_probe test timed out on FreeBSD, possibly causing more errors after.

I'm guessing the max 10s wait for suspend is too short. Bumped it to max 30s, why not.

I meant the CI timed out after 10 minutes. Something might be wrong with the test, so that it never completed.

[robn]

I meant the CI timed out after 10 minutes. Something might be wrong with the test, so that it never completed.

I'm glad you pushed; it made me take another look at it, and took a little while to figure out (at least, I think I figured it out).

Test: /usr/local/share/zfs/zfs-tests/tests/functional/cli_root/zinject/zinject_probe (run as root) [10:00] [KILLED]
07:49:48.31 ASSERTION: Check zinject can correctly inject a probe failure.
07:49:48.43 Added handler 14 with the following properties:
07:49:48.43   pool: testpool
07:49:48.43   vdev: c1b926c8812118f1
07:49:48.44 SUCCESS: zinject -d md0 -e io -T probe testpool
07:49:48.44 Added handler 15 with the following properties:
07:49:48.44   pool: testpool
07:49:48.44   vdev: c1b926c8812118f1
07:49:48.45 SUCCESS: zinject -d md0 -e io -T write testpool

ZTS run /usr/local/share/zfs/zfs-tests/tests/functional/cli_root/zinject/zinject_probe
Solaris: WARNING: Pool 'testpool' has encountered an uncorrectable I/O failure and has been suspended.
Jan 16 07:49:48 vm2 ZFS[32666]: vdev probe failure, zpool=testpool path=/dev/md0
Jan 16 07:49:48 vm2 ZFS[32742]: catastrophic pool I/O failure, zpool=testpool

Two observations:

• dd never finished, or it would have logged a SUCCESS line
• the probe failure is within half a second of the injections being added

So probably, it suspended before the dd could complete; maybe before it started.

Last push tries a lot harder to control the timing, by pushing out the txg timeout, syncing the pool, doing the write (to cache) before setting the injections, and then forcing it out. It tests ok here; we'll see if it ends up being flaky or not.