tests: varlink interfaces should probably be fuzzed #23785
Comments
|
@mrc0mmand given that the varlink stuff can't be introspected I have to admit I'm not sure how it can be implemented automatically. I think in theory it should be possible to autodiscover method signatures and endpoints parsing the source code but it seems to be fragile. |
Having a "vfuzzer" along with I'll tie up a couple of loose ends regarding |
|
I guess the fundamental question is - which language do we want to use for the vfuzzer? Looking at https://github.com/varlink there are bindings for C, Python, Go, Rust and Java. I guess only C and Rust make sense in the end, as fuzzer in Python might be a bit slow (and I'm not sure if we want to introduce the Java/Go stack to systemd). Using Rust would probably satisfy my inner desire to finally learn the language, but given that I know barely Rust basics, it might... take a while. So the only viable option is, I guess, C? Any opinions? |
|
There's also sd-varlink in systemd itself but that would mean vfuzzer would have to be part of the systemd repo since sd-varlink isn't public yet. Not sure if this is actually a good idea, but just throwing the option out there. |
|
I took a look at libvarlink and a few seconds later varlink/libvarlink#51 was opened. The rust cli keeps throwing unsymbolized backtraces at me so I'm a bit wary of it as well.
As far as I know it's the only varlink library fuzzed on a regular basis (in fact, it was even mentioned in a paper related to fuzzing. There it was concluded that the fuzz target couldn't find a lot of issues so it couldn't be used to benchmark various fuzzing engines) so personally I think it would be useful to be able to use it outside of systemd. Having said that given that |
|
I'd like to export sd-varlink. We use it internally and I think it'd be nice to make it more widely available. |
|
Looking at #12230 (comment) I don't think it was meant to be exported in the sense that as far as I understand the idea was to let it cover systemd use cases only (without |
|
There are also issues like #20330 where it isn't clear what should happen on restarts/reloads. I ran into a bit different issue though. Since that socket is world-readable/writable my scripts managed to subscribe to that interface when |
While varlink may be relatively easy to reimplement, particularly in C it is not trivial. Just generating JSON from some structures and the other way around is a lot of work. Also the whole callback structure. So I think it'd be quite useful to export the library for programs in C or C++ and other low-level languages. |
This was discussed elsewhere but it's going to be rather involved to make sd-varlink work for anything other than C due to sd-json using compound initializers (and macros) very heavily. Those aren't even supported in C++ and will cause compilation errors. @poettering preferred to only expose sd-varlink for usage in other C programs and have other programs use libvarlink instead (instead of decoupling sd-varlink from sd-json). It should still be useful to expose it for other C stuff though |
yuwata
added
RFE 🎁
|
Here's another issue found in #22532 with half-baked vfuzzer: I haven't figured out how to automate that stuff yet unfortunately. |
After delving into the docs & stuff for a bit, it looks like it should be possible to "introspect" services to get a list of interfaces they implement via a call to However, in systemd we chose to not implement the Lines 867 to 870 in f63d1b0 which complicates things a bit :-) |
|
Also, it looks like For example, in the Sadly, the format of the description is not specified anywhere, but I guess that's as close to the introspection-like stuff as we can get for now. [0] varlink/libvarlink@7f28847 |
|
Aug 09 02:33:01 H systemd-oomd[231]: Assertion '!strstr("Failed to get cgroup %s owner uid: %m", "%m")' failed at src/oom/oomd-manager.c:86, function process_managed_oom_message(). Aborting.
Aug 09 02:33:51 H systemd[1]: systemd-oomd.service: Main process exited, code=dumped, status=6/ABRT
Aug 09 02:33:51 H systemd[1]: systemd-oomd.service: Failed with result 'core-dump'.
Aug 09 02:33:51 H systemd[1]: systemd-oomd.service: Scheduled restart job, restart counter is at 4.
Aug 09 02:33:51 H systemd[1]: Stopped systemd-oomd.service.As far as I can tell it's just 63275a7 in action though. |
Fixes systemd/systemd#23785 (comment). (cherry picked from commit b6f6df4) (cherry picked from commit a3348ba)
I fuzz I'll go ahead and close this issue because my understanding is that it's totally fine for systemd to crash/overflow when it receives data through world-writable sockets so it doesn't make much sense to test it probably. |
|
@poettering just out of curiosity looking at the recently announced bug bounty program I wonder if it would be possible for STF to help with this? In terms of "Bug Resilience" it would make much sense to cover the varlink interfaces and prevent issues like #22480 in the first place instead. Either way I'll go ahead and reopen this. |
|
On a somewhat related note the integration test improvements are going to break my downstream networkd/resolved fuzzers (as discussed in #32540 (comment)) and I'm unlikely to resurrect them when they break (or, more precisely, it's not a priority to me) so if STF could help with covering those parts too it would be great too because the upstream fuzz targets are kind of sloppy :-) |
|
I took a closer look at https://www.sovereigntechfund.de/programs/bug-resilience and it appears STF can do that since they work with the OSTIF folks (who have the expertise in those things and can improve the fuzzing infrastructure and stuff like that). I'm not sure who STF talked to and why it was decided to roll out the bug bounty program instead of covering the basics. It's kind of weird. |
|
@evverx Any chance you could make your downstream fuzzers public somewhere? I'm happy to take a look at upstreaming them into the mkosi stuff. |
|
@DaanDeMeyer unfortunately that stuff can't be open sourced. |
|
On the bright side https://nvd.nist.gov/vuln/detail/CVE-2024-5564 kind of came out of all this and was fixed because I switched the fuzzer to NetworkManager at some point. I'm repurposing the mkosi stuff to keep it going too. @DaanDeMeyer I have to say I'm not planning to fuzz systemd-networkd though. |
Thanks! Can you provide a bit more information on the backtrace issue? Is this just #33206 or are they messed up locally as well? |
|
I'm 99% sure that I fiddled with CFLAGS too much and forgot to pass something to a bunch of additional packages I install into images. Configs aren't in the right places either :-) Previously I assembled those images a bit differently.
I think #33206 would be nice but the part extracting backtraces/coredumps and so on on my side kind of works once containers are stopped so it doesn't prevent me from doing whatever it is I'm doing. |
|
At least for the systemd packages we build with mkosi the sources have to be mounted into the image to get the best possible backtraces (with |
|
@DaanDeMeyer I wonder if there is a recommended way to bisect stuff using mkosi? The idea is to figure out when bugs are introduced/fixed by analogy with, say, OSS-Fuzz showing commit ranges. With the bash framework |
|
@evverx I hope that within a year or so this setup will have sufficiently stabilized that you can just run something like |
|
@DaanDeMeyer got it. I don't run the integration tests though.
I think I bisected one bug by rebuilding stuff inside containers that were up and running and reinstalling it over and over again there and rebooting it :-) |
|
@evverx It doesn't have to be an integration test of course, you can run mkosi without meson to try to reproduce some issue. The approach for that is to write a systemd unit that reproduces the issue and put it inside the image via the [Unit]
Wants=basic.target
After=basic.target
SuccessAction=exit
FailureAction=exit
[Service]
StandardOutput=journal+consoleOf course you'll need to add whatever settings here required to reproduce the issue and also include a script with mkosi.extra to use in You can then invoke |
|
@DaanDeMeyer got it. Thanks. My use case is a bit different in that I only need images with certain snapshots. Crashes, reproducers and things like that are handled outside. |
|
I'll go ahead and close this because I don't think the varlink interfaces are ever going to be fuzzed upstream. (The D-Bus interfaces are unlikely to be fuzzed better upstream either) @DaanDeMeyer I'm OK with dropping the bash framework because it should no longer affect my stuff any more. Not that it matters much though. (It would be great if changes like that were communicated going forward. I think mkosi is cool stuff, the testing part got better in the end and I started using it in places where it covers my use cases but it all was totally unexpected to me) |
By analogy with
dfuzzerfuzzing DBus interfaces I think there should bevfuzzerfuzzing the varlink stuff. It should help to catch issues like #22480 and somewhat cover PRs like #22845 (where malformed varlink messages kick out all listeners) and #23750 (which introduces a memory leak at least)The text was updated successfully, but these errors were encountered: