[WIP] SVM: Integration of SolFuzz-Agave

[buffalojoec]

NOTE: This change is intended for the new SVM repository, not the Agave monorepo.
Its draft inclusion here is only for review & collaboration purposes.

Problem

Both the Anza and Firedancer teams have been using SolFuzz-Agave
for crucial testing of validator components as well as BPF programs. It's also a
potentially very useful tool for other developers in each domain.

There are actually multiple problems being addressed by these changes - which
include only the addition of new libraries and no changes to Agave's hot
path.

First and foremost, the Firedancer team maintains SolFuzz-Agave standalone, and
whenever changes are made to Agave's execution layer, they must cherry-pick and
update the harness to be compatible with the new APIs. It would be much easier
if we maintained changes to SolFuzz-Agave alongside any API-breaking PRs to Agave.

Secondly, Anza's developer tooling team has been working on both migrating
builtins to BPF programs as well as rewrites of SPL Token, SPL Token-2022 and
more. We use SolFuzz-Agave and Mollusk
to test BPF programs in many different ways, including conformance testing
between two versions of the same program.

However, the same internal entrypoint into Agave's program runtime is
reimplemented across both tools. It would be much easier - and safer - if this
entrypoint lived in one place, and was instead used by both tools. This would
also mean that it could be used for other future tooling, which would be a huge
benefit.

https://github.com/buffalojoec/mollusk/blob/c01523016195c315870bd9e4bfa3a64a6cffa659/harness/src/lib.rs#L182-L218

The above problems are the motivation for both the integration of SolFuzz-Agave
itself, as well as the structure of the libraries included in this PR.

Including SolFuzz-Agave in Agave's SVM maintenance allows a much more stable API
for testing and fuzzing validator execution-layer components, which helps
Firedancer as well as any new validator clients who wish to benefit from the
same tooling. It also allows us to build more robust BPF program tooling, which
can help the broader ecosystem in obvious ways.

Summary of Changes

The overall change is straightforward: integrate SolFuzz-Agave into the SVM
stack by creating a new set of standalone crates that comprise a
reimplementation of the original SolFuzz-Agave.

I've refactored a lot of the original code to make it easier to maintain and
change, and I've included testing from the original repository to ensure maximum
compatibility.

NOTE: This change is only handling the instr (instruction) entrypoint. The
others (transaction, VM) will come later, and follow a similar pattern.

I've also decided to break the tooling up into multiple crates. They are
summarized as follows:

• -fixture: The library for working with fixtures created by Firedancer's
  Protosol protobuf definitions. It
  offers conversions between prost-generated protobuf types and Agave SDK
  types.
• -fixture-fs: An extension crate for working with fixtures (above) on the
  local filesytem (ie. load_from_blob_file).
• -instr-entrypoint: The main entrypoint into Agave's program runtime. This
  library has nothing to do with fixtures, and can be used standalone to build
  tooling (ie. Mollusk).
• -instr: The actual SolFuzz-Agave instruction harness itself, which depends
  on the above libraries to offer the same fuzzing and testing entrypoint as the
  original tool, including any custom setup or checks that are not imposed on
  the base entrypoint.

[LucasSte]

Thank you so much for organizing this.

Is this working for all instruction test vectors? If so, we can delete my conformance code from here.

Are you planning to add conformance testing for transactions as well?

[buffalojoec]

Not working for all just yet. Over 20,000 are passing, and about 140 are failing still. I think I'm just missing one or two very small things.

Will push an update and comment here when it's finally 100%.

Are you planning to add conformance testing for transactions as well?

Yep, we will add the rest of the harness and then this library can be used to build out the coverage for SVM from VM up to tx.

[mjain-jump]

This is awesome @buffalojoec thank you so much for this. Please let us know if you run into any issues with some of the failing tests. It would also be great to define a spec for this so that other validator clients can implement their own harnesses as well.

[buffalojoec]

@mjain-jump @LucasSte This PR over on my fork is the same commits but rebased on top of v2.1.6, since that's the latest Agave version supported by SolFuzz-Agave.

buffalojoec#32

As of right now, all instruction fixtures pass except for one BPF Loader v2 fixture, and I'm wondering if that's because I haven't done the FD stubs yet. @mjain-jump let me know what you think, the fixture is listed in the last commit.