[native_toolchain_rust] Introduce a package:native_toolchain_rust

[dcharkes]

Edit: being developed on https://pub.dev/packages/native_toolchain_rust.

We should consider introducing a package to help calling the Rust compiler from Dart code.

Things this package could contain:

• A mapping from native_assets_clis Target and IOSSdk to Rust target triples (These seem to deviate from other toolchains. Apple clang: arm64-apple-ios-simulator vs Rust aarch64-apple-ios-sim.
• (Feel free to edit and add wishes.)
• ...

This package could be used in multiple ways:

1. Running the Rust compiler from the user machine inside build.dart scripts.
2. Running the Rust compiler from some tools/compile_all.dart script on the GitHub CI (after which the artifacts can be downloaded in a build.dart script).

cc @fzyzcjy @robertbastian Please post any wishes you have here.

[fzyzcjy]

Wow this is super great and helpful for flutter_rust_bridge, and looking forward to it! Option 1 is already very good, since it allows quite easy integration; and option 2 makes it even greater because we can now create a package with precompiled binaries easily, then users of packages do not need to spend time compiling. Some thoughts (I guess most are already in your mind, but anyway let me post it for completeness):

• Support Web as well (related: Build hook for Flutter Web? flutter/flutter#138992. flutter_rust_bridge currently uses this script to compile to web: https://github.com/fzyzcjy/flutter_rust_bridge/blob/562f8db130eec6d27d20f3e6d60e89f3d875997f/frb_dart/lib/src/cli/build_web/executor.dart#L53, mainly wasm-pack, which combines multiple tools to compile rust code into wasm files.)
• Cargokit only requires user to install a rustup, and automatically install cargo etc when needed, thus users do not need to install too many things. I guess this is helpful for users. (Since flutter_rust_bridge is currently using https://github.com/irondash/cargokit as one of the approaches of integration, some features in cargokit may be helpful.)
• Support dynamic libraries (.so/.dylib etc), since flutter's default suggestions are to use dynamic libraries, and static libraries may have things like symbol conflicts.
• Understand debug/profile/release mode automatically.
• Allow users to (temporarily) disable some target architectures for faster development. For example, when compiling (full) Android, IIRC we need things like 32bit arm, 64bit arm, 32bit x86, 64bit x86. So, when developing Rust, even if we only change 1 line of code, we need to execute cargo build four times. It would be great if, when developing, we can disable all architecture but only one (say 64bit x86) that is really needed.
• (Low priority) Understand MSRV (e.g. say flutter_rust_bridge requires rust 1.74, then if users have rust 1.60, toolchain needs to automatically install a latest version; but this is of low priority, since an error is already descriptive enough).
• ...

[knopp]

@fzyzcjy, I'm currently working on native_toolchain_rust (as a replacement for cargokit). Unlike cargokit however, it will not silently install thing during build. I think there is a cleaner approach to that: A separate tool - native_doctor that does all the installing and updates.

When native_toolchain_rust finds out that rustup is not installed, or some toolchains/targets are missing or outdated, it will fail the build with following (or similar) message:

Rustup not found.
Please run native_doctor in your project to fix the issue:

dart pub global activate native_doctor
dart pub global run native_doctor

When user runs native_doctor, it will check project dependencies to determine minimum required Rust and NDK version and outputs something like this:

Project: example (Flutter)
Buildable platforms: macos, ios, android

Native toolchain: NDK

  [✗] NDK installed, but too old
       ! Installed versions: 23.1.7779620
       ! Required minimum version: 25.1.8937393

Native toolchain: Rust

  [✗] Rustup not found
  [✗] Toolchain stable not installed
       ! Required minimum version: 1.77.2
       ! Missing targets: arm-linux-androideabi, aarch64-linux-android, i686-linux-android,
         x86_64-linux-android, aarch64-apple-ios, x86_64-apple-ios, aarch64-apple-ios-sim, 
         aarch64-apple-darwin, x86_64-apple-darwin

Proposed actions:

  • (NDK)  Install NDK 25.1.8937393 or newer
  • (Rust) Install rustup
  • (Rust) Install toolchain stable
  • (Rust) Install targets arm-linux-androideabi, aarch64-linux-android, i686-linux-android,
           x86_64-linux-android, aarch64-apple-ios, x86_64-apple-ios, aarch64-apple-ios-sim,
           aarch64-apple-darwin, x86_64-apple-darwin for toolchain stable

Do you want native doctor to perform proposed actions? (y/N)

When user confirms it, it will automatically install all required items.

@dcharkes, I am wondering if down the line, instead of separate tool, this could be somehow integrated with native assets.

[dcharkes]

@dcharkes, I am wondering if down the line, instead of separate tool, this could be somehow integrated with native assets.

That sounds amazing! Yes, maybe we can pull it in to Dart and Flutter.

We've been discussing:

1. dart doctor in similar style to flutter doctor
2. Adding the NDK to flutter doctor

Building this logic as a separate package native_doctor sounds great! For the time being users could do dart pub global run native_doctor. Oh you already did! 🎉

Once native_doctor is stable and working well, I can have a chat with some peeps on the team here whether it makes sense to have it in the SDKs. 👍 flutter native_tools? flutter doctor (but that doesn't install anything currently.)

Thanks for the great ideas and contributions @knopp and @fzyzcjy! 🚀

[knopp]

@dcharkes, I'd maybe like some feedback on native_toolchain_rust. I think it's in a decent shape for MVP, the build tests pass on all platforms (including android on all platforms). There are examples for both dart and flutter packages. It provides actionable message when the environment is not ready (and native_doctor tool to fix it). What would be required to publish it on pub? And somewhat related, is there even a very rough estimation for when native assets get enabled in Flutter main (and Dart dev) without the flag?

[mosuem]

Happy to take a look, what's the best course of action for that in Github? Creating a (draft) pull request for the native repository?

[dcharkes]

I'm happy to let the project be in its own repo. (Unless there are some benefits to having it in this mono repo.)

Maybe just a PR to another branch on the existing repo?

[mosuem]

Yes, the PR would be for reviewing only. Github doesn't allow adding comments on non-changed files in PRs, so I thought that might be the only way to review..

[knopp]

I don't see why it would need to live in the mono-repo.

I'm not sure if I understand things correctly, but here is a PR with code for all packages.

[dcharkes]

You need to be an admin in both publishers to do a transfer.
They can invite you as admin, you do the transfer and then they can remove you from being admin.
Alternatively you may create a third publishers where only you two are admin and it acts only as intermediary.

Can you invite [email protected] to a publisher @knopp so I can transfer the package to you on pub.dev?

[fzyzcjy]

[...] Please post any wishes you have here.

Another wish may be hot restart / hot reload Rust (related e.g. fzyzcjy/flutter_rust_bridge#1966). Here are my brainstorms:

• If using the old DynamicLibrary.open approach, I guess we only need to figure out an approach to point to the updated Rust binary file after Rust recompiles, thus maybe related to Cargokit. However, this is the old way and I guess supporting new native assets would be great.
• If using the new native assets approach, have not checked it deeply yet, but I guess maybe Dart needs to provide some kind of support for this hot restarting. For example, for a @Native<Int64 Function(Int64, Int64)>() external int sum(int a, int b);, it seems that the hack cannot be as easy as "just DynamicLibrary.open another binary file and done".

[sneurlax]

@dcharkes is there any indication as to when native assets will be activated in a Flutter or Dart release without the flag?

edit: sorry, I just saw you were working on https://github.com/dart-lang/sdk/milestone/112 as of just last week; sorry to've pinged you, please launch on 🚀

[dcharkes]

@dcharkes is there any indication as to when native assets will be activated in a Flutter or Dart release without the flag?

edit: sorry, I just saw you were working on https://github.com/dart-lang/sdk/milestone/112 as of just last week; sorry to've pinged you, please launch on 🚀

We've got another milestone on this repo as well https://github.com/dart-lang/native/milestone/15. (And some Flutter issues that are not tracked on a milestone as it is in another org...)

Also, this issue itself is being worked on via https://pub.dev/packages/native_toolchain_rust, so I'll close this.