Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Demonstrating Native Addons #2

Merged
merged 9 commits into from
May 22, 2022
Merged

Demonstrating Native Addons #2

merged 9 commits into from
May 22, 2022

Conversation

CMCDragonkai
Copy link
Member

@CMCDragonkai CMCDragonkai commented May 21, 2022

Description

In helping solve the snapshot isolation problem in MatrixAI/js-db#18, we needed to lift the hood and go into the C++ level of nodejs.

To do this, I need to have a demonstration of how native addons can be done in our demo lib here.

There are 2 ecosystems for building native addons:

  • prebuild
  • node-pre-gyp

Of the 2, the prebuild ecosystem is used by UTP and leveldb. So we will continue using that. Advantages from 2016 was commented here: prebuild/prebuild#159

The basic idea is that Node supports a "NAPI" system that enables node applications to call into C++. So it's a the FFI system of NodeJS. It's also a bidirectional FFI as C++ code can call back into the NodeJS JS functions.

The core library is node-gyp. In the prebuild ecosystem is wrapped with node-gyp-build, which you'll notice is the one that we already using in this repo. The main feature here is the ability to supply prebuilt binaries instead of expecting the end-user to always compile from source.

Further details here: https://nodejs.github.io/node-addon-examples/build-tools/prebuild (it also compares it to node-pre-gyp).

The node-gyp-build has to be a dependency, not devDependencies, because it is used during runtime to automatically find the built shared-object/dynamic library and to load it.

It looks like this:

import nodeGypBuild from 'node-gyp-build';
const bindings = nodeGypBuild('./path/to/dir/containing/gyp/file');
bindings.someNativeFunction()

Internally nodeGypBuild ends up calling the require() function inside NodeJS. Which supports the ability to load *.node binaries (which is the shared-object that is compiled using the NAPI C++ headers). See: https://github.com/prebuild/node-gyp-build/blob/2e982977240368f8baed3975a0f3b048999af40e/index.js#L6

The require is supplied by the NodeJS runtime. If you execute the JS with a different runtime, they may support the commonjs standard, and thus understand the require calls, but they may be compatible with native modules that are compiled with NAPI headers. This is relevant since, you also have to load the binary that matches your OS libraries and CPU architecture. It's all dynamic linking under the hood. This is also why you use node-gyp-build which automates some of this lookup procedure.

As a side-note about bundlers. Bundlers are often used part of the build process that targets web-platforms. Since the web platform does not understand require calls, bundlers will perform some sort of transclusion. This is also the case when ES6 import targets files on disk. Details on this process is here: https://github.com/evanw/esbuild/blob/master/docs/architecture.md#notes-about-linking. Bundlers will often call this "linking", and when targetting web-platforms, this is basically a form of static linking since JS running in browsers cannot load JS files from disk. This is also why in some cases, one should replace native addons with WASM instead, as bundlers can support static linking of WASM (which are cross-platform) into a web-bundle. But some native addons depend on OS features (like databases with persistence), and fundamentally cannot be converted into WASM binaries. In the future, our crypto code would make sense to turn into WASM binaries. But DB code is likely to always be native, as they have to be persistent. As the web develops can gains extra features, then eventually it may be possible that all native code can be done via WASM (but this may be a few years off).

Now the native module itself is just done with a C++ file like index.cpp. We should prefer using .cpp and .h as the most portable extensions.

Additionally, there must be binding.gyp file that looks like this:

{
  "targets": [{
    "target_name": "somename",
    "include_dirs": [
      "<!(node -e \"require('napi-macros')\")"
    ],
    "sources": [ "./index.cpp" ]
  }]
}

Basically another configuration file that configures node-gyp and how it should be compiling the C++ code. The target_name specifies the name of the addon file, so the output result will be somename.node. The sources are self-explanatory. The include_dirs entries have the ability to execute shell commands, in this case, it is using node -e to execute a script that will return some string that is a path to C++ headers that will be included during compilation.

The C++ code needs to use the NAPI headers, however there's a macro library that makes writing NAPI addons easier: https://github.com/hyperdivision/napi-macros. I've seen this used in the utp-native and classic-level.

The C++ code may look like this:

#include <node_api.h>
#include <napi-macros.h>

NAPI_METHOD(times_two) {
  NAPI_ARGV(1)
  NAPI_ARGV_INT32(number, 0)

  number *= 2;

  NAPI_RETURN_INT32(number)
}

NAPI_INIT() {
  NAPI_EXPORT_FUNCTION(times_two)
}

This ends up exporting a native module containing the times_two function that multiples a number by 2, and returns an int32 number.

It's also important that node-gyp-build is setup as a install script in the package.json:

  "scripts": {
    "install": "node-gyp-build"
  }

This means when you run npm install (which is used to install all the dependencies for a NPM package, or to install a specific NPM package), it will run the node-gyp-build durin the installation process.

This means that currently in our utils.nix node2nixDev expression still requires the npm install command. This used to exist, however I removed it during MatrixAI/TypeScript-Demo-Lib#37 thinking it had no effect. But it was confirmed by svanderburg/node2nix#293 (comment) that the npm install command is still run in order to execute build scripts. And node-gyp-build is now part of the installation process. We should include: https://github.com/svanderburg/node2nix/blob/8264147f506dd2964f7ae615dea65bd13c73c0d0/nix/node-env.nix#L380-L387 with all the necessary flags and parameters too. We may be able to make it work if we hook our build command prior to npm install. I imagine that this should be possible since the npm rebuild command is executed prior. So we need to investigate this.

In order to make this all work, our Nix environment is going to need all the tools for source compilation. Now according to https://github.com/nodejs/node-gyp#on-unix we will need python3, make and gcc. Our shell.nix naturally has make and gcc because we are using pkgs.mkShell which must extend from stdenv.mkDerivation. However python3 will be needed as well.

The node2nix has some understanding of native dependencies (this is why it also brings in python in its generated derivation svanderburg/node2nix#281), and I believe it doesn't actually build from source (except in some overridden dependencies).

Some npm dependencies are brought in via nixpkgs nodePackages because node2nix derivation isn't enough to build them (because they have complex native dependencies). Such as node-gyp-build itself or vercel's pkg. This is also why I had to provide nodePackages.node-gyp-build in our buildInputs overrides in utils.nix. It is important that any dependencies acquired via nixpkgs must be the same version we use in our package.json. And this is the case for:

    "node-gyp-build": "4.4.0"
    "pkg": "5.6.0",

Ideally we won't need to do this our own native packages if js-db ends up forking classic-level or leveldown. I think this trick is only relevant in our "build tools" and not our runtime dependencies.

The remaining problem is cross-compilation, as this only enables building from source if you are on NixOS and/or using Nix. Windows and MacOS will require their own setup. Since our development environment is all Nix focused, we don't have to worry about those, but for end-users who may want to rebuild from scratch, they will need to setup their development environent based on information in https://github.com/nodejs/node-gyp. A more pressing question is how we in our Nix development environment will be capable of cross-platform native addons for distribution.

This is where the prebuild ecosystem comes in and in particular https://github.com/prebuild/prebuildify-cross. This is used in leveldb to enable them to build for different platforms, and then save these cross-compiled objects. These objects are then hosted on GitHub releases, and automatically downloaded upon installation for downstream users. In the case they are not downloadable, they are then built from source. https://github.com/Level/classic-level/blob/f4cabe9e6532a876f6b6c2412a94e8c10dc5641a/package.json#L21-L26

However in our Nix based environment, I wonder if we can avoid using docker to do cross compilation, and instead use Nix to provide all the tooling to do cross-compilation. We'll see how this plays out eventually.

Some additional convenience commands now:

# install the current package and install all its dependencies, and build them ALL from source
npm install --build-from-source
# install a specific dependency and build it from source
npm install classic-level --build-from-source
# runs npm build on current package and all dependencies, and also recompiles all C++ addons
npm rebuild
# runs npm build on current package and all dependencies, and specifically recompiles sqlite3 package which has C++ addon
npm rebuild --build-from-source=sqlite3

Issues Fixed

Tasks

  • 1. Integrate node-gyp-build
  • 2. Create a native module exporting a demo functions like addOne for primitives and setProperty for reference-passing procedure and makeArray for heap allocation
  • 3. Fix the nix expressions to support node-gyp-build and other build scripts, and see if we can eliminate our postInstall hook, by relying on package.json hooks instead
  • 4. Integrate prebuildify to precompile binaries and host them on our git release... but this depends on whether typescript-demo-lib is used as a library or as an application, if used as an application, then the pkg builds is used, if used as a library, then one must install the native binary from the same github release, this means the native binary must be part of the same release page.
    • The pkg integration may just be a matter of setting the assets path in package.json to the local prebuilds directory.
    • See the scripts that other projects used WIP: Demonstrating Native Addons TypeScript-Demo-Lib#38 (comment)
    • Ensure that compiled native addons in nix have their rpath removed, because nodejs addons shouldn't have an rpath set, and this enables them to be portable
  • [ ] 5. Cross compilation, prebuildify-cross or something else that uses Nix - we must use CI/CD to do cross compilation (not sure about other architectures like ARM)
  • 6. Update the @typescript-eslint packages to match js-db to avoid the warning message.
  • 7. Add typescript typings to a native module
  • [ ] 8. Update README.md to indicate the 2 branches of typescript-demo-lib, the main and the native branch, where the native branch indicates how to build native addons - this will be done in a separate repo: https://github.com/MatrixAI/TypeScript-Demo-Lib-Native based off https://gitlab.com/MatrixAI/Employees/matrix-team/-/issues/8#note_885403611
  • 9. Migrate changes to https://github.com/MatrixAI/TypeScript-Demo-Lib-Native and request mac access to it the repository on gitlab. This branch will just be for development first. The changes here are too significant to keep within the same template repository.
  • 10. The pkg bundle can receive optimisation on which prebuild architectures it bundles, right now it bundles all architectures, when the target architecture implies only a single architecture is required. This can slim the final output pkg so it's not storing random unnecessary things. This may mean that pkg requires dynamic --config to be generated.
  • 11. See if nix-build ./release.nix -A application can be use prebuilds/ directory as well, as this can unify with pkg. That way all things can use prebuilds/ directory. But we would want to optimise it with task 10.
  • [ ] 12. Ensure that npm test can automatically run general tests, and platform-specific tests if detected on the relevant platform - this can be done in polykey as a script
  • 13. Automatic npm publish for prerelease and release based on staging and master branches, add these to the CI/CD jobs
  • 14. Ensure that integration CI/CD jobs are passing by running the final executable with all the bundled prebuilt binaries

Future Tasks

Final checklist

  • Domain specific tests
  • Full tests
  • Updated inline-comment documentation
  • Lint fixed
  • Squash and rebased
  • Sanity check the final build

@CMCDragonkai CMCDragonkai self-assigned this May 21, 2022
@CMCDragonkai CMCDragonkai changed the base branch from master to staging May 21, 2022 09:55
@ghost
Copy link

ghost commented May 21, 2022

👆 Click on the image for a new way to code review
  • Make big changes easier — review code in small groups of related files

  • Know where to start — see the whole change at a glance

  • Take a code tour — explore the change with an interactive tour

  • Make comments and review — all fully sync’ed with github

    Try it now!

Review these changes using an interactive CodeSee Map

Legend

CodeSee Map Legend

@CMCDragonkai
Copy link
Member Author

The npm publish problem is a bit more complicated. So the situation is that the dev still needs to do npm version prerelease/prepatch/premajor/preminor --preid alpha if they want to produce a prerelease.

That should create a git tag.

Then when we push this git tag, a job called build:prerelease will run that will perform the npm publish --tag staging. The --tag here is a bit a strange thing because it makes it seem this only runs in staging.

Technically this was meant to be like this, however, you can submit tag pointing to any commit, it doesn't have to be pointing to a staging branch.

Ideally, upon a commit coming into staging branch, and if that commit had a tag pre-release associated, then we would perform the build:prerelease job.

However there's no actual good rules for triggering something like this. What we could do is run a job that checks whether to do a prerelease by checking if the commit also has a tag. It seems like this should be possible, as long as the git repo also has the tag information when cloned in.

The other alternative is to run the job any time a prerelease tag occurs. This is basically how I started, but the prerelease job has needs on build:linux, build:windows, build:macos. And these along with the check:lint were all manual for tags. And that basically meant the job was blocked waiting for manual running of the final pipeline for prerelease. This of course doesn't make sense since it should be done automatically.

@CMCDragonkai CMCDragonkai merged commit fe29c7f into staging May 22, 2022
@CMCDragonkai
Copy link
Member Author

One of the problems with relying on commits is that pipelines are interruptible. So right now, if a commit that is meant to mean some sort of release is pushed, that may be hidden in other commits that interrupt it.

Currently Gitlab CI doesn't have a way of having conditional interruption: https://gitlab.com/gitlab-org/gitlab/-/issues/194023

This means we have to use tags.

However the previous reason why tagging wasn't working before was because I had it limited to protected refs, and I had to temporarily disable protection on these refs in order to test out these different ways of doing things.

@CMCDragonkai
Copy link
Member Author

The only downside to having tag pipelines is that it doubles up the pipeline when a commit is submitted too. That is since npm version command produces both a tag and commit, we performing the pipeline twice.

So have a tradeoff here. Either we allow a pipeline to run twice which is a bit of a waste, or we allow for the possibility of skipped prerelease commits. It isn't just when another commit is pushed, but one may perform npm version then do another commit on top before pushing altogether.

@CMCDragonkai
Copy link
Member Author

I believe for the second condition we will need to only perform the task on the tag pipeline. As for the first problem, we would need to add something that checks if the commit is associated to a release commit, and if so, skip it.

@CMCDragonkai
Copy link
Member Author

This was accidentally closed due to merge into staging. Need to reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

1 participant