Skip to content

Latest commit

 

History

History
579 lines (487 loc) · 20.8 KB

README.md

File metadata and controls

579 lines (487 loc) · 20.8 KB

JNI Zero

A zero-overhead (or better!) middleware for JNI.

Overview

JNI (Java Native Interface) is the mechanism that enables Java code to call native functions, and native code to call Java functions.

  • Native code calls into Java using apis from <jni.h>, which basically mirror Java's reflection APIs.
  • Java code calls native functions by declaring body-less functions with the native keyword, and then calling them as normal Java functions.

JNI Zero generates boiler-plate code with the goal of making our code:

  1. easier to write,
  2. typesafe,
  3. more optimizable.

JNI Zero uses regular expressions to parse .java files, so don't do anything too fancy. E.g.:

  • Classes must be either explicitly imported, or are assumed to be in the same package. To use java.lang classes, add an explicit import.
  • Inner classes need to be referenced through the outer class. E.g.: void call(Outer.Inner inner)

Exposing Native Methods

There are two ways to have native methods be found by Java:

  1. Explicitly register the name -> function pointer mapping using JNI's RegisterNatives() function.
  2. Export the symbols from the shared library, and let the runtime resolve them on-demand (using dlsym()) the first time a native method is called.

(2) Is generally preferred due to a smaller code size and less up-front work, but (1) is sometimes required (e.g. when OS bugs prevent dlsym() from working). Both ways are supported by this tool.

Exposing Java Methods

Java methods just need to be annotated with @CalledByNative. By default the generated method stubs on the native side are not namespaced. The generated functions can be put into a namespace using @JNINamespace("your_namespace").

Usage

Writing Build Rules

  1. Find or add a generate_jni target with your .java file, then add this generate_jni target to your srcjar_deps of your android_library target:

    generate_jni("abcd_jni") {
  sources = [ "path/to/java/sources/with/jni/Annotations.java" ]
}

android_library("abcd_java") {
  ...
  # Allows the java files to see the generated `${OriginalClassName}Jni`
  # classes.
  srcjar_deps = [ ":abcd_jni" ]
}

source_set("abcd") {
 ...
 # Allows the cpp files to include the generated `${OriginalClassName}_jni.h`
 # headers.
 deps = [ ":abcd_jni" ]
}

Calling Java -> Native

  • For each JNI method:
    • C++ stubs are generated that forward to C++ functions that you must write. By default the c++ functions you are expected to implement are not associated with a class.
    • If the first parameter is a C++ object (e.g. long native${OriginalClassName}), then the bindings will not call a static function but instead cast the variable into a cpp ${OriginalClassName} pointer type and then call a member method with that name on said object.

To add JNI to a class:

  1. Create a nested-interface annotated with @NativeMethods that contains the declaration of the corresponding static methods you wish to have implemented.
  2. Call native functions using ${OriginalClassName}Jni.get().${method}
  3. In C++ code, #include the header ${OriginalClassName}_jni.h. (The path will depend on the location of the generate_jni BUILD rule that lists your Java source code.) Only include this header from a single .cc file as the header defines functions. That .cc must implement your native code by defining non-member functions named JNI_${OriginalClassName}_${UpperCamelCaseMethod} for static methods and member functions named ${OriginalClassName}::${UpperCamelCaseMethod} for non-static methods. Member functions need be declared in the header file as well.

Example:

Java

class MyClass {
  // Cannot be private. Must be package or public.
  @NativeMethods
  /* package */ interface Natives {
    void foo();
    double bar(int a, int b);
    // Either the |MyClass| part of the |nativeMyClass| parameter name must
    // match the native class name exactly, or the method annotation
    // @NativeClassQualifiedName("MyClass") must be used.
    //
    // If the native class is nested, use
    // @NativeClassQualifiedName("FooClassName::BarClassName") and call the
    // parameter |nativePointer|.
    void nonStatic(long nativeMyClass);
  }

  void callNatives() {
    // MyClassJni is generated by the generate_jni rule.
    // Storing MyClassJni.get() in a field defeats some of the desired R8
    // optimizations, but local variables are fine.
    Natives jni = MyClassJni.get();
    jni.foo();
    jni.bar(1,2);
    jni.nonStatic(mNativePointer);
  }
}

C++

#include "third_party/jni_zero/jni_zero.h"
#include "<path to BUILD.gn>/<generate_jni target name>/MyClass_jni.h"

class MyClass {
public:
  void NonStatic(JNIEnv* env);
}

// Notice that unlike Java, function names are capitalized in C++.
// Static function names should follow this format and don't need to be declared.
void JNI_MyClass_Foo(JNIEnv* env) { ... }
void JNI_MyClass_Bar(JNIEnv* env, jint a, jint b) { ... }

// Member functions need to be declared.
void MyClass::NonStatic(JNIEnv* env) { ... }

Calling Native -> Java

Because the generated header files contain definitions as well as declarations, the must not be #included by multiple sources. If there are Java functions that need to be called by multiple sources, one source should be chosen to expose the functions to the others via additional wrapper functions.

  1. Annotate some methods with @CalledByNative, the generator will now generate stubs in ${OriginalClassName}_jni.h header to call into those java methods from cpp.

    • Inner class methods must provide the inner class name explicitly (ex. @CalledByNative("InnerClassName"))

  2. In C++ code, #include the header ${OriginalClassName}_jni.h. (The path will depend on the location of the generate_jni build rule that lists your Java source code). That .cc can call the stubs with their generated name Java_${OriginalClassName}_${UpperCamelCaseMethod}.

Note: For test-only methods, use @CalledByNativeForTesting which will ensure that it is stripped in our release binaries.

Automatic Type Conversions using @JniType

Normally, Java types map to C++ types from <jni.h> (e.g. jstring for java.lang.String). The first thing most people do is convert the jni spec types into standard C++ types.

@JniType to the rescue. By annotating a parameter or a return type with @JniType("cpp_type_here") the generated code will automatically convert from the jni type to the type listed inside the annotation. See example:

Original Code:

class MyClass {
  @NativeMethods
  interface Natives {
    void foo(
            String string,
            String[] strings,
            MyClass obj,
            MyClass[] objs)
  }
}
#include "third_party/jni_zero/jni_zero.h"
#include "<path to BUILD.gn>/<generate_jni target name>/MyClass_jni.h"

void JNI_MyClass_Foo(JNIEnv* env, const JavaParamRef<jstring>&, const JavaParamRef<jobjectArray>&, const JavaParamRef<jobject>&, JavaParamRef<jobjectArray>&) {...}

After using @JniType

class MyClass {
  @NativeMethods
  interface Natives {
    void foo(
            @JniType("std::string") String convertedString,
            @JniType("std::vector<std::string>") String[] convertedStrings,
            @JniType("myModule::CPPClass") MyClass convertedObj,
            @JniType("std::vector<myModule::CPPClass>") MyClass[] convertedObjects);
  }
}
#include "third_party/jni_zero/jni_zero.h"
#include "<path to BUILD.gn>/<generate_jni target name>/MyClass_jni.h"

void JNI_MyClass_Foo(JNIEnv* env, std::string&, std::vector<std::string>>&, myModule::CPPClass&, std::vector<myModule::CPPClass>&) {...}

Implementing Conversion Functions

Conversion functions must be defined for all types that appear in @JniType. Forgetting to add one will result in errors at link time.

// The conversion function primary templates.
template <typename O>
O FromJniType(JNIEnv*, const JavaRef<jobject>&);
template <typename O>
O FromJniType(JNIEnv*, const JavaRef<jstring>&);
template <typename O>
ScopedJavaLocalRef<jobject> ToJniType(JNIEnv*, const O&);

An example conversion function can look like:

#include "third_party/jni_zero/jni_zero.h"

namespace jni_zero {
template <>
EXPORT std::string FromJniType<std::string>(
    JNIEnv* env,
    const JavaRef<jstring>& input) {
  // Do the actual conversion to std::string.
}

template <>
EXPORT ScopedJavaLocalRef<jstring> ToJniType<std::string>(
    JNIEnv* env,
    const std::string& input) {
  // Do the actual conversion from std::string.
}
}  // namespace jni_zero

If a conversion function is missing, you will get a linker error since we forward declare the conversion functions before using them.

Array Conversion Functions

Array conversion functions look different due to the partial specializations. The ToJniType direction also takes a jclass parameter which is the class of the array elements, because java requires it when creating a non-primitive array.

template <typename O>
struct ConvertArray {
  static O FromJniType(JNIEnv*, const JavaRef<jobjectArray>&);
  static ScopedJavaLocalRef<jobjectArray> ToJniType(JNIEnv*, const O&, jclass);
};

JniZero provides implementations for partial specializations to wrap and unwrap std::vector for object arrays and some primitive arrays.

Nullability

All non-primitive default JNI C++ types (e.g. jstring, jobject) are pointer types (i.e. nullable). Some C++ types (e.g. std::string) are not pointer types and thus cannot be nullptr. This means some conversion functions that return non-nullable types have to handle the situation where the passed in java type is null.

You can get around this by having the conversion be to std::optional<T> rather than just T if T is not a nullable type.

Testing Mockable Natives

/**
 * Tests for {@link AnimationFrameTimeHistogram}
 */
@RunWith(RobolectricTestRunner.class)
public class AnimationFrameTimeHistogramTest {
    // Optional: Resets test overrides during tearDown().
    // Not needed when using Chrome's test runners.
    @Rule public JniResetterRule jniResetterRule = new JniResetterRule();

    @Mock
    AnimationFrameTimeHistogram.Natives mNativeMock;

    @Before
    public void setUp() {
        MockitoAnnotations.initMocks(this);
        AnimationFrameTimeHistogramJni.setInstanceForTesting(mNativeMock);
    }

    @Test
    public void testNatives() {
        AnimationFrameTimeHistogram hist = new AnimationFrameTimeHistogram("histName");
        hist.startRecording();
        hist.endRecording();
        verify(mNativeMock).saveHistogram(eq("histName"), any(long[].class), anyInt());
    }
}

If a native method is called without setting a mock in a unit test, an UnsupportedOperationException will be thrown.

Special case: APK Splits

Each APK split with its own native library has its own generated GEN_JNI, which is <module_name>_GEN_JNI. In order to get your split's JNI to use the <module_name> prefix, you must add your module name into the argument of the @NativeMethods annotation.

So, for example, say your module was named test_module. You would annotate your Natives interface with @NativeMethods("test_module"), and this would result in test_module_GEN_JNI.

Testing for readiness: use get()

JNI Generator automatically produces asserts that verify that the Natives interface can be safely called. These checks are compiled out of Release builds, making these an excellent way to determine whether your code is called safely.

It is not sufficient, however, to use <Class>Jni.get() to guarantee native is initialized - it is only a debugging tool to ensure that you're using native after native is loaded.

If you expect your code to be called by an external caller, it's often helpful to know ahead of time that the context is valid (ie. either native libraries are loaded or mocks are installed). In this case it is helpful to call get() method, that performs all the Debug checks listed above, but does not instantiate a new object for interfacing Native libraries. Note that the unused value returned by the get() method will be optimized away in release builds so there's no harm in ignoring it.

Addressing Jni.get() exceptions.

When you identify a scenario leading to an exception, relocate (or defer) the appropriate call to be made to a place where (or time when) you know the native libraries have been initialized (eg. onStartWithNative, onNativeInitialized etc).

Please avoid calling LibraryLoader.isInitialized() / LibraryLoader.isLoaded() in new code. Using LibraryLoader calls makes unit-testing more difficult:

  • this call can not verify whether Mock object is used, making the use of mocks more complicated,
  • using LibraryLoader.setLibrariesLoadedForNativeTests() alters the state for subsequently executed tests, inaccurately reporting flakiness and failures of these victim tests.
  • Introducing LibraryLoader.is*() calls in your code immediately affects all callers, forcing the authors of the code up the call stack to override LibraryLoader internal state in order to be able to unit-test their code.

However, if your code is going to be called both before and after native is initialized, you are forced to call LibraryLoader.isInitialized() to be able to differentiate. Calling <Class>Jni.get() only provides assertions, and will fail in debug builds if you call it when native isn't ready.

Java Objects and Garbage Collection

All pointers to Java objects must be registered with JNI in order to prevent garbage collection from invalidating them.

For Strings & Arrays - it's common practice to use the //base/android/jni_* helpers to convert them to std::vectors and std::strings as soon as possible.

For other objects - use smart pointers to store them:

  • ScopedJavaLocalRef<> - When lifetime is the current function's scope.
  • ScopedJavaGlobalRef<> - When lifetime is longer than the current function's scope.
  • JavaObjectWeakGlobalRef<> - Weak reference (do not prevent garbage collection).
  • JavaParamRef<> - Use to accept any of the above as a parameter to a function without creating a redundant registration.

Additional Guidelines / Advice

Minimize the surface API between the two sides. Rather than calling multiple functions across boundaries, call only one (and then on the other side, call as many little functions as required).

If a Java object "owns" a native one, store the pointer via "long mNativeClassName". Ensure to eventually call a native method to delete the object. For example, have a close() that deletes the native object.

The best way to pass "compound" types across in either direction is to create an inner class with PODs and a factory function. If possible, mark all the fields as "final".

Build Rules

  • generate_jni - Given a set of Java files, generates a header file to call into Java for all @CalledByNative functions. If @NativeMethods is present, also generates a .srcjar containing <ClassName>Jni.java, which should be depended on via the generated GN target <generate_jni's target name>_java.
  • generate_jar_jni - Given a .jar file, generates a header file similar to generate_jni, if every method and public field were annotated by @CalledByNative.
  • generate_jni_registration - Generates a whole-program Java and native link - required for all Java that calls into native via @NativeMethods.
  • shared_library_with_jni - A wrapper around a native shared_library, which also inserts a __jni_registration target for the library.
  • component_with_jni - Same as shared_library but for a component.

Refer to jni_zero.gni for more about the GN templates.

JNI Benchmarking

Refer to the performance README.

Under the Hood

For @CalledByNative, we directly call the <jni.h> methods, which are basically just reflection APIs, and then add a proguard rule to ensure the annotated method/field is kept in Java. The registration step does nothing for this direction of JNI, since we do not do any sort of proxying. However, using the registration step for @CalledByNatives has been discussed before: go/proxy-called-by-natives-proposal.

JNI Zero has 2 primary modes for @NativeMethods. In each, we insert a "proxy" class per annotated class which allows us to fake for tests and optimize better. We insert a class with the name of <EnclosingClass>Jni, and this class is just a testable shim into the "real" GEN_JNI class. This GEN_JNI class is generated at the registration step, and how the registration works is different in different modes.

For examples, we will imagine we have the following two classes:

class org.foo.Foo {
  @NativeMethods
  interface Natives {
    int f();
  }
}
class org.bar.Bar {
  @NativeMethods
  interface Natives {
    int b();
  }
}

Which will have the 2 generate_jni steps output something like:

// Java .srcjar outputs
class FooJni {
  public int f() {
    return GEN_JNI.org_foo_Foo_f();
  }
}
class BarJni {
  public int b() {
    return GEN_JNI.org_bar_Bar_b();
  }
}
// C++ header outputs
class FooJni {
int Java_GEN_JNI_org_foo_Foo_f() {
  return JNI_Foo_f(); // User implements this native function.
}
int Java_GEN_JNI_org_bar_Bar_b() {
  return JNI_Bar_b(); // User implements this native function.
}

Debug Mode

In debug mode, the GEN_JNI is a file containing native methods that match every single @NativeMethods from every generate_jni in our program.

class GEN_JNI {
  public static native int org_foo_Foo_f();
  public static native int org_bar_Bar_b();
}

Release Mode

In release mode, the GEN_JNI.java is just a callthrough shim to N.java (a short name to reduce size), and N uses multiplexing by signature type to reduce the number of JNI functions. Then, we generate a C++ file with matching names to the smaller list of functions in N, which de-multiplexes back into the original functions.

class GEN_JNI {
  public static int org_foo_Foo_f() {
    return N._I(0);
  }
  public static int org_bar_Bar_b() {
    return N._I(1);
  }
}
class N {
  public static native int _I(int switchNum);
}
// Generated C++ to be compiled into the final binary.
int Java_N__1V(jint switch_num) {
  switch (switch_num) {
    case 0:
      return org_foo_Foo_f();
    case 1:
      return org_bar_Bar_b();
  }
}

We also have the concept of "priority" classes, which are classes which need to be in the front of the multiplexing numbers. This is not a performance thing, it's so that Chrome can support multiple ABIs with a single Java file - we put the smaller (subset) ABI switch numbers first, and the superset ABI's unique classes get the final switch numbers.

Legacy Modes

These are modes which JNI provides currently, but we hope to remove. Please do not add any new uses of these.

Hashed Names

This was our old release mode. GEN_JNI would call into N, just as it does for our current release mode, but instead of multipelxing, we'd just take a short hash of the name so we have shorter exported string literals. This would also change the output of the headers made by generate_jni, as they needed to likewise have a hashed name generated.

class GEN_JNI {
  public static int org_foo_Foo_f() {
    return N.MaQxW612();
  }
  public static int org_bar_Bar_b() {
    return N.M2R2WaZb();
  }
}
class N {
  public static native int MaQxW612();
  public static native int M2R2WaZb();
}

Per-File Natives

This was added to make transitioning to JNI Zero easier. The idea is that this allows you to partially onboard without needing to use a registration step, so no GEN_JNI is generated at all, and the generate_jni step's outputs look different than "normal" mode.

class FooJni {
  public static int f() {
    nativeF();
  }
  public static native nativeF();
}
class BarJni {
  public static int b() {
    nativeB();
  }
  public static native nativeB();
}

Changing JNI Zero

  • Python golden tests live in test/integration_tests.py
  • A working demo app exists as sample:jni_zero_sample_apk and this app is tested in sample:jni_zero_sample_apk_test.
  • Compile-only tests exist in test:jni_zero_compile_check_apk
  • We are a Chromium project developed in the Chromium repo, but we intend to have no dependencies on Chromium, to allow this project to be easily portable.
  • jni_zero.py contains our flags and is the entry point, jni_generator.py is the main file for the per-library generation step, and jni_registration_generator.py is the main file for the whole-program registration step.