INTEROP.md

Kotlin/Native interoperability

Introduction

Kotlin/Native follows general tradition of Kotlin to provide excellent existing platform software interoperability. In case of native platform most important interoperability target is a C library. Thus Kotlin/Native comes with an cinterop tool, which could be used to quickly generate everything needed to interact with an external library.

Following workflow is expected when interacting with the native library.

• create .def file describing what to include into bindings
• use cinterop tool to produce Kotlin bindings
• run Kotlin/Native compiler on an application to produce the final executable

Interoperability tool analyses C headers and produces "natural" mapping of types, function and constants into the Kotlin world. Generated stubs can be imported into an IDE for purposes of code completion and navigation.

Simple example

Build the dependencies and the compiler (see README.md).

Prepare stubs for the system sockets library:

cd samples/socket
../../dist/bin/cinterop -def src/main/c_interop/sockets.def \
 -o sockets

Compile the echo server:

../../dist/bin/kotlinc src/main/kotlin/EchoServer.kt \
 -library sockets -o EchoServer

This whole process is automated in build.sh script, which also support cross-compilation to supported cross-targets with TARGET=raspberrypi ./build.sh (cross_dist target must be executed first).

Run the server:

./EchoServer.kexe 3000 &

Test the server by connecting to it, for example with telnet:

telnet localhost 3000

Write something to console and watch server echoing it back.

Creating bindings for a new library

To create bindings for a new library, start by creating .def file. Structurally it's a simple property file, looking like this:

headers = zlib.h
compilerOpts = -std=c99

Then run cinterop tool with something like (note that for host libraries not included in sysroot search paths for headers may be needed):

cinterop -def zlib.def -copt -I/opt/local/include -o zlib

This command will produce zlib.klib compiled library and zlib-build/kotlin directory containing Kotlin source code for the library.

If behavior for certain platform shall be modified, one may use format like compilerOpts.osx or compilerOpts.linux to provide platform-specific values to options.

Note, that generated bindings are generally platform-specific, so if developing for multiple targets, bindings need to be regenerated.

After generation of bindings they could be used by IDE as proxy view of the native library.

For typical Unix library with config script compilerOpts will likely contain output of config script with --cflags flag (maybe without exact paths).

Output of config script with --libs shall be passed as -linkedArgs kotlinc flag value (quoted) when compiling.

Selecting library headers

When library headers are imported to C program with #include directive, all of the headers included by these headers are also included to the program. Thus all header dependencies are included in generated stubs as well.

This behaviour is correct but may be very inconvenient for some libraries. So it is possible to specify in .def file which of the included headers are to be imported. The separate declarations from other headers may also be imported in case of direct dependencies.

Filtering headers by globs

It is possible to filter header by globs. The headerFilter property value from the .def file is treated as space-separated list of globs. If the included header matches any of the globs, then declarations from this header are included into the bindings.

The globs are applied to the header paths relative to the appropriate include path elements, e.g. time.h or curl/curl.h. So if the library is usually included with #include <SomeLbrary/Header.h>, then it would probably be correct to filter headers with

headerFilter = SomeLbrary/**

If headerFilter is not specified, then all headers are included.

Filtering by module maps

Some libraries have proper module.modulemap or module.map files among its headers. For example, macOS and iOS system libraries and frameworks do. The module map file describes the correspondence between header files and modules. When the module maps are available, the headers from the modules that are not included directly can be filtered out using experimental excludeDependentModules option of the .def file:

headers = OpenGL/gl.h OpenGL/glu.h GLUT/glut.h
compilerOpts = -framework OpenGL -framework GLUT
excludeDependentModules = true

When both excludeDependentModules and headerFilter are used, they are applied as intersection.

Adding custom declarations

Sometimes it is required to add custom C declarations to the library before generating bindings (e.g. for macros). Instead of creating additional header file with these declarations, you can include them directly to the end of the .def file, after separating line, containing only the separator sequence ---:

headers = errno.h

---

static inline int getErrno() {
    return errno;
}

Note that this part of the .def file is treated as part of the header file, so functions with body should be declared as static. The declarations are parsed after including the files from headers list.

Including static library in your klib

Sometimes it is more convenient to ship a static library with your product, rather that assuming it is available within the user environment. To include a static library into .klib use staticLibrary and libraryPaths clauses. For example:

staticLibraries = libfoo.a 
libraryPath = /opt/local/lib /usr/local/opt/curl/lib

When given the above snippet the cinterop tool will search libfoo.a in /opt/local/lib and /usr/local/opt/curl/lib, and if found include the library binary into klib.

When using such klib in your program the library is linked automatically.

Using bindings

Basic interop types

All supported C types have corresponding representations in Kotlin:

• Singed, unsigned integral and floating point types are mapped to their Kotlin counterpart with the same width.
• Pointers and arrays are mapped to CPointer<T>?.
• Enums can be mapped to either Kotlin enum or integral values, depending on heuristics and definition file hints (see "Definition file hints" below).
• Structs are mapped to types having fields available via dot notation, i.e. someStructInstance.field1.
• typedefs are represented as typealiases.

Also any C type has the Kotlin type representing the lvalue of this type, i.e. the value located in memory rather than simple immutable self-contained value. Think C++ references, as similar concept. For structs (and typedefs to structs) this representation is the main one and has the same name as the struct itself, for Kotlin enums it is named ${type}.Var, for CPointer<T> it is CPointerVar<T>, and for most other types it is ${type}Var.

For those types that have both representations, the "lvalue" one has mutable .value property for accessing value.

Pointer types

The type argument T of CPointer<T> must be one of the "lvalue" types described above, e.g. the C type struct S* is mapped to CPointer<S>, int8_t* is mapped to CPointer<int_8tVar>, and char** is mapped to CPointer<CPointerVar<ByteVar>>.

C null pointer is represented as Kotlin's null, and the pointer type CPointer<T> is not nullable, but the CPointer<T>? is. The values of this type support all Kotlin operations related to handling null, e.g. ?:, ?., !! etc:

val path = getenv("PATH")?.toKString() ?: ""

Since the arrays are also mapped to CPointer<T>, it supports [] operator for accessing values by index:

fun shift(ptr: CPointer<BytePtr>, length: Int) {
    for (index in 0 .. length - 2) {
        ptr[index] = ptr[index + 1]
    }
}

The .pointed property for CPointer<T> returns the lvalue of type T, pointed by this pointer. The reverse operation is .ptr: it takes the lvalue and returns the pointer to it.

void* is mapped to COpaquePointer – the special pointer type which is the supertype for any other pointer type. So if the C function takes void*, then the Kotlin binding accepts any CPointer.

Casting any pointer (including COpaquePointer) can be done with .reinterpret<T>, e.g.:

val intPtr = bytePtr.reinterpret<IntVar>()

or

val intPtr: CPointer<IntVar> = bytePtr.reinterpret()

As in C, those reinterpret casts are unsafe and could potentially lead to subtle memory problems in an application.

Also there are unsafe casts between CPointer<T>? and Long available, provided by .toLong() and .toCPointer<T>() extension methods:

val longValue = ptr.toLong()
val originalPtr = longValue.toCPointer<T>()

Note that if the type of the result is known from the context, the type argument can be omitted as usual due to type inference.

Memory allocation

The native memory can be allocated using NativePlacement interface, e.g.

val byteVar = placement.alloc<ByteVar>()

or

val bytePtr = placement.allocArray<ByteVar>(5):

The most "natural" placement is object nativeHeap. It corresponds to allocating native memory with malloc and provides additional .free() operation to free allocated memory:

val buffer = nativeHeap.allocArray<ByteVar>(size)
<use buffer>
nativeHeap.free(buffer)

However the lifetime of allocated memory is often bound to lexical scope. It is possible to define such scope with memScoped { ... }. Inside the braces the temporary placement is available as implicit receiver, so it is possible to allocate native memory with alloc and allocArray, and the allocated memory will be automatically freed after leaving the scope.

For example, the C function returning values through pointer parameters can be used like

val fileSize = memScoped {
    val statBuf = alloc<statStruct>()
    val error = stat("/", statBuf.ptr)
    statBuf.st_size
}

Passing pointers to bindings

Although C pointers are mapped to CPointer<T> type, the C function pointer-typed parameters are mapped to CValuesRef<T>. When passing CPointer<T> as the value of such parameter, it is passed to C function as is. However, the sequence of values can be passed instead of pointer. In this case the sequence is passed "by value", i.e. the C function receives the pointer to the temporary copy of that sequence, which is valid only until the function returns.

The CValuesRef<T> representation of pointer parameters is designed to support C array literals without explicit native memory allocation. To construct the immutable self-contained sequence of C values, the following methods are provided:

• ${type}Array.toCValues(), where type is the Kotlin primitive type
• Array<CPointer<T>?>.toCValues(), List<CPointer<T>?>.toCValues()
• cValuesOf(vararg elements: ${type}), where type is primitive or pointer

For example:

C:

void foo(int* elements, int count);
...
int elements[] = {1, 2, 3};
foo(elements, 3);

Kotlin:

foo(cValuesOf(1, 2, 3), 3)

Working with the strings

Unlike other pointers, the parameters of type const char* are represented as Kotlin String. So it is possible to pass any Kotlin string to the binding expecting C string.

There are also available some tools to convert between Kotlin and C strings manually:

• fun CPointer<ByteRef>.toKString(): String

• val String.cstr: CValuesRef<ByteRef>.

  To get the pointer, .cstr should be allocated in native memory, e.g.

  val cString = kotlinString.cstr.getPointer(nativeHeap)
  

In all cases the C string is supposed to be encoded as UTF-8.

Passing and receiving structs by value

When C function takes or returns a struct T by value, the corresponding argument type or return type is represented as CValue<T>.

CValue<T> is an opaque type, so structure fields cannot be accessed with appropriate Kotlin properties. It could be acceptable, if API uses structures as handles, but if field access is required, there are following conversion methods available:

• fun T.readValue(): CValue<T>. Converts (the lvalue) T to CValue<T>. So to construct the CValue<T>, T can be allocated, filled and then converted to CValue<T>.

• CValue<T>.useContents(block: T.() -> R): R. Temporarily places the CValue<T> to the memory, and then runs the passed lambda with this placed value T as receiver. So to read a single field, the following code can be used:

  val fieldValue = structValue.useContents { field }
  

Callbacks

To convert Kotlin function to pointer to C function, staticCFunction(::kotlinFunction) can be used. It is also allowed to provide the lambda instead of function reference. The function or lambda must not capture any values.

Note that some function types are not supported currently. For example, it is not possible to get pointer to function that receives or returns structs by value.

Passing user data to callbacks

Often C APIs allow passing some user data to callbacks. Such data is usually provided by user when configuring the callback. It is passed to some C function (or written to the struct) as e.g. void*. However references to Kotlin objects can't be directly passed to C. So they require wrapping before configuring callback and then unwrapping in the callback itself, to safely swim from Kotlin to Kotlin through the C world. Such wrapping is possible with StableObjPtr class.

To wrap the reference:

val stablePtr = StableObjPtr.create(kotlinReference)
val voidPtr = stablePtr.value

where the voidPtr is COpaquePointer and can be passed to the C function.

To unwrap the reference:

val stablePtr = StableObjPtr.fromValue(voidPtr)
val kotlinReference = stablePtr.get()

where kotlinReference is the original wrapped reference (however it's type is Any so it may require casting).

The created StableObjPtr should eventually be manually disposed using .dispose() method to prevent memory leaks:

stablePtr.dispose()

After that it becomes invalid, so voidPtr can't be unwrapped anymore.

See samples/libcurl for more details.

Macros

Every C macro that expands to a constant is represented as Kotlin property. Other macros are not supported. However they can be exposed manually by wrapping with supported declarations. E.g. function-like macro FOO can be exposed as function foo by adding the custom declaration to the library:

headers = library/base.h

---

static inline int foo(int arg) {
    return FOO(arg);
}

Definition file hints

The .def file supports several options for adjusting generated bindings.

• excludedFunctions property value specifies a space-separated list of names of functions that should be ignored. This may be required because a function declared in C header is not generally guaranteed to be really callable, and it is often hard or impossible to figure this out automatically. This option can also be used to workaround a bug in the interop itself.

• strictEnums and nonStrictEnums properties values are space-separated lists of the enums that should be generated as Kotlin enum or as integral values correspondingly. If the enum is not included into any of these lists, than it is generated according to the heuristics.

Portability

Sometimes the C libraries have function parameters or struct fields of platform-dependent type, e.g. long or size_t. Kotlin itself doesn't provide neither implicit integer casts nor C-style integer casts (e.g. (size_t) intValue), so to make writing portable code in such cases easier, the following methods are provided:

• fun ${type1}.signExtend<${type2}>(): ${type2}
• fun ${type1}.narrow<${type2}>(): ${type2}

where each of type1 and type2 must be an integral type.

The signExtend converts the integer value to more wide, i.e. the result must have the same or greater size. The narrow converts the integer value to smaller one (possibly changing the value due to loosing significant bits), so the result must have the same or less size.

Any allowed .signExtend<${type}> or .narrow<${type}> have the same semantics as one of the .toByte, .toShort, .toInt or .toLong methods, depending on type.

The example of using signExtend:

fun zeroMemory(buffer: COpaquePointer, size: Int) {
    memset(buffer, 0, size.signExtend<size_t>())
}

Also the type parameter can be inferred automatically and thus may be omitted in some cases.