Dyalog-Kafka

The aim of the Dyalog-Kafka project is to provide a binding to part of the Confluent librdkafka library such that we can access Kafka from Dyalog APL.

Note KAFKA is a registered trademark of The Apache Software Foundation and has been licensed for use by Dyalog Ltd. Dyalog Ltd has no affiliation with and is not endorsed by The Apache Software Foundation.

Build and installation

At the moment, this library must be built from source. Clone this repository. You will need a C++ development toolchain installed.

Windows

Open kafka.sln in Visual Studio 2022, and build the project.

Copy all the dlls from the OutDir (kafka\x64\Debug for 64 bit debug version) to a directory and Start Apl in that directory.

Linux

To build on linux for 64 bit

PLATFORM=linux ./mk_kafka.sh 64

The output files are in distribution/linux/x64

AIX

TODO

MacOS

Install the librdkafka via Homebrew:

brew install librdkafka

Build the wrapper:

cd <path>/kafka
c++ -shared -fpic -oMACbin/kafka.dylib -DFORmac  kafka/kafka.cpp -lrdkafka -L/opt/homebrew/opt/librdkafka/lib -I/opt/homebrew/opt/librdkafka/include/librdkafka

or like linux but this requires dotnet to be installed to get the librdkafka package

PLATFORM=mac ./mk_kafka.sh 64

Initialising

Now start Dyalog. You need a running Kafka instance. In the session, type

]cd path/to/kafka/repo
]link.create # aplsource

Initialise the library, passing as the argument the path where the kafka.[so|a|dylib] shared library was installed:

Init 'path/to/dir/housing/shared/lib' ⍝ to load the dll

For further instructions, see the file SPEC.md, and sample code examples:

1. Sample 1
2. Sample 2
3. Sample 3
4. Sample 4