Stream transaction event handler events to a Kafka topic
Ideas based on a now removed repository from workday.
Currently for kafka
Build locally
mvn clean install
-
Copy
target/neo4j-kafka-*.jar
into$NEO4J_HOME/plugins
-
Restart Neo4j
You can set the following configuration values in your neo4j.conf
, here are the defaults.
kafka.zookeeper.connect=localhost:2181 kafka.bootstrap.servers=localhost:9092 kafka.acks=1 kafka.num.partitions=1 kafka.retries=2 kafka.batch.size=16384 kafka.buffer.memory=33554432 kafka.reindex.batch.size=1000 kafka.session.timeout.ms=15000 kafka.connection.timeout.ms=10000 kafka.replication=1 kafka.group.id=neo4j kafka.topics=neo4j kafka.patterns=neo4j:*
See the Apache Kafka documentation for details on these settings.
To control which nodes are sent to Kafka, and which of their properties you can define node-patterns in the config.
You can chose Labels and properties for inclusion or exclusion, with *
meaning all.
Patterns are separated by semicolons ;
.
The basic syntax is:
kafka.topics=topic1,topic2 kafka.patterns=topic1:*;topic2:Label1 {prop1, prop2}, topic1:Label1:Label2{*,-prop1}, topic2:Label3{-prop1,-prop2,prop3}
pattern | meaning |
---|---|
|
all nodes go to this topic |
|
all nodes with this label with all their properties go to this topic |
|
all nodes with this label with all their properties go to this topic |
|
nodes with these two labels are sent to this topic |
|
nodes with Label1 are sent to topic1, nodes with Label2 are sent to topic2 |
|
the prop1, prop2 of all nodes with this label are sent to the topic |
|
properties of the node with prop1, prop2 excluded, with the label are sent to the topic |
Following these instructions.
-
Download & Unzip Kafka
-
Run Zookeeper, server and a test consumer.
bin/zookeeper-server-start.sh config/zookeeper.properties bin/kafka-server-start.sh config/server.properties bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic neo4j [--from-beginning]
Implementation for a Kafka Sink / Source
based on work in https://github.com/pegerto/kafka-connect-neo4j