Updating docu

README.md

@@ -1,5 +1,8 @@
 # Kafka proto 
-This repo contains a kafka producer for different type of proto files.
+This repo is used to interact with protobuf data for kafka.
+We have a python producer that creates several outputs on kafka, each topic with its own schema.
+From here on you can use kafka connect or other technologies to push the data to other solutions.
+
 The data itself was taken from a kaggle competition.
 
 ## Getting started
@@ -13,10 +16,48 @@ You will need the following tools to get started:
 The following proto files are present
 * etf.proto: A simple proto file with basic datatypes
 * etf_complex.proto: A proto file with complex data types
+* etf_http_ref.proto: A proto file that has an http link as a reference
+
+## Starting the docker environment
+The docker environment can be started with the single command:
+```bash
+docker-compose -f docker-compose.yml up -d
+```
 
-## Running the program
+The following technologies will be started:
+* Confluent center and all related technologies
+* ElasticSearch and Kibana
+* Postgresql
+
+## Running the python producer
 The following script needs to be executed:
 ```python
 .../kafka_proto_py/kafka_proto_api/start_producer.py
 ```
-You will need to set the working directory to the root of the project
+You will need to set the working directory to the root of the project
+
+## Running the kafka connector
+The repo also contains an example of a kafka connector under the folder `kafka-connect`.
+The file present will start a kafka connect job that will read the data from the configured topic and write it to 
+elasticsearch.
+You can run the following command from the `kafka-connect directory`
+```bash
+curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '@etf-2-elk.json'
+```
+
+## Installing additional connectors
+If you want you can install additional kafka connectors by editing the file in `docker/DockerfileConnect`.
+To make the maintenance easier, we install connectors through the command `confluent-hub install`.
+
+## Querying the data 
+If you want to verify if the data is in elasticsearch you will need to create an index. This can be done from the 
+kibana console, which is reachable from the URL `http://localhost:5601`
+
+Once the index is created you can create the dev tool to query the data as per the screenshot
+![screenshot](docu_img/elasticsearch_index.JPG)
+
+## Contributing
+If you want to contribute, please abide by the following rules:
+* Create a feature branch and add your changes
+* Create a pull request to merge into master
+* Have a reviewer and merge