Fix various issues with SQL generation, flink job routing, integration-tests

[jogrogan]

• Fixed issue with FLINK SQL syntax where query table: KAFKA.existing-topic-2 should have just been existing-topic-2.
• Got deployment of Flink jobs working through Flink deployment cluster
  • Updated Flink connector options in kafkadb.yaml to the minimum required set needed to pass through to Flink
  • Needed to change Kafka connection mechanism. Connection on the command line using the forwarded port 9092 works via localhost to the 'external' strimzi cluster. However, this doesn't work for k8s pod to pod communication (Flink deployment pod -> Kafka cluster), pod connections need to use the 'internal' strimzi connection mechanisms one-kafka-bootstrap.kafka.svc.cluster.local:9094.
• Integration tests has apparently always been failing but were incorrectly showing as passing due to the ... || kill cat port-forward.pid bit.
  • Fixed kind cluster routing that was causing these failures, also removes the ability to false report passing.
  • Added artifacts to the integration test workflow so on failure you can download the failure report since errors aren't printed to the terminal.
• hoptimator-operator was failing on startup because of two separate issues:
  • rbac was missing the "pipelines" resources
  • The hoptimator-operator K8s pod doesn't have access to .kube/config, needed its own mechanism to determine namespace information.

Testing:

Set up flink datagen connector to generate data and insert it into existing-topic-1.
Ran:

0: Hoptimator> create or replace materialized view kafka."existing-topic-2" as select * from kafka."existing-topic-1";

which setup a new Flink job to insert the data from existing-topic-1 into existing-topic-2. Was able to verify the data was inserted via CLI Kafka consumer.

Put many helpful commands in the README

Also make integration-tests-kind works locally as well if you setup a kind cluster locally.

[ryannedolan]

🔥🔥🔥

[ryannedolan]

this seems important :)

[ryannedolan]

wow totally missed this