-
Notifications
You must be signed in to change notification settings - Fork 238
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Spark Streaming using Spark Operator #516
Conversation
streaming/spark-streaming/examples/consumer/manifests/01_spark_application.yaml
Outdated
Show resolved
Hide resolved
streaming/spark-streaming/examples/consumer/manifests/01_spark_application.yaml
Outdated
Show resolved
Hide resolved
streaming/spark-streaming/examples/consumer/manifests/00_rbac_permissions.yaml
Outdated
Show resolved
Hide resolved
streaming/spark-streaming/examples/consumer/manifests/01_spark_application.yaml
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work on this PR @lusoal ! PR looks good to me. I presume that you have tested the blueprint end to end. Please ask some of our internal teams to run through the instructions. I would also add a warning to the website doc for any security requirements that needs to be tighten if they implement this in prod.
Yes I tested indeed! Will ask to someone else to execute the steps. Thanks @vara-bonthu |
streaming/spark-streaming/examples/consumer/manifests/01_spark_application.yaml
Show resolved
Hide resolved
…e, added dynamic allocation and prometheus annotations to manifests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
What does this PR do?
🛑 Please open an issue first to discuss any significant work and flesh out details/direction - we would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.
This will provision a stack for Spark streaming from Amazon MSK, doesn't need to be from MSK, could be replaced with Kafka in strimzi for example. The most value of the solution is showing examples of applications working in this stack using the operator and spark streaming.
Motivation
Customer requested PoC
More
website/docs
orwebsite/blog
section for this featurepre-commit run -a
with this PR. Link for installing pre-commit locallyFor Moderators
Additional Notes