Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Kinesis streams for SPARK computations #1053

Open
XxChauhanxX opened this issue Feb 13, 2019 · 1 comment
Open

Support for Kinesis streams for SPARK computations #1053

XxChauhanxX opened this issue Feb 13, 2019 · 1 comment

Comments

@XxChauhanxX
Copy link

XxChauhanxX commented Feb 13, 2019

Child of #1089

As a Product owner I want to make Tempus application in Native AWS, since most of the customers prefers and trusts the Native AWS architecture and applications.

ACs:

  1. Need to start supporting Kinesis streaming for SPARK computations.
@pal-abhishek
Copy link
Contributor

Possible approach to do the same:-

-> Existing computations will be refactored to implement three steps of spark computation (load, transform and extract).
-> The loaders will be plug-able (kafka or kinesis) depending upon the selected value on UI.
-> Descriptor will have parameters for both kafka and kinesis.
-> The form loading of the UI will have some changes depending on the selected value (kafka or kinesis). We need to the values associated to the specific stream.
-> The contract of loader and transform can be defined in a separate module in Tempus-Extensions. The contract will be used to implement loader or transformer (current computations). The module can be repackaged in Uber jar for uploading in Tempus.
-> Combining the three implementation spark Pipeline can be created Ex - Pipeline(KafkaLoader, drillBitCompulationTranformer, MqttPublisher).
-> These will be done by driver, only the transformer will use the parallel processing across nodes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants