Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] : Implementation of event streaming through Kafka #38

Open
majorbruteforce opened this issue Mar 2, 2024 · 10 comments
Open

[FEAT] : Implementation of event streaming through Kafka #38

majorbruteforce opened this issue Mar 2, 2024 · 10 comments

Comments

@majorbruteforce
Copy link
Member

majorbruteforce commented Mar 2, 2024

Events are any real world match instances which contain a snapshot of the match. For instance, in cricket: a ball, a wicket, a substitution, an interruption by nature, a new batsman entering the crease etc are all events.

This particular service will be responsible for accepting the data from client (admin), processing it (if required) and pushing it to other layers of services managing SSE connections to the clients (viewers).

Need for Kafka:

  • Ensuring no data is lost (consistent writes to DB )and the system is fault tolerant
  • Managing dead letter queues to ensure chronological delivery of events
  • Synchronising multiple microservices

Extras to be implemented:

  • Dedicated logger for Kafka
  • Unit and Integration testing

All implementations are to be done using the go library sarama

Refer past discussions regarding usage of Kafka and Sarama: #4

All queries and suggestions related to the event streaming service(s) can be done here.

@majorbruteforce
Copy link
Member Author

majorbruteforce commented Mar 2, 2024

CC: @soumil-kumar17 @zakhaev26 @punitkr03

@zakhaev26
Copy link
Member

lets roll with the implementation for event streams once i m done with the core services 💪🏻

@majorbruteforce
Copy link
Member Author

majorbruteforce commented Mar 2, 2024

lets roll with the implementation for event streams once i m done with the core services 💪🏻

Aye aye, Sarge!

@majorbruteforce
Copy link
Member Author

@zakhaev26 do share good resources that you have come across regarding Sarama

@zakhaev26
Copy link
Member

zakhaev26 commented Mar 3, 2024

From my experience, after going through various Kafka tutorials, I found implementing it in Go quite straightforward. I found the Sarama package to be more reliable than the packages provided by Confluent on Go. Even though there isn't a video specifically focused on implementing Sarama with Go, I managed to do it by referring to Sarama's documentation, GitHub examples, and some articles on Medium.

Here's a good video on the basics of Go + Kafka heer . (At around 11:10 in the video, the man also mentions that Sarama was a dominant package at the time)

BTW,i found a good stuff --> Transition sarama ownership

@zakhaev26
Copy link
Member

zakhaev26 commented Mar 3, 2024

BTW,are we looking into security measures of kafka?I think they provide a lot of things over security of kafka servers.

@majorbruteforce
Copy link
Member Author

BTW,are we looking into security measures of kafka?I think they provide a lot of things over security of kafka servers.

Sure why not.

@majorbruteforce
Copy link
Member Author

BTW,i found a good stuff --> Transition sarama ownership

Ahh. Interesting.

@majorbruteforce
Copy link
Member Author

Here's a good video on the basics of Go + Kafka heer . (At around 11:10 in the video, the man also mentions that Sarama was a dominant package at the time)

Yup was following this one. OG. (Heeer!)

@zakhaev26
Copy link
Member

zakhaev26 commented Mar 15, 2024

As the api services are almost completed with some essential components being added simultaneously, we shall start implementing the live streaming feature - the last piece of the puzzle.
As the POC has been made and explored,we should discuss about schemas and structure.
Lets get this done boys!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants