Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎯 New Source Connector Request: Twitter #251

Open
ehisakhile opened this issue Mar 7, 2023 · 2 comments
Open

🎯 New Source Connector Request: Twitter #251

ehisakhile opened this issue Mar 7, 2023 · 2 comments
Assignees
Labels

Comments

@ehisakhile
Copy link
Contributor

ehisakhile commented Mar 7, 2023

Source Connector Twitter

What is a Vanus Connector

Vanus Connect allows you to skip the complex integration with external services by offering out-of-the-box connectors. Each connector acts as a proxy between outside services and your system, enabling users to have faster integrations with no codes. All data produced or processed by our connectors conform to the CloudEvents specification, which helps enterprises leverage event-driven architecture to do business in the age of events.

What is a Source Connector

A Source connector obtains data from an underlying data producer and delivers it to the target after the original data has been transformed into a CloudEvents. For example:

  • MySQL Source: extracts data from the Binlog, transforms it into CloudEvents, and sends it to the target.

  • GitHub Source: uses a Webhook server to receive the event from GitHub, transform them to CloudEvents and send them to the target.

  • AWS S3 Source: Uses the AWS SDK to get the data, transforms it into CloudEvents, and sends them to the target.

There are two types of connectors push or pull connectors.
A push connector is passive and only does its logic when it receives an event, and a pull connector must pull the data in an interval. We use a push method when it’s possible.

Features of this Source Connector

  • Receive events from Twitter when a user’s tweet is liked, retweeted, quote-tweeted, DMs, tweet comments and put the data into a CloudEvent to send it to the target
  • Each event type (like, dm, comment, etc ...) Should be specified in the event type specification of the CloudEvent.
  • Configuration parameters

Specific data needed

When someone likes the user's tweet.

  • Tweet ID: The unique identifier for the tweet that was liked.
  • User ID: The unique identifier for the user who liked the tweet.
  • User screen name: The screen name (i.e., username) of the user who liked the tweet.
  • User name: The full name of the user who liked the tweet.
  • Tweet text: The text of the tweet that was liked.
  • Like creation time: The timestamp indicating when the like was created.
  • Tweet URL: The URL of the tweet that was liked.
  • Number of likes: The total number of likes the tweet has received.
  • Number of retweets: The total number of retweets the tweet has received.
  • Number of replies: The total number of replies the tweet has received.

When someone retweets or quote retweets

  • Tweet ID: The unique identifier for the tweet that was retweeted.
  • User ID: The unique identifier for the user who retweeted the tweet.
  • User screen name: The screen name (i.e., username) of the user who retweeted the tweet.
  • User name: The full name of the user who retweeted the tweet.
  • Retweet ID: The unique identifier for the retweet itself.
  • Retweet text: The text of the retweet, if any.
  • Retweet creation time: The timestamp indicating when the retweet was created.
  • Tweet URL: The URL of the original tweet.
  • Number of likes: The total number of likes the original tweet has received.
  • Number of retweets: The total number of retweets the original tweet has received.
  • Number of replies: The total number of replies the original tweet has received.

When someone Dms the user

  • DM ID: The unique identifier for the DM.
  • Sender ID: The unique identifier for the user who sent the DM.
  • Sender screen name: The screen name (i.e., username) of the user who sent the DM.
  • Sender name: The full name of the user who sent the DM.
  • Recipient ID: The unique identifier for the user who received the DM (i.e., you).
  • DM text: The text of the DM.
  • DM creation time: The timestamp indicating when the DM was created.

When someone replies to the user's tweet

  • Tweet ID: The unique identifier for the tweet that was replied to.
  • User ID: The unique identifier for the user who replied to the tweet.
  • User screen name: The screen name (i.e., username) of the user who replied to the tweet.
  • User name: The full name of the user who replied to the tweet.
  • Reply ID: The unique identifier for the reply itself.
  • Reply text: The text of the reply.
  • Reply creation time: The timestamp indicating when the reply was created.
  • Tweet URL: The URL of the original tweet.
  • Number of likes: The total number of likes the original tweet has received.
  • Number of retweets: The total number of retweets the original tweet has received.
  • Number of replies: The total number of replies the original tweet has received.

How to create a new connector

  1. Fork the vanus-connect repo.
  2. Create a source_twitter_proposal.md based on the source_proposal_example.md under the vanus-connect/proposals directory. The proposal will introduce how a developer tends to implement the connector in detail. The proposal PR must be approved and merged before going to the next step.
  3. Copy one of the source templates (Golang template, Java template) to vanus-connect/connectors/ and rename it as source-twitter.
  4. Implement your connector and submit a PR.
  5. Wait for the connector to be reviewed and approved.

How to claim to implement this connector

If you want to solve this issue, please leave a comment on this issue like:

I'd like to implement the connector, please assign it to me.

Vanus community will assign the issue to you on time.

@Michaelg22 Michaelg22 changed the title New Source Connector Request: Twitter 🎯 New Sink Connector Request: Twitter Mar 7, 2023
@allensuvorov
Copy link
Contributor

I'd like to implement the connector, please assign it to me.

@ehisakhile ehisakhile changed the title 🎯 New Sink Connector Request: Twitter 🎯 New Source Connector Request: Twitter Mar 7, 2023
@allensuvorov
Copy link
Contributor

As an update. Just rechecked Twitter API
Looks like, at this point, sourcing data requires a paid account:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants