Mitigating Twitter rate-limit with mobile endpoints?

[Infinitay]

I've noticed that anytime I am rate-limited on Twitter as indicated by their desktop website, I can't use the site as normal. I will list of most of the things that I can/can not do on Twitter's desktop website versus the mobile iOS app.

tl;dr I can use Twitter as normal on the iOS app when rate-limited whereas on the Desktop website I can't do much if anything

Twitter Desktop vs Mobile Application

Common issues/rate-limits faced on Twitter's desktop website

• View my feed (home/following/lists)
  • Sometimes a few latest tweets show up, but then at the top it says "Unlock more posts by subscribing", and I can't scroll down.
  • Other times I can't see any latest tweet and am met with just the "Unlock more posts by subscribing. You have reached the limit for seeing posts today. Subscribe to see more posts every day." message.
• View a specific user's feed (via "Posts" tab)
  • https://twitter.com/<user_name>
• View a specific user's likes (via "Likes" tab)
  • https://twitter.com/<user_name>/likes
• View the latest search results (via "Latest" tab)
  • https://twitter.com/search?q=<search_query>&src=recent_search_click&f=live
• View the tweets directly in any of my lists
  • https://twitter.com/i/lists/<list_id>
  • You can't view your lists either on your home feed or directly via the "Lists" tab
• View the parent (replied-to) or children (replies) of a specific tweet
  • https://twitter.com/<user_name>/status/<tweet_id>
  • Typically met with "Sorry, you are rate limited. Please wait a few moments then try again."

The list probably goes on, but these are some of the things I have experienced personally. Now, there are ways to get around some of these rate-limits on the desktop version.

Possible caveats and tricks to getting around some desktop rate-limits

• Unable to view a specific user's feed (via "Posts" tab)
  • You can instead view the user's "Replies" tab as sometimes the tweets will be able to load here instead. Some of the content you see could be different, but majority of the tweets are there and viewing tweets isn't fully blocked by the rate-limit.
    • https://twitter.com/<user_name>/with_replies
  • You can instead view the user's "Media" tab as sometimes the tweets will be able to load here instead. Some of the content you see will different because it will only show tweets from the user containing media, but viewing tweets isn't fully blocked by the rate-limit.
    • https://twitter.com/<user_name>/media
• View the latest search results (via "Latest" tab)
  - You can instead view the top search results. It's not the same as the latest, but at least there are some tweets listed here that otherwise wouldn't be. I suppose you could further restrict this to get latest tweets with a search filter to get tweets from today from a specific user
  - https://twitter.com/search?q=<search_query>&src=recent_search_click
• Viewing specific tweets, replied-to, or replies
  • You need to have the Twitter app on a mobile device for this
  • If you view the tweet in question on your mobile app, you can bookmark the tweet on the app and then visit your bookmarks on Twitter desktop to view the tweet in question

Now you can see there are a lot of restrictions in place when using Twitter on the desktop website as a result of the rate-limit. Sometimes the rate-limit lasts for a few minutes, sometimes tweets occasionally load as your rate-limits replenish. Even when writing this post I was rate-limit and when I refreshed my home page again, one tweet loaded for me, but when I refreshed again, I was met with the rate-limit instantly. Despite all these rate-limits and restrictions on the desktop version of Twitter, you can use the mobile app with little to no problem — at least that is the case on the iOS app. Things that you would be normally faced with a rate-limit for on desktop and unable to load, you would be able to view and load the tweets on the app instead with no problem.

Things you can do on the mobile app that you wouldn't be able to on desktop when facing the rate-limit

• View my feed (home/following/lists)
• View a specific user's feed (via "Posts" tab)
• View a specific user's likes (via "Likes" tab)
• View the latest search results (via "Latest" tab)
  • Sometimes you will be met with a something went wrong please try again error, but you could still view latest tweets unlike on desktop where it refuses to load at all
• View the tweets directly in any of my lists
  • You can view them on your home feed or directly via the lists tab
• View the parent (replied-to) or children (replies) of a specific tweet

Conclusion

In conclusion, using Twitter's mobile app lets you mitigate any rate-limits you are facing whereas on Twitter's desktop website you will be met with errors. If I had to assume, they do this because (they assume) mobile users are real users and not scraping tweets or otherwise making api calls for other purposes than to use their app. That's why I think it would be beneficial to add support for using Twitter's mobile endpoints for gallerydl's Twitter module so that users could mitigate the rate-limit. At the very least, I think that there should be an option disabled by default to prevent abuse for gallerydl to fallback on Twitter's mobile endpoint when facing rate-limits.

[Infinitay]

I hope this is appropriate to do, but I wanted to cc some existing issues

#4353 #5330