Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat:Proxy Support to Enhance Scraping Capabilities #1030

Closed
3 tasks done
thedarknight01 opened this issue May 21, 2024 · 1 comment
Closed
3 tasks done

Feat:Proxy Support to Enhance Scraping Capabilities #1030

thedarknight01 opened this issue May 21, 2024 · 1 comment

Comments

@thedarknight01
Copy link

thedarknight01 commented May 21, 2024

Describe the feature

Feature Request: Add Proxy Support to Scrape-Up

To improve the robustness and efficiency of Scrape-Up, I propose adding proxy support. This feature will help users scrape data from the internet more effectively by reducing the risk of getting blocked by websites.

Benefits:

  • Avoid IP Blocking
  • Access Geo-Restricted Content
  • Improve Scraping Performance

Implementation Details

  • Prepare a Proxy List File: Create a text file with a list of proxy addresses, one per line.
  • Call the Function: Use the get_valid_proxy function to retrieve a valid proxy from the list.
  • Implement in Scraping Logic: Use the retrieved proxy in your web scraping requests to help distribute the load and avoid IP blocking.
I believe adding this feature will significantly enhance the usability and effectiveness of Scrape-Up. I am happy to help

Add ScreenShots

i2Uls
When we scrape websites, each request we make is sent from our known IP address. Making too many requests from the same IP can lead to being blocked by the website.
This proxy server uses proxy addresses to avoid getting blocked by websites.

Record

  • I agree to follow this project's Code of Conduct
  • I'm a GSSoC'24 contributor
  • I want to work on this issue
Copy link

Hi there! Thanks for opening this issue. We appreciate your contribution to this open-source project. We aim to respond or assign your issue as soon as possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant