Skip to content

Commit

Permalink
Rewrite entire requests backend with rust using the PyO3 binding.
Browse files Browse the repository at this point in the history
- Add a SessionRs struct/class to efficiently handle requests.
- Add a HttpResponse struct/class to store response data and handle
decompression
Additional Updates
- Remove ip scanning functions
- Move all tor related functions to the tor module for more
intuitive library structure
- Remove autosession functionality as it provided less impressive
performence improvement compared to rust
  • Loading branch information
akneni committed Jan 13, 2024
1 parent b5b09e9 commit 232e336
Show file tree
Hide file tree
Showing 10 changed files with 860 additions and 504 deletions.
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
/pygrab/tester.*
/pygrab/test.*
/pygrab/unit-tester.*
/pygrab/unit-tester.*
/pygrab/rust_dependencies/*.dll
/pygrab/rust_dependencies/*.so
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# PyGrab

PyGrab is a Python library for making HTTP requests. It is designed to be fully interoperable with the popular `requests` package in Python. All functions in PyGrab that make HTTP requests return `requests.Response` objects, making it easy to integrate into existing projects that use `requests`.
PyGrab is a simple and performant Python library for making HTTP requests written in rust. It is designed to be fully interoperable with the popular `requests` package in Python.


## References
Expand All @@ -13,12 +13,16 @@ PyGrab is a Python library for making HTTP requests. It is designed to be fully

- **Simple**: PyGrab's API is designed to be straightforward and easy to use. If you're familiar with `requests`, you'll feel right at home.
- **Flexible**: Whether you need to make a simple GET request, post data to a server, or download a file, PyGrab has you covered.
- **Performant**: With its Rust-powered backend, PyGrab offers enhanced performance for CPU-bound tasks, providing faster data decompression, thread handling, and network handling..
- **Asynchronous Support**: PyGrab includes functions for making asynchronous HTTP requests, allowing you to efficiently grab data from multiple URLs at once.
- **JavaScript Support**: PyGrab can render JavaScript-enabled websites, allowing you to grab data from dynamic web pages.
- **Interface with Tor Network**: PyGrab includes built-in support for routing requests through the tor network.
- **Automated IP Rotation**: PyGrab includes built-in support for rotating connections to the Tor Network.


## Limitationa
- PyGrab is only supported on windows an linux currently.

## Installation

You can install PyGrab using pip:
Expand All @@ -38,7 +42,7 @@ response = pygrab.get('http://example.com')
print(response.text)
```

In this example, `response` is a `requests.Response` object. You can use it just like you would in `requests`.
In this example, `response` is syntactically similar to the `requests.Response` object. You can use it just like you would in `requests`.

Here's an example of how to use PyGrab to make asynchronous GET requests through the Tor Network:

Expand All @@ -54,7 +58,7 @@ for response in responses.values():
print(response.text)
```

In this example, `responses` is a dictionary of urls matched to their respective `requests.Response` objects. Each response corresponds to the URL at the same index in the `urls` list.
In this example, `responses` is a dictionary of urls matched to their respective `pygrab.HttpResponse` objects. Each response corresponds to the URL at the same index in the `urls` list.


## Contributing
Expand Down
3 changes: 2 additions & 1 deletion pygrab/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
from .pygrab import *
from .pygrab import *
from pygrab.rust_dependencies.rust_lib import SessionRs, HttpResponse
161 changes: 0 additions & 161 deletions pygrab/autosession.py

This file was deleted.

Loading

0 comments on commit 232e336

Please sign in to comment.