Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate if we can easily import requests only if necessary #269

Open
whimboo opened this issue Mar 11, 2015 · 2 comments
Open

Investigate if we can easily import requests only if necessary #269

whimboo opened this issue Mar 11, 2015 · 2 comments

Comments

@whimboo
Copy link
Contributor

whimboo commented Mar 11, 2015

As seen on PR #222 it takes about 2s to import the requests module. That's kinda long and especially if we want to get the bash completion working, where we actually do not need this module at all. I wonder if it would be helpful to only import this module if really necessary in the scraper.py module.

@Nebelhom
Copy link
Collaborator

Nebelhom commented May 9, 2015

@whimboo

FYI.
There are 5 occurrences (apart from the import statement) where we make use of requests, 3 of which are except requests.exceptions.RequestException.
If we can circumvent those, we would only need to import it for the download method and when the DailyScraper is being used...

Here are the lines where we are currently using the requests module in the code:

...Under Scraper.__init__
except (NotFoundError, requests.exceptions.RequestException), e:
https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l146

...Under Scraper.binary
except (NotFoundError, requests.exceptions.RequestException), e:
https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l191

...Under Scraper.download
r = requests.get(self.final_url, stream=True,
https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l309

and
except (requests.exceptions.RequestException, TimeoutError), e:
https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l344

...and finally under DailyScraper.get_latest_build_date
r = requests.get(url + parser.entries[-1], auth=self.authentication, headers=headers)
https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l431

@whimboo
Copy link
Contributor Author

whimboo commented May 29, 2015

...Under |Scraper.init|
|except (NotFoundError, requests.exceptions.RequestException), e:|
https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l146

On which base exception does this exception class base on? Maybe we
could this one.

...Under |Scraper.binary|
|except (NotFoundError, requests.exceptions.RequestException), e:|
https://github.com/mozilla/mozdownload/blob/master/mozdownload/scraper.py#l191

Would we have to make thoughts about those at all? Only the first
pointed out location is part of the init method, and will always be
called. In terms of download or the binary it shouldn't matter given
that for command line auto-complete those methods wouldn't be called, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants