Help switch yfinance to API

[ValueRaider]

If interested in replacing web scraping with Yahoo API, post here. Of course you're welcome to code solo and submit a pull request, but this is a big job so an actual team might work better.

yfinance already uses API for price history and Ticker.info, so you can see coding not difficult it's just GET requests. The tricky part is discovering the URLs and parameters to use. The Yahoo webpages uses the same API so "just" need to inspect your browser traffic investigate how Yahoo webpage Javascript fetches data - e.g. Firefox Developer Tools debugger & network tabs.

[bfgoh]

Hi,

The Yahoo webpages uses the same API so "just" need to inspect your browser traffic.

Do you mean it by going to the browser, clicking on the developer mode, and under the network tab, find out the XHR that is sent to Yahoo?

[ValueRaider]

Yes that's what I meant, but I've just checked and you will also need to inspect their Javascript files (possibly only the Javascript), using e.g. the Firefox Developer debugger tab. I recommend first understanding how Ticker.info fetch works because that already uses API.

[bfgoh]

Okay. Not sure if this makes any sense, however, previously I was working on a project that was similar. That requires a full blown website to get an auth key from Developer debugger tab, with the token send a request to authenticate to the API.

The solution does not really require a browser, however, it requires a docker container instead.

https://github.com/scrapinghub/splash

So calls would be called through this splash container and it can return you .har file, which contains the information that you can find in the developer mode.

[ValueRaider]

That seems very complex, I just now managed to fix the annual cash-flow table using only Firefox.

[ericpien]

Following up from #154 (comment) - I'm interested in implementing the top gainers feature to resolve the issue.

Based on a quick check, it should be done by implementing screener (https://finance.yahoo.com/screener/) via API. It doesn't seem too big on its own so will 'code solo and submit PR'.

[ValueRaider]

Want me to delay release for this?

[ericpien]

I can get the PR up today but I don't think it's worth delaying a release as it's not a largely requested item in issues.

I will catch the next release for this. Thanks!