widget$status_code == 200 is not TRUE

[datacubed]

I've read through a lot of other people having this issue. I assume we're seeing this due to being "blocked" in some way by Google.

This interesting part for us is, we've been using this package+function to scrap keyword trend data for around 6+ months, but only this week we've been stopped.

Stranger still is we usually spin up differenet servers in our AWS cloud to run this, so we're usually gettting around the IP blocking problem.
Is it possible for Google (well anyone I guess) to block all traffic from our AWS account? Despite the boxes having different public Ip addresses?

Also, is it possible to get more of the error message? We're trying the tryCatch() func, but all we see is:

Error in get_widget(comparison_item, category, gprop, hl, cookie_url,  :
  widget$status_code == 200 is not TRUE

I'm not sure if there's more to print, possibly?

We tried using the python pytrends, and got hit with : The request failed: Google returned a response with code 429
So I assume we're being blocked.

As I said, I'm curious how we're being blocked regardless of different servers and IPs being used?

[mgahan-he]

I am having a similar issue. Just started about a week ago.

[eddelbuettel]

Folks we just discussed this yesterday. Can we please ask you to maybe also look at closed issues ?

Also note that a pinned issue is right there at the top.

[datacubed]

I saw that post, but it's from 2018 to last year, and the solutions discussed didnt work.

I just wanted to double check this is an issue with our "aws > network > google", rathern than the package changing recently.

Is there a way to see more about the errors in the package rather than just "status code not 200". It would be good to get the actual status code, if possible?

[janush1985]

Same issue here. I am now locked out with this error on 3 different machines!

I have previously encountered "widget$status_code == 200 is not TRUE" when making excessive requests,
but this time, if this is the root cause, the threshold appears lower.

Additionally, buddy working off same network as myself, is also getting same error even though he has not used the package or google trends in the last few days.

[mgahan-he]

@janush1985 same issue. Feels like there is a weird interaction going on with AWS and Google Trends API starting last week.

[dr-paul]

finding the same issue - started yesterday - and running R on my machine (without AWS)

[eddelbuettel]

Still works here (for an arguably casual user). I thought I could be cute and run res <- gtrends("GameStop") which, while taking a little longer than usual, still works. plot(res) yielded:

[datacubed]

Still works here (for an arguably casual user). I thought I could be cute and run res <- gtrends("GameStop") which, while taking a little longer than usual, still works. plot(res) yielded:

Is this from a local box or ec2 in aws ?

@janush1985 yeah same issue. Tried on multiple machines within AWS, seems like it is a weird connection issue there. I'm really not sure what the best approach/solution is here for us all ?

@mgahan-he yeah i think it's this too. But its hard (impossible ?) to find anything online about it. I'm not even sure what I would start a discussion about it - with the intention of looking for a resolution.

[PMassicotte]

I am not sure how I reproduce the issue to start digging into it. I will have to search for more information.

[janush1985]

I have been unblocked after ~24h and I am back up and running.

Below took 2.5 seconds on average to process:

time=rep(NA,10) for(i in 1:10){ a=Sys.time() temp=gtrends("gamestop") time[i]=as.numeric(Sys.time()-a) Sys.sleep(65) }

I have done some calculations and I was very likely exceeding 1000 calls within 24h period. This ban also affected other user on same network.

[eddelbuettel]

It really looks like "they" just block an entire network segment coming from AWS and there is exactly zero issue with gtrendsR per se so we really should close this.

[datacubed]

Yeah should close this, there is nothing wrong with gtrendsR.

But it might be worth linking to this or mentioning somewhere? As many people will be using the lib behind AWS via ec2s for ease of data pulling.