PR for feature #58 (access private datasets and be able to upsert and full replace datasets) #64

marks · 2015-11-01T15:02:31Z

Please see issue (feature request) #58; This is the re-do of PR #63 since it was not correctly based on the dev branch

NOTE the new branch. Original work I did was on a branch that was not based on `dev` - Added `email` and `password` parameters to `returnData.R` and its dependent methods such as those in `errorHandling.R` and `metadata.R` - Added tests in `test-readPrivateDataset.R` and ran `document()` - Was able to run `build()` successfully

- created `writeData.R` which defines `write.socrata()` - Added tests in `test-writeData` and ran `document()` - Was able to run `build()` successfully

marks · 2015-11-01T15:03:08Z

I am seeing some tests fail locally but I think it is code not related to my changes in the dev branch. Please do let me know if this is not the case!

dmpe · 2015-11-01T15:48:06Z

https://travis-ci.org/Chicago/RSocrata/builds/88642764#L1424

dmpe · 2015-11-01T15:48:31Z

R/writeData.R

+}
+
+
+#' @description Method for updating Socrata datasets 


title is missing

@dmpe

as pointed out by @dmpe at Chicago#64 (comment) thanks!

marks · 2015-11-02T11:09:04Z

How may I transmit the username/password used for the tests that require a Socrata login? I have been able to successfully test on my own travis-ci.org repo using environment variables.

Thanks!

dmpe · 2015-11-02T15:19:00Z

http://kieranhealy.org/blog/archives/2015/10/16/using-containerized-travis-ci-to-check-r-in-rmarkdown-files/ may help or again what i said in #58

marks · 2015-11-02T15:21:27Z

You can't use containerized Travis unless sudo isn't required.. I also think it seems an admin of the repo is best off just adding the environment variables from the Travis interface

Mark Silverberg
(m) 512 826 7004
http://twitter.com/skram

On Nov 2, 2015, at 10:19 AM, John Malc [email protected] wrote:

http://kieranhealy.org/blog/archives/2015/10/16/using-containerized-travis-ci-to-check-r-in-rmarkdown-files/ may help or again what i said in #58

—
Reply to this email directly or view it on GitHub.

dmpe · 2015-11-02T15:25:37Z

oh, i forgot, sorry.

---------- Původní zpráva ----------

Od: Mark Silverberg [email protected]

Komu: Chicago/RSocrata [email protected]

Datum: 2. 11. 2015 16:21:35

Předmět: Re: [RSocrata] PR for feature #58 (access private datasets and be
able to upsert and full replace datasets) (#64)

"
You can't use containerized Travis unless sudo isn't required.. I also think
it seems an admin of the repo is best off just adding the environment
variables from the Travis interface

Mark Silverberg

(m) 512 826 7004

http://twitter.com/skram

On Nov 2, 2015, at 10:19 AM, John Malc [email protected] wrote:

http://kieranhealy.org/blog/archives/2015/10/16/using-containerized-travis
-ci-to-check-r-in-rmarkdown-files/ may help or again what i said in #58

—

Reply to this email directly or view it on GitHub.

—

Reply to this email directly or view it on GitHub
(#64 (comment)).

"=

dmpe · 2015-11-06T18:17:48Z

cc @tomschenkjr

tomschenkjr · 2015-11-16T05:01:48Z

Still reviewing this pull request. Some issues are showing-up in the process. Here are some quick notes so far:

read.socrata(url = "https://data.cityofchicago.org/Health-Human-Services/Food-Inspections/4ijn-s7e5") generates an error because of the rowCount line. Specifically, I'm trying to understand the grander concept around getMetadata(), which is trying to fetch a lot of information, but is using little of it. Does it make sense to just keep it simple to help troubleshooting?
read.socrata(url = "https://data.cityofchicago.org/Buildings/Building-Permits/ydr8-5enu") is rather FUBAR, but this is an existing problem. For a reason I cannot determine yet, the list is not being properly transformed into a data frame. Instead, it's being transformed into an ugly list.

There are some other issues, but am looking at these two first.

marks · 2015-11-16T12:53:51Z

@tomschenkjr: I am seeing those errors as well but I am not confident they are part of this PR. I am getting those errors on the dev branch (which doesn't have this PR merged in). Do let me know how I can help though!

Thanks,

Mark

tomschenkjr · 2015-11-16T14:10:11Z

Yeah, it's unrelated to the write functions you added so as an FYI at this point. Once I clear up those other issues I'll be looking at these new functions.

Sent by Outlookhttp://taps.io/outlookmobile for Android

On Mon, Nov 16, 2015 at 4:54 AM -0800, "Mark Silverberg" <[email protected]mailto:[email protected]> wrote:

@tomschenkjrhttps://github.com/tomschenkjr: I am seeing those errors as well but I am not confident they are part of this PR. I am getting those errors on the dev branch (which doesn't have this PR merged in)

Thanks,

Mark

Reply to this email directly or view it on GitHubhttps://github.com//pull/64#issuecomment-157020153.

This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail (or the person responsible for delivering this document to the intended recipient), you are hereby notified that any dissemination, distribution, printing or copying of this e-mail, and any attachment thereto, is strictly prohibited. If you have received this e-mail in error, please respond to the individual sending the message, and permanently delete the original and any copy of any e-mail and printout thereof.

dmpe · 2015-11-16T15:01:21Z

Any issue now can be profiled using https://github.com/rstudio/profvis.

devtools::install_github("rstudio/profvis")
library(profvis)
library(RSocrata)

profvis({
  g <- read.socrata(url = "https://data.cityofchicago.org/Health-Human-Services/Food-Inspections/4ijn-s7e5")
  print(g)
})

profvis({
  b <- read.socrata(url = "https://data.cityofchicago.org/Buildings/Building-Permits/ydr8-5enu")
  print(b)
})

For the second issue e.g. like this :

Specifically, I'm trying to understand the grander concept around getMetadata(), which is trying to fetch a lot of information, but is using little of it.

Yes, there are 2 ways: Either you use execute another query on the desired dataset with =Count(*) (getQueryRowCount does that) or you fetch that information from the socrata's undocumented columns.json view. Here, @marks can provide us some help and tell what is better and more realible (I guess the first one, otherwise there would be no such method). In ither case, we can use the first option only and provide the second (from me) as just another function for end-users.

marks · 2015-11-16T17:41:25Z

@dmpe - i would suggest using the COUNT(*) as opposed to using columns.json. Documented metadata APIs coming soon!

dmpe · 2015-11-16T18:49:27Z

Thanks, one issue solved. Some code changes will be needed. Assumption must be made that if from me then at earliest in Christmas break.

With the second issue, @marks can also try help us. (Not so helpful and not so nicely summarized and I am sorry for it.) Do you know why do some datasets (and btw. some not; see above) have an "invalid" json file, which may cause our issues. (well, I guess these were attempts to fix them: #19 #3 )

library(jsonlite)
df<-jsonlite::fromJSON("https://data.cityofchicago.org/resource/ydr8-5enu.json")

Cannot test it now, but as i remember it was failing at least some time ago.

marks · 2015-11-17T14:57:11Z

@dmpe - could you help me understand this issue by sharing a list of datasets that are giving invalid JSON? I am very confident all JSON that we produce and put out fits the standard but it might not be exactly what jsonlite/R is expecting. I just need a little more info to be able to look into this more with the team. I will likely end up creating an issue at https://github.com/socrata/dev.socrata.com/issues/new if you want to go ahead and do that instead of posting here. Whatever is best for you!

dmpe · 2015-11-17T17:02:47Z

Yeeh, code from #3 is blocked so i cannot test it anymore.
At least these are working, now. Most probably an error in our code.

df<-jsonlite::fromJSON("https://data.cityofchicago.org/resource/ydr8-5enu.json")
df1<-jsonlite::fromJSON("https://data.cityofchicago.org/resource/kn9c-c2s2.json")
df1<-jsonlite::fromJSON("https://data.cityofchicago.org/resource/4ijn-s7e5.json")
dfs1<-jsonlite::fromJSON("http://data.undp.org/resource/wxub-qc5k.json")

dmpe · 2015-11-18T21:01:22Z

After spending huge amount of time on it, I start to think the error is comming somewhere from httr or curl package.
In fact, when I run code below, I got this:

gM3 <- getMetadata(url = "https://data.cityofchicago.org/resource/6zsd-86xi.json")
 Show Traceback

 Rerun with Debug
 Error in curl::curl_fetch_memory(url, handle = handle) : 
  Couldn't resolve host name

But the same code for

gM2 <- getMetadata(url = "https://data.cityofboston.gov/resource/awu8-dc52")

is not an issue. WTF is it (with my PC) ?

…ow splitted.

marks · 2015-12-23T18:57:04Z

@dmpe @tomschenkjr is there anything I can do to help keep this moving forward?

Appreciate all your hard work!

tomschenkjr · 2015-12-24T06:20:49Z

I think we have a bit of scope creep in this PR, but I'm going to continue to evaluate the parts of it to clean-up the errors. Hope to do it this week.

Tom Schenk Jr.

Chief Data Officer

Department of Innovation and Technology

City of Chicago

(312) 744-2770

[email protected]

data.cityofchicago.org

From: Mark Silverberg [email protected]
Sent: Wednesday, December 23, 2015 12:57 PM
To: Chicago/RSocrata
Cc: Schenk, Tom
Subject: Re: [RSocrata] PR for feature #58 (access private datasets and be able to upsert and full replace datasets) (#64)

@dmpehttps://github.com/dmpe @tomschenkjrhttps://github.com/tomschenkjr is there anything I can do to help keep this moving forward?

[https://avatars3.githubusercontent.com/u/1799661?v=3&s=400]https://github.com/dmpe

dmpe (John Malc) · GitHubhttps://github.com/dmpe
github.com
dmpe has 26 repositories written in R, Java, and CSS. Follow their code on GitHub.

Appreciate all your hard work!

Reply to this email directly or view it on GitHubhttps://github.com//pull/64#issuecomment-166970369.

This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail (or the person responsible for delivering this document to the intended recipient), you are hereby notified that any dissemination, distribution, printing or copying of this e-mail, and any attachment thereto, is strictly prohibited. If you have received this e-mail in error, please respond to the individual sending the message, and permanently delete the original and any copy of any e-mail and printout thereof.

…ow splitted. update description file & squash all the commits

Mark Silverberg added 2 commits November 1, 2015 08:59

Chicago#58 support for writing to datasets (full replace and upset)

772a01a

- created `writeData.R` which defines `write.socrata()` - Added tests in `test-writeData` and ran `document()` - Was able to run `build()` successfully

dmpe reviewed Nov 1, 2015
View reviewed changes

R/writeData.R

}

#' @description Method for updating Socrata datasets

Copy link

Contributor

dmpe Nov 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

title is missing

Mark Silverberg added 2 commits November 1, 2015 11:52

Chicago#58 Chicago#63 add method title

5201e79

as pointed out by @dmpe at Chicago#64 (comment) thanks!

Chicago#58 Chicago#64 addressing documentation check warnings

e4ee270

dmpe added a commit to dmpe/RSocrata that referenced this pull request Nov 18, 2015

this fixes metadata methods so that Chicago#64 (comment) rowline is n…

5c426e6

…ow splitted.

dmpe mentioned this pull request Nov 18, 2015

Replace getMetadata for row count with getQueryRowCount (which uses =COUNT(*)) #65

Closed

dmpe added a commit to dmpe/RSocrata that referenced this pull request Dec 24, 2015

this fixes metadata methods so that Chicago#64 (comment) rowline is n…

683e1cc

…ow splitted. update description file & squash all the commits

tomschenkjr closed this Jan 11, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PR for feature #58 (access private datasets and be able to upsert and full replace datasets) #64

PR for feature #58 (access private datasets and be able to upsert and full replace datasets) #64

marks commented Nov 1, 2015

marks commented Nov 1, 2015

dmpe commented Nov 1, 2015

dmpe Nov 1, 2015

marks commented Nov 2, 2015

dmpe commented Nov 2, 2015

marks commented Nov 2, 2015

dmpe commented Nov 2, 2015

dmpe commented Nov 6, 2015

tomschenkjr commented Nov 16, 2015

marks commented Nov 16, 2015

tomschenkjr commented Nov 16, 2015

dmpe commented Nov 16, 2015

marks commented Nov 16, 2015

dmpe commented Nov 16, 2015

marks commented Nov 17, 2015

dmpe commented Nov 17, 2015

dmpe commented Nov 18, 2015

marks commented Dec 23, 2015

tomschenkjr commented Dec 24, 2015

PR for feature #58 (access private datasets and be able to upsert and full replace datasets) #64

PR for feature #58 (access private datasets and be able to upsert and full replace datasets) #64

Conversation

marks commented Nov 1, 2015

marks commented Nov 1, 2015

dmpe commented Nov 1, 2015

dmpe Nov 1, 2015

Choose a reason for hiding this comment

marks commented Nov 2, 2015

dmpe commented Nov 2, 2015

marks commented Nov 2, 2015

dmpe commented Nov 2, 2015

dmpe commented Nov 6, 2015

tomschenkjr commented Nov 16, 2015

marks commented Nov 16, 2015

tomschenkjr commented Nov 16, 2015

dmpe commented Nov 16, 2015

marks commented Nov 16, 2015

dmpe commented Nov 16, 2015

marks commented Nov 17, 2015

dmpe commented Nov 17, 2015

dmpe commented Nov 18, 2015

marks commented Dec 23, 2015

tomschenkjr commented Dec 24, 2015