GeoDataFrame init #909

AndreiKingsley · 2024-10-04T09:15:38Z

No description provided.

Jolanrensen

good initial version :) just some comments here and there

Jolanrensen · 2024-10-08T18:17:56Z

dataframe-geo/build.gradle.kts

+    implementation(libs.ktor.serialization.kotlinx.json)
+
+}
+


Is it still required to be a friend module?

Yep, need some internal stuff for integration

Only for the jupyter integration?

Jolanrensen · 2024-10-08T18:24:02Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/GeoDataFrame.kt

+import org.jetbrains.kotlinx.dataframe.api.with
+
+class GeoDataFrame<T : GeoFrame>(val df: DataFrame<T>, val crs: CoordinateReferenceSystem?) {
+    fun update(updateBlock: DataFrame<T>.() -> DataFrame<T>): GeoDataFrame<T> {


Did you consider other names? Like

updateDf

replaceDf (more consistent with what replace does in DF compared to update)

replace implies that we have df1 and df2, and we change df1 to df2. But in reality we are only modifying df1 slightly, using it as a receiver.

Btw, what do you think about modifyDf then?

@koperagen @zaleslaw

In my opinion "Df" is redundant and doesn't look good. Plus it's already clear from signature that it applies to DataFrame

Jolanrensen · 2024-10-08T18:24:50Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/GeoDataFrame.kt

+        return GeoDataFrame(df.updateBlock(), crs)
+    }
+
+    fun applyCRS(targetCRS: CoordinateReferenceSystem? = null): GeoDataFrame<T> {


Proper camel casing would be "applyCrs" and "targetCrs"

Jolanrensen · 2024-10-08T18:47:38Z

dataframe-geo/build.gradle.kts

+    with(libs.plugins) {
+        alias(kotlin.jvm)
+        alias(publisher)
+        alias(jupyter.api)


Could you include ktlint too?

Ok, but comment it because it adds too many linting warnings

Jolanrensen · 2024-10-08T18:49:44Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/geocode/Geocoder.kt

+
+object Geocoder {
+
+    private val url = "https://geo2.datalore.jetbrains.com/map_data/geocoding"


Should this be included in the stock library? As far as I remember this was a reverse engineering endeavour and since it depends on an external URL behaviour might change or stop working entirely

For this experimental module, I am totally fine with including in library.

It's stable for enough time

We not guarantee for a while all our releases of experimental geo-module

But we could also include this solution for the @Jolanrensen question:

Checking the API availability and handling the unavailability with some exception

Checking the API consistency and notification for the users or raise an exception

Customize it with a URL parameter (if this API just will be moved in another place)

is it true, that the main problem was that we could not add fileds by some reason to the descendants of the DataFrame? @Jolanrensen @koperagen

Jolanrensen · 2024-10-09T09:36:39Z

settings.gradle.kts

@@ -41,3 +41,4 @@ plugins {
 }
 include("dataframe-excel")
 include("core")
+include("dataframe-geo")


if you want dataframe-geo to be imported by default when users depend on the generic dataframe package, also add dataframe-geo in build.gradle.kts

No, I'd like to be it separated

Jolanrensen · 2024-10-09T09:39:38Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/jupyter/IntegrationGeo.kt

+        return execute(codeWithConverter, variableName)
+    }
+
+    override fun Builder.onLoaded() {


what does this jupyter integration do exactly? could you add some docs?

Hmm, ok, but there's no docs for core integration

Maybe a test like below will make things clearer
for reference:
org.jetbrains.kotlinx.dataframe.jupyter.DataFrameJupyterTest and org.jetbrains.kotlinx.dataframe.jupyter.CodeGenerationTests

test:

val geo = **create geo** val updatedGeo = geo.update { add("col") { 1 } } updatedGeo.df.col

Hmm, ok, but there's no docs for core integration

That's not a good argument haha. We also need kdocs there :)

Jolanrensen · 2024-10-09T09:41:01Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/io/read.kt

+import java.net.URL
+
+
+fun GeoDataFrame.Companion.readGeoJSON(path: String): GeoDataFrame<*> {


readGeoJson is proper camel casing

Ok, but geotools use GeoJSON.

Jolanrensen · 2024-10-09T09:42:30Z

...eo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/geotools/toSimpleFeatureCollection.kt

+    typeBuilder.name = name ?: "geodata"
+    typeBuilder.setCRS(crs)
+    val geometryClass = if (singleGeometryType) {
+        // todo singleOrNull() ?: error()


many todo's haha

I believe it's ok since it's very alpha.

Jolanrensen · 2024-10-09T09:43:39Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/geotools/ToGeoDataFrame.kt

+import org.jetbrains.kotlinx.dataframe.geo.GeoFrame
+import org.locationtech.jts.geom.Geometry
+
+fun SimpleFeatureCollection.toGeoDataFrame(): GeoDataFrame<*> {


please rename the file to "toGeoDataFrame" like you did with the other file

zaleslaw

Generally, I am approving these changes, but want to summarize some thoughts and make this piece of code more mature and maintanable

zaleslaw · 2024-11-04T15:49:42Z

dataframe-geo/build.gradle.kts

+repositories {
+    // geo repositories should come before Maven Central
+    maven("https://maven.geotoolkit.org")
+    maven("https://repo.osgeo.org/repository/release")


Is it true, that these maven repo links will be hidden in the module, and the final user don't need to refer to them somehow in its Gradle script? (just for self checking)

Unfortunately, the user will need to add the repository. However, I believe we can fix this with shadow jar.

zaleslaw · 2024-11-04T15:50:13Z

dataframe-geo/build.gradle.kts

+    useJUnitPlatform()
+}
+kotlin {
+    jvmToolchain(11)


Could we downgrade it somehow to 8?

We have a global problem - DataFrame uses 8 and the Jupyter API uses 11. Therefore, in the future it will be necessary to allocate separate module(s) for integrations.

but for now, for convenience, I'd keep it 11

zaleslaw · 2024-11-04T15:56:24Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/jupyter/IntegrationGeo.kt

+import kotlin.reflect.full.isSubtypeOf
+import kotlin.reflect.typeOf
+
+internal class IntegrationGeo : JupyterIntegration() {


Let's define the final scope of this integration: what we want to achieve with that? Please add an explanation for this class.

zaleslaw · 2024-11-04T16:06:00Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/GeoFrame.kt

+import org.locationtech.jts.geom.Polygon
+
+@DataSchema
+interface GeoFrame {


The name is lead in the wrong direction. It's not GeoFrame. It's close to wrappers for Geometry or Form or GeometryForm

I agree, that geometry is produced from Geo, but it's misleading me

Because we have Geometry from jts, what do you think about Form or Figure or GeometryFigure/GeometryType or GeometryColumnType, verbose but clear

WithGeometry?

zaleslaw · 2024-11-04T16:11:18Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/geocode/Geocoder.kt

+
+object Geocoder {
+
+    private val url = "https://geo2.datalore.jetbrains.com/map_data/geocoding"


For this experimental module, I am totally fine with including in library.

It's stable for enough time

We not guarantee for a while all our releases of experimental geo-module

But we could also include this solution for the @Jolanrensen question:

Checking the API availability and handling the unavailability with some exception

Checking the API consistency and notification for the users or raise an exception

Customize it with a URL parameter (if this API just will be moved in another place)

zaleslaw · 2024-11-04T16:49:54Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/geocode/Geocoder.kt

+
+object Geocoder {
+
+    private val url = "https://geo2.datalore.jetbrains.com/map_data/geocoding"


is it true, that the main problem was that we could not add fileds by some reason to the descendants of the DataFrame? @Jolanrensen @koperagen

zaleslaw · 2024-11-04T16:56:02Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/jts/geometryExtenstions.kt

+import org.locationtech.jts.geom.Geometry
+import org.locationtech.jts.geom.util.AffineTransformation
+
+fun Geometry.scale(value: Double): Geometry {


Please document functions

zaleslaw · 2024-11-04T16:57:29Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/jts/geometryExtenstions.kt

+    return transformation.transform(this)
+}
+
+fun Geometry.translate(valueX: Double, valueY: Double): Geometry {


Could you consider renaming to somthing like shift/move for simplicity

I think "translate" is a common name in the math/geometry-world

the main issue, that we are doing this not for math guys, but for Kotlin developers who are not familiar with all these concepts, and I am trying to make the naming simpler and more obvious

it's a well-established concept, so I can't agree:
https://en.wikipedia.org/wiki/Translation_(geometry)

zaleslaw · 2024-11-04T16:58:51Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/GeoDataFrame.kt

+    }
+
+    companion object {
+        val DEFAULT_CRS = CRS.decode("EPSG:4326", true)


Could be ENUM of different constants

Yes, it even exists in some form in GeoTools (see DefaultEngineeringCRS). But here is a more global question - whether it is necessary to use geotools API directly or should we wrap it completely.

zaleslaw · 2024-11-04T17:00:50Z

dataframe-geo/src/main/kotlin/org/jetbrains/kotlinx/dataframe/geo/io/write.kt

+
+fun GeoDataFrame<*>.writeGeoJson(path: String): Unit = writeGeoJson(File(path))
+
+fun GeoDataFrame<*>.writeGeoJson(file: File) {


Please add for IO part some JUnit tests for reading/writing files close to your examples https://gist.github.com/AndreiKingsley/8eada06f174c681e953f8d939248d153

GeoDataFrame init

f0967bb

AndreiKingsley requested review from zaleslaw, koperagen and Jolanrensen October 4, 2024 09:15

AndreiKingsley added 2 commits October 4, 2024 13:32

GeoDataFrame add publication

3458ac5

Merge branch 'master' into geo

4bd8105

Jolanrensen requested changes Oct 9, 2024

View reviewed changes

devcrocod mentioned this pull request Oct 11, 2024

Support GeoDataFrame plotting. Kotlin/kandy#441

Open

3 tasks

zaleslaw requested changes Nov 4, 2024

View reviewed changes

AndreiKingsley added 2 commits November 6, 2024 02:02

PR changes & docs

b724f9d

fix crs equals checks

0ff0ee8

AndreiKingsley requested review from zaleslaw and Jolanrensen November 5, 2024 22:11


		object Geocoder {

		private val url = "https://geo2.datalore.jetbrains.com/map_data/geocoding"

		import java.net.URL


		fun GeoDataFrame.Companion.readGeoJSON(path: String): GeoDataFrame<*> {


		fun GeoDataFrame<*>.writeGeoJson(path: String): Unit = writeGeoJson(File(path))

		fun GeoDataFrame<*>.writeGeoJson(file: File) {

GeoDataFrame init #909

Are you sure you want to change the base?

GeoDataFrame init #909

Conversation

AndreiKingsley commented Oct 4, 2024

Jolanrensen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndreiKingsley Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zaleslaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndreiKingsley Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zaleslaw Nov 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndreiKingsley Nov 5, 2024 •

edited

Loading

AndreiKingsley Nov 5, 2024 •

edited

Loading

zaleslaw Nov 4, 2024 •

edited

Loading