Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for Spark Datasource API v2 #12

Open
jornfranke opened this issue Mar 1, 2018 · 4 comments
Open

Add support for Spark Datasource API v2 #12

jornfranke opened this issue Mar 1, 2018 · 4 comments
Labels
enhancement

Comments

@jornfranke
Copy link
Member

Since Spark 2.3.0 (https://spark.apache.org/releases/spark-release-2-3-0.html) a new datasource API v2 is available (experimental).

We should add support in addition to the v1 API.
@jornfranke
Copy link
Member Author

https://docs.google.com/document/d/1n_vUVbF4KD3gxTmkNEon5qdQ-Z8qU5Frf6WMQZ6jJVM

@jornfranke
Copy link
Member Author

Data Source API V2 will get more stable in 3.0.0, we will however do initial experiments before

@jornfranke
Copy link
Member Author

It seems that the V2 API is still experimental and Spark 3.0.0 is not yet out

@jornfranke
Copy link
Member Author

Spark 3.0.0 is coming soon and we will release also an initial v2 support despite that v1 is still supported in Spark 3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jornfranke