Implementations status #26

roll · 2017-11-09T13:03:35Z

Frictionless Data Implementations

In a requirements section we list all required functionality. Then in per language sections we list what's missed for now. Not every missing thing should be implemented right now - some stuff (like foreign keys) could be backlogged for a while for some implementation. But in this document we're trying to draw the whole picture.

Requirements

High-level requirements reference. It uses italics for requirements not directly mentioned in the implementation requirements document:

tableschema
- read and validate a table schema descriptor
- create and edit a table schema descriptor
- provide an API to interact with a table schema descriptor
- infer a table schema descriptor from a supplied sample of data
- validate a data source against the table schema descriptor, including in response to editing the descriptor
- streaming and reading of a data source through a table schema with cast on iteration
- save a table schema descriptor to a file path
- cover the latest changes in the spec such as bareNumber, true/falseValues, string PK/FK etc
- support true data streaming
- support all data constraints
- support all data types
datapackage
- read and validate a data package descriptor, including profile-specific validation via the registry of JSON Schemas
- create and edit a data package descriptor, including methods to add and remove resources and validation after editing
- provide an API to interact with a data package descriptor
- streaming and reading of resources through a table schema with cast on iteration
- save a data package descriptor to a file path
- save a data package as a zip file on disc
- read a zip file containing a data package
- cover the latest changes in the specs such as string/array for path etc
- support for schema dereferencing before validation
- support for multipart resources
- support for csv dialects

Python

Repositories:

Maintainers:

Evgeny Karev

JavaScript

Not implemented requirements:

datapackage
- save a data package as a zip file on disc - #93
- read a zip file containing a data package - #93
- support for multipart resources - #79
- support for csv dialects - #86

Repositories:

Maintainers:

Evgeny Karev

Ruby

Not implemented requirements:

datapackage
- save a data package as a zip file on disc - #49
- read a zip file containing a data package - #49
- support for multipart resources - #37
- support for csv dialects - #51

Repositories:

Maintainers:

Evgeny Karev

PHP

Not implemented requirements:

tableschema
- infer a Table Schema descriptor from a supplied sample of data - no stand-alone function - #33
- cover the latest changes in specs such as bareNumber, true/falseValues, string PK/FK etc - #34
datapackage
- read a zip file containing a data package? - #31
- support for schema dereferencing before validation? - #37
- support for multipart resources? - #33
- support for csv dialects - #34

Repositories:

Maintainers:

Ori Hoch

Java

Requirements to implement:

tableschema
- support true data streaming - it seems for now we read to memory in this.data = this.reader.readAll() - #9
- cover the latest changes in the spec such as bareNumber, true/falseValues, string PK/FD - #4
datapackage
- read and validate a data package descriptor, including profile-specific validation via the registry of JSON Schemas - #6
- create and edit a data package descriptor, including methods to add and remove resources and validation after editing - #7
- provide an API to interact with a data package descriptor - #8
- streaming and reading of resources through a table schema with cast on iteration - #9
- save a data package descriptor to a file path - #10
- save a data package as a zip file on disc - #11
- read a zip file containing a data package - #12
- cover the latest changes in the specs such as string/array for path etc - #13
- support for schema dereferencing before validation - #14
- support for multipart resources - #15
- support for csv dialects - #16

Repositories:

Maintainers:

Georges Labrèche

Go

Not implemented requirements:

datapackage
- read and validate a data package descriptor, including profile-specific validation via the registry of JSON Schemas
- create and edit a data package descriptor, including methods to add and remove resources and validation after editing
- provide an API to interact with a data package descriptor
- streaming and reading of resources through a table schema with cast on iteration
- save a data package descriptor to a file path
- save a data package as a zip file on disc
- read a zip file containing a data package
- cover the latest changes in the specs such as string/array for path etc
- support for schema dereferencing before validation
- support for multipart resources
- support for csv dialects

Repositories:

Maintainers:

Daniel Fireman

R

Requirements to implement:

tableschema
- read and validate a table schema descriptor
- create and edit a table schema descriptor
- provide an API to interact with a table schema descriptor
- infer a table schema descriptor from a supplied sample of data
- validate a data source against the table schema descriptor, including in response to editing the descriptor
- streaming and reading of a data source through a table schema with cast on iteration
- save a table schema descriptor to a file path
- cover the latest changes in the spec such as bareNumber, true/falseValues, string PK/FK etc
- support true data streaming
datapackage
- read and validate a data package descriptor, including profile-specific validation via the registry of JSON Schemas
- create and edit a data package descriptor, including methods to add and remove resources and validation after editing
- provide an API to interact with a data package descriptor
- streaming and reading of resources through a table schema with cast on iteration
- save a data package descriptor to a file path
- save a data package as a zip file on disc
- read a zip file containing a data package
- cover the latest changes in the specs such as string/array for path etc
- support for schema dereferencing before validation
- support for multipart resources
- support for csv dialects

Repositories:

Maintainers:

OKI Greece

Clojure

Requirements to implement:

tableschema
- read and validate a table schema descriptor
- create and edit a table schema descriptor
- provide an API to interact with a table schema descriptor
- infer a table schema descriptor from a supplied sample of data
- validate a data source against the table schema descriptor, including in response to editing the descriptor
- streaming and reading of a data source through a table schema with cast on iteration
- save a table schema descriptor to a file path
- cover the latest changes in the spec such as bareNumber, true/falseValues, string PK/FK etc
- support true data streaming
- support all data constraints
- support all data types
datapackage
- read and validate a data package descriptor, including profile-specific validation via the registry of JSON Schemas
- create and edit a data package descriptor, including methods to add and remove resources and validation after editing
- provide an API to interact with a data package descriptor
- streaming and reading of resources through a table schema with cast on iteration
- save a data package descriptor to a file path
- save a data package as a zip file on disc
- read a zip file containing a data package
- cover the latest changes in the specs such as string/array for path etc
- support for schema dereferencing before validation
- support for multipart resources
- support for csv dialects

Repositories:

Maintainers:

Matthew Thompson

Julia

Requirements to implement:

tableschema
- read and validate a table schema descriptor
- create and edit a table schema descriptor
- provide an API to interact with a table schema descriptor
- infer a table schema descriptor from a supplied sample of data
- validate a data source against the table schema descriptor, including in response to editing the descriptor
- streaming and reading of a data source through a table schema with cast on iteration
- save a table schema descriptor to a file path
- cover the latest changes in the spec such as bareNumber, true/falseValues, string PK/FK etc
- support true data streaming
- support all data constraints
- support all data types
datapackage
- read and validate a data package descriptor, including profile-specific validation via the registry of JSON Schemas
- create and edit a data package descriptor, including methods to add and remove resources and validation after editing
- provide an API to interact with a data package descriptor
- streaming and reading of resources through a table schema with cast on iteration
- save a data package descriptor to a file path
- save a data package as a zip file on disc
- read a zip file containing a data package
- cover the latest changes in the specs such as string/array for path etc
- support for schema dereferencing before validation
- support for multipart resources
- support for csv dialects

Repositories:

Maintainers:

Oleg Lavrovsky

The text was updated successfully, but these errors were encountered:

roll · 2017-11-09T13:12:06Z

Java update:

read and validate a table schema descriptor - not supported remote descriptors - Read and validate a table schema descriptor - not supported remote descriptors tableschema-java#5
provide an API to interact with a table schema descriptor - fieldNames, primaryKey etc properties? - Provide an API to interact with a table schema descriptor - fieldNames, primaryKey etc properties? tableschema-java#6
save a table schema descriptor to a file path - Save a table schema descriptor to a file path tableschema-java#8
read and validate a table schema descriptor - not supported remote descriptors -
Read and validate a table schema descriptor - not supported remote descriptors tableschema-java#5 (comment)

roll · 2017-11-09T13:23:21Z

Go update:

support all data constraints - Implement "unique" constraint and other constraints? tableschema-go#42

roll · 2017-11-09T13:46:41Z

@serahrono @OriHoch @georgeslabreche @danielfireman @kleanthisk10 @cblop @loleg
I think the best way for us to communicate updates will be this issue. So I've moved the hackmd document to here. I've added a few updates as examples (and updated the main issue message)

danielfireman · 2017-11-13T15:10:11Z

tableschema-go v1.0 released: https://github.com/frictionlessdata/tableschema-go/releases

roll · 2017-11-13T15:27:57Z

@danielfireman
Great! Congratulations!

danielfireman · 2017-11-21T23:44:08Z

Go update:

~~read and validate a data package descriptor, including profile-specific validation via the registry of JSON Schemas~~
~~create and edit a data package descriptor, including methods to add and remove resources and validation after editing~~
~~provide an API to interact with a data package descriptor~~
~~streaming and reading of resources through a table schema with cast on iteration~~

@roll , could you please take a look?

roll · 2017-11-22T15:02:41Z

@danielfireman
Great. Thanks! I'll review in coming days (hopefully it's not blocking).

danielfireman · 2017-11-23T17:35:50Z

Go update:

~~save a data package descriptor to a file path~~
~~save a data package as a zip file on disc~~
~~read a zip file containing a data package~~
~~cover the latest changes in the specs such as string/array for path etc~~

danielfireman · 2017-11-25T05:14:52Z

Go update:

~~support for schema dereferencing before validation~~
~~support for multipart resources~~
~~support for csv dialects~~

All done. Ready for a complete review 😃

georgeslabreche · 2017-11-28T00:40:46Z

Java update for the datapackage:

read and validate a data package descriptor, including profile-specific validation via the registry of JSON Schemas - Design documentation infrastructure #6
create and edit a data package descriptor, including methods to add and remove resources and validation after editing - Fix umd library names for javascript packages #7
provide an API to interact with a data package descriptor - Update specs.frictionlessdata.io links in libraries #8
save a data package descriptor to a file path - #10
save a data package as a zip file on disc - Update tableschema plugins to work with renamed core #11
read a zip file containing a data package - Open a specs-v1 issue on each partner library #12
support for multipart resources - Review and provide feedback for PHP implementations #15

roll · 2017-11-29T11:03:32Z

@danielfireman
@georgeslabreche
As discussed with Daniel unfortunately I'll be able to review and commit changes to first post only on next week. For now trying to unblock everyone and looking forward for next week reviews.

georgeslabreche · 2017-11-30T04:26:04Z

@roll I'll just keep updating that list in the meantime.

OriHoch · 2017-11-30T14:13:56Z

Php update:

tableschema: all issues fixed
datapackage:
- ~~read a zip file containing a data package?~~ done
- ~~support for csv dialects~~ done

danielfireman · 2017-12-21T14:40:57Z

Hi @roll, good morning. Could you please finish the datapackage-go review?

roll · 2017-12-25T13:19:08Z

@danielfireman
Hi, unfortunately I still can't have allocated time for it.. We're considering it's done.

Stephen-Gates · 2018-01-17T23:32:25Z

I wonder if it would also be useful for apps that use these implementations to provide an implementation statement. e.g. Data Curator implements:

data package (partial)
- only tabular data packages
- cannot add properties:
  - homepage
  - image
  - keywords
  - created
data resource (partial)
- only tabular data packages
- cannot add properties:
  - data in-line data
  - path data at a url
  - bytes
  - hash
tabular data package
tabular data resources (partial)
- cannot add:
  - JSON tabular data
  - schema at url
  - CSV dialect at a url
table schema
CSV dialect (planned partial support)
- "delimiter": ,, ;, or /t
- "header": true
Data package version pattern
Table Schema: Foreign Keys to Data Packages pattern (planned)
...

Suggest this would be done at the spec level unless there is partial implementations

roll · 2020-04-13T12:44:49Z

High-level statuses are here now - https://github.com/frictionlessdata/software/blob/master/STATUS.md

roll added the enhancement label Nov 9, 2017

roll changed the title ~~Implementation status~~ Implementations status Nov 9, 2017

roll mentioned this issue Nov 13, 2017

Read and validate a data package descriptor, including profile-specific validation via the registry of JSON Schemas frictionlessdata/datapackage-java#6

Closed

7 tasks

Stephen-Gates mentioned this issue Jan 17, 2018

Add FD Specification compliance statement qcif/data-curator#394

Closed

Stephen-Gates mentioned this issue Feb 23, 2018

Indicate version support on each tool, library, platform frictionlessdata/website-old#135

Open

danielfireman mentioned this issue Feb 26, 2018

Introduce Resource.RawRead() frictionlessdata/datapackage-go#13

Closed

roll closed this as completed Apr 6, 2020

roll reopened this Apr 6, 2020

roll closed this as completed Apr 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementations status #26

Implementations status #26

roll commented Nov 9, 2017 •

edited

Loading

roll commented Nov 9, 2017 •

edited

Loading

roll commented Nov 9, 2017 •

edited

Loading

roll commented Nov 9, 2017 •

edited

Loading

danielfireman commented Nov 13, 2017

roll commented Nov 13, 2017

danielfireman commented Nov 21, 2017

roll commented Nov 22, 2017

danielfireman commented Nov 23, 2017 •

edited

Loading

danielfireman commented Nov 25, 2017

georgeslabreche commented Nov 28, 2017 •

edited

Loading

roll commented Nov 29, 2017

georgeslabreche commented Nov 30, 2017

OriHoch commented Nov 30, 2017 •

edited

Loading

danielfireman commented Dec 21, 2017

roll commented Dec 25, 2017

Stephen-Gates commented Jan 17, 2018 •

edited

Loading

roll commented Apr 13, 2020

Implementations status #26

Implementations status #26

Comments

roll commented Nov 9, 2017 • edited Loading

Frictionless Data Implementations

Requirements

Python

JavaScript

Ruby

PHP

Java

Go

R

Clojure

Julia

roll commented Nov 9, 2017 • edited Loading

roll commented Nov 9, 2017 • edited Loading

roll commented Nov 9, 2017 • edited Loading

danielfireman commented Nov 13, 2017

roll commented Nov 13, 2017

danielfireman commented Nov 21, 2017

roll commented Nov 22, 2017

danielfireman commented Nov 23, 2017 • edited Loading

danielfireman commented Nov 25, 2017

georgeslabreche commented Nov 28, 2017 • edited Loading

roll commented Nov 29, 2017

georgeslabreche commented Nov 30, 2017

OriHoch commented Nov 30, 2017 • edited Loading

danielfireman commented Dec 21, 2017

roll commented Dec 25, 2017

Stephen-Gates commented Jan 17, 2018 • edited Loading

roll commented Apr 13, 2020

roll commented Nov 9, 2017 •

edited

Loading

roll commented Nov 9, 2017 •

edited

Loading

roll commented Nov 9, 2017 •

edited

Loading

roll commented Nov 9, 2017 •

edited

Loading

danielfireman commented Nov 23, 2017 •

edited

Loading

georgeslabreche commented Nov 28, 2017 •

edited

Loading

OriHoch commented Nov 30, 2017 •

edited

Loading

Stephen-Gates commented Jan 17, 2018 •

edited

Loading