Scope of schema validation (Dataset or scientificMetadata)

[sbliven]

(Documenting SciCatCon discussion)

Background

There are different requirements for schemas for the top-level dataset fields and the scientificMetadata.

Top level fields...

• change infrequently
• must match to the DTO for the installed backend version
• are mostly consistent across all facilities (possibly excepting some policies on what information should be required, eg proposal or keywords)
• should be generated from the backend code
• are already checked against the correct DTO by the backend

scientificMetadata...

• differs between facilities and often between communities within a facility
• should default to an untyped object
• changes frequently in a decentral way

Top-Level validation

One option is to validate the full dataset DTO. This would involve

• top-level schema generated from the backend
• Additional schemas must inherit from the top-level schema and add scientificMetadata constraints

Example

(JSON-LD syntax)

{
    "@context": "https://.../dataset_schema.json",
    "pid": "123...",
    "scientificMetadata": { "validatedField": true }
}

where dataset_schema.json inherits from the main scicat dataset schema and includes both top-level and scientific fields.

Advantages:

• metadata.json files can be validated with standard tools
  • if we use JSON-LD, metadata.json becomes a valid LD file without filtering
• Clear to communicate with scientists since the schema includes both scientific and data management fields

Disadvantages:

• Need to merge update all schemas if the base schema changes. This might be mitigated by tools that allow merging two schemas

Only validate scientificMetadata

Here the schema applies only to the scientificMetadata. Top-level fields are in principle validated in the code, rather than following the custom schema.

Example

(JSON-LD syntax)

{
    # Might have a fixed @context filled here by scicat covering top-level fields
    "pid": "123...",
    "scientificMetadata": {
        "@context": "https://.../metadata_schema.json",
        "validatedField": true
    }
}

where 'metadata_schema.json' only validates within the scientificMetadata subtree.

JSON Schema would probably make it a top-level field:

{
    "pid": "123...",
    "metadataSchema": "https://.../metadata_schema.json",
    "scientificMetadata": {
        "validatedField": true
    }
}

Advantages:

• no dependence of the custom schema on scicat fields
• Trivial with JSON-LD or straightforward with JSON Schema
• Seems to be the first solution most people think of

Disadvantages:

• Complete validation of a metadata.json file with standard tools requires two passes. This is mitigated by just calling the scicat /dataset/validate endopoint, which will check everything.
• The custom schema omits fields like 'datasetName' and 'description', which could be confusing

[sbliven]

Related to #966

[sbliven]

I think there are use cases for both structural and semantic specification of the metadata.

Structural validation

A structural schema defines the shape of the data. JSON Schema is the most common technology for specifying the data model and validating that input metadata conforms to it.

Use cases:

• Assert that particular fields will be defined, eg. scientificMetadata.experimentalMethod is required and matches a pre-defined enum
• Validate inputs at ingestion time to ensure that ingestion scripts function properly
• Provide a contract to data consumers ensuring that required metadata is available for processing. Note that this applies mostly to non-NeXus data, since processing of NeXus data is expected to use embedded NeXus metadata rather than getting data from SciCat.
• Selectively enable Scicat features for known schemas
  • Search features, eg auto-complete, unit conversion, typed searching
  • Custom visualizations for particular schemas
  • Automatic form generation for ingesting datasets according to some schema

Semantic labeling

Semantic labels assign meaning to data fields. Annotations are supported by JSON-LD or RDF formats, and generally consist of annotating data with a Compact uniform resource identifiers (CURIE). Semantic validation typically does not enforce a particular data shape, but rather relies on normalization technices such as graph expansion/compression to validate semantic properties.

Use cases:

• Provide clear, human- and machine-readable definitions of the definition of each field
  • Terms can be hierarchical
• Annotate synonyms and related terms in other ontologies
• Interoperability with other tools in a shape-independent manner
• Support SparQL queries
• Deeper validation (eg this property uses a length unit), although sometimes at a performance cost
• Selectively enable Scicat features without knowing the schema details
  • Search features, eg unit conversion
  • Custom visualizations for particular schemas
  • Autocompletion of terms from ontologies, although in practice this can be complex without a schema