Skip to content

Commit

Permalink
docs/schema: Restructure package reference
Browse files Browse the repository at this point in the history
  • Loading branch information
duncandewhurst committed Oct 11, 2023
1 parent 63227a6 commit 910ec05
Show file tree
Hide file tree
Showing 13 changed files with 102 additions and 113 deletions.
4 changes: 2 additions & 2 deletions docs/guidance/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,8 @@ If you have any issues using OCDS tools, contact the [Data Support Team](../../s
* If your source data is in **Excel files**, you can alternately transform Excel files to OCDS JSON by using the [Open Contracting Explorer](https://developmentgateway.org/expertise/open-contracting-procurement/), which includes a web interface and web API for users to access and explore the OCDS data. (This tool is authored by Development Gateway.)
* If your source data is in **SQL tables**, you can use [Kavure'i](https://gitlab.com/dncp-opendata/opendata-etl/-/blob/master/README_en.md) to transform it to OCDS. To use it, you write SQL queries to extract data from SQL tables, setting the columns for the query results to match the JSON paths in OCDS (for example, `buyer/name`). The query results are saved to CSV files, which are transformed to OCDS JSON using [Flatten Tool](https://flatten-tool.readthedocs.io/en/latest/usage-ocds/). (Kavure'i is authored by Paraguay's Dirección Nacional de Contrataciones Públicas (DNCP).)
* To **make OCDS data available via an API**, you can use another component of [Kavure'i](https://gitlab.com/dncp-opendata/opendata-etl/-/blob/master/README_en.md) to load OCDS data into [ElasticSearch](https://www.elastic.co/), and then use [Pitogüé](https://gitlab.com/dncp-opendata/opendata-api-v3/blob/master/README_en.md) to make it available via an API. (Both tools are authored by Paraguay's Dirección Nacional de Contrataciones Públicas (DNCP).)
* If you intend to **publish [record packages](../../schema/record_package)**, [OCDS Merge](https://ocds-merge.readthedocs.io/en/latest/) is the best software library for creating OCDS [records](../../primer/releases_and_records). If you use the [Python](https://www.python.org/) programming language, you can use it directly. If not, you can use its [test cases](https://ocds-merge.readthedocs.io/en/latest/#test-cases) to test your implementation of the [merge routine](../../schema/merging), and you can read its [commented code](https://github.com/open-contracting/ocds-merge) as inspiration for your implementation.
* If you have [release packages](../../schema/release_package) and want to have [record packages](../../schema/record_package), if you have data that follows an older version of OCDS, or if you otherwise need to transform your OCDS data, you can use [OCDS Kit](https://ocdskit.readthedocs.io/en/latest/) as a command-line tool or [Python](https://www.python.org/) library.
* If you intend to **publish [record packages](../../schema/packaging/record_package)**, [OCDS Merge](https://ocds-merge.readthedocs.io/en/latest/) is the best software library for creating OCDS [records](../../primer/releases_and_records). If you use the [Python](https://www.python.org/) programming language, you can use it directly. If not, you can use its [test cases](https://ocds-merge.readthedocs.io/en/latest/#test-cases) to test your implementation of the [merge routine](../../schema/merging), and you can read its [commented code](https://github.com/open-contracting/ocds-merge) as inspiration for your implementation.
* If you have [release packages](../../schema/packaging/release_package) and want to have [record packages](../../schema/packaging/record_package), if you have data that follows an older version of OCDS, or if you otherwise need to transform your OCDS data, you can use [OCDS Kit](https://ocdskit.readthedocs.io/en/latest/) as a command-line tool or [Python](https://www.python.org/) library.

If you aren't creating or updating an IT system, but are instead reusing an existing [data collection tool](build/data_collection_tools), you can customize it:

Expand Down
2 changes: 1 addition & 1 deletion docs/guidance/build/easy_releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ These approaches can be useful for most situations but they are not meant to be

### Packaging

Releases in OCDS need to be packaged using a [release package](../../schema/release_package). This is to provide consistency and important metadata.
Releases in OCDS need to be packaged using a [release package](../../schema/packaging/release_package). This is to provide consistency and important metadata.

In an 'Easy' releases scenario it is still necessary to package data. Therefore the release needs to be wrapped in a *release package*. It is *not* appropriate to use an OCDS record to contain the release because `record.releases` is a list of all the releases and not just the latest one.

Expand Down
2 changes: 1 addition & 1 deletion docs/guidance/build/hosting.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ To meet the [basic criteria for data quality](../publish/quality.md#basic-criter

## Bulk downloads

The [release package](../../schema/release_package) and [record package](../../schema/record_package) can provide **bulk access** to releases and records, respectively.
The [release package](../../schema/packaging/release_package) and [record package](../../schema/packaging/record_package) can provide **bulk access** to releases and records, respectively.

However, very large files can be difficult for users to download and process. The following section suggests good practices to assist users in accessing data. These are not requirements of the standard, but are based on experiences of maximizing the number of users able to work with datasets with their existing hardware and software.

Expand Down
4 changes: 2 additions & 2 deletions docs/history/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -446,7 +446,7 @@ See the changelogs for:
### Schema

* [#251](https://github.com/open-contracting/standard/issues/251) Allow optional field `parties.role` to be `null`.
* [#479](https://github.com/open-contracting/standard/issues/479) Remove `releases` as a required field in [`record-package-schema.json`](../schema/record_package).
* [#479](https://github.com/open-contracting/standard/issues/479) Remove `releases` as a required field in [`record-package-schema.json`](../schema/packaging/record_package).
* [#475](https://github.com/open-contracting/standard/issues/475) Add an `enum` property to every field in the schema with a closed codelist.

### Codelists
Expand Down Expand Up @@ -483,7 +483,7 @@ See the changelogs for:
* [#323](https://github.com/open-contracting/standard/issues/323) **[Standard should specify use of UTF-8 for encoding JSON](../guidance/build/serialization.md#json)** - We now recommend use of I-JSON and UTF-8 for JSON publication
* [#166](https://github.com/open-contracting/standard/issues/166) **[Indicate encoding for CSV serializations](../guidance/build/serialization.md#multi-table)** - We have added information on CSV encoding to the implementation guidance to recommend UTF-8 or Windows-1252.
* [#336](https://github.com/open-contracting/standard/issues/336) **[Codelist properties in schema](https://github.com/open-contracting/standard/commit/ee1db256d7364ee70e3553f4384d9908bff604a6)** - We have included explicit references to the codelist files (CSV versions) in the schema. This will be used in future versions of the validator to support validation against codelists.
* [#301](https://github.com/open-contracting/standard/issues/301) **[Specifying versions and extensions in package metadata](../schema/records_reference.md#package-metadata)** - We have updated the package schemas to require that version is declared, and to provide a means of declaring extensions in use. This will be used in future versions of the validator to support validation against extensions.
* [#301](https://github.com/open-contracting/standard/issues/301) **[Specifying versions and extensions in package metadata](../schema/packaging/index)** - We have updated the package schemas to require that version is declared, and to provide a means of declaring extensions in use. This will be used in future versions of the validator to support validation against extensions.
* [#287](https://github.com/open-contracting/standard/issues/287) [#447](https://github.com/open-contracting/standard/pull/447) **[Change how merge strategies are expressed](../schema/merging)** - The `omitWhenMerged` and `wholeListMerge` properties replace the `mergeStrategy` and `mergeOptions` properties, with different semantics.
* [#431](https://github.com/open-contracting/standard/pull/431) **Add `minLength` property to required string fields**.

Expand Down
5 changes: 2 additions & 3 deletions docs/schema/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

The Open Contracting Data Standard is maintained using JSON Schema.

In this section you will find the schema for [releases](release) along with the schemas for [release packages](release_package) and [record packages](record_package), which act as envelopes for releases and records, respectively.
In this section you will find the schema for [releases](release) along with the schemas for [packaging](packaging/index.md), which act as envelopes for releases and records.

The [release schema reference](reference) provides guidance on using each of the [sections](reference.md#release-structure) and [building blocks](reference.md#building-block-reference) in the schema, and the [record schema reference](records_reference) provides additional information on publishing records with compiled and versioned releases.

Expand All @@ -13,9 +13,8 @@ OCDS data must follow the I-JSON (Internet JSON) specification in [RFC7493](http
reference
release
release_package
records_reference
record_package
packaging/index
merging
identifiers
codelists
Expand Down
10 changes: 10 additions & 0 deletions docs/schema/packaging/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Packaging

In this section you will find the schema for [release packages](release_package.md) and [record packages](record_package.md), which act as envelopes for releases and records, respectively.

```{toctree}
:hidden:
release_package
record_package
```
34 changes: 34 additions & 0 deletions docs/schema/packaging/record_package.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Record Package Schema

The record package schema describes the structure of the container for publishing records. The package contains important metadata.

````{admonition} Example
:class: hint
```{jsoninclude} ../../examples/merging/updates/versioned.json
:jsonpointer:
:title: Record package
```
````

For this version of OCDS, the canonical URL for the record package schema is <https://standard.open-contracting.org/schema/1__1__5/record-package-schema.json> and for the versioned release schema is <https://standard.open-contracting.org/schema/1__1__5/versioned-release-validation-schema.json>. Using the canonical URL guarantees that your software, documentation or other resources will always refer to the specific version of the schema with which they were authored and tested.

In addition to publishing the structured metadata described by the record package schema, you ought to license your data and provide guidance for data users. For more information, see the [licensing guidance](../../guidance/publish.md#license-your-data) and [publication policy guidance](../../guidance/publish.md#finalize-your-publication-policy).

```{admonition} Browsing the schema
:class: note
This page presents the record package schema in an interactive browser and in a table, with additional information in paragraphs. You can also download the canonical version of the record package schema as [JSON Schema](../../../build/current_lang/record-package-schema.json).
```

## Schema browser

Click on schema elements to expand the tree, or use the '+' icon to expand all elements. Use { } to view the underlying schema for any section. Required fields are indicated in **bold**. [Deprecated fields](../../governance/deprecation) and [multilingual fields](../reference.md#language) are omitted.

<script src="../../../_static/docson/public/js/widget.js" data-schema="../../../../record-package-schema.json"></script>

## Reference table

```{jsonschema} ../../../build/current_lang/record-package-schema.json
:collapse: records
```
34 changes: 34 additions & 0 deletions docs/schema/packaging/release_package.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Release Package Reference

The release package schema describes the structure of the container for publishing releases. The package contains important metadata.

````{admonition} Example
:class: hint
```{jsoninclude} ../../examples/release_schema_reference/release_package.json
:jsonpointer:
:title: Release package
```
````

For this version of OCDS, the canonical URL of the release package schema is: <https://standard.open-contracting.org/schema/1__1__5/release-package-schema.json>. Using the canonical URL guarantees that your software, documentation or other resources will always refer to the specific version of the schema with which they were authored and tested.

In addition to publishing the structured metadata described by the release package schema, you ought to license your data and provide guidance for data users. For more information, see the [licensing guidance](../../guidance/publish.md#license-your-data) and [publication policy guidance](../../guidance/publish.md#finalize-your-publication-policy).

```{admonition} Browsing the schema
:class: note
This page presents the release package schema in an interactive browser and in a table, with additional information in paragraphs. You can also download the canonical version of the release package schema as [JSON Schema](../../../build/current_lang/release-package-schema.json).
```

## Schema browser

Click on schema elements to expand the tree, or use the '+' icon to expand all elements. Use { } to view the underlying schema for any section. Required fields are indicated in **bold**. [Deprecated fields](../../governance/deprecation) and [multilingual fields](../reference.md#language) are omitted.

<script src="../../../_static/docson/public/js/widget.js" data-schema="../../../../release-package-schema.json"></script>

## Reference table

```{jsonschema} ../../../build/current_lang/release-package-schema.json
:collapse: releases
```
17 changes: 0 additions & 17 deletions docs/schema/record_package.md

This file was deleted.

28 changes: 5 additions & 23 deletions docs/schema/records_reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,38 +6,18 @@ There should be a single record per ocid per [distribution](https://www.w3.org/T

**Note: If any conflicts are found between this text, and the text within the schema, the schema takes precedence.**

Records must be published within a [record package](packaging/record_package).

```{admonition} Browsing the schema
:class: note
This page presents the record package schema as tables. You can also download the canonical version of the record package schema as [JSON Schema](../../build/current_lang/record-package-schema.json), or view it in an [interactive browser](record_package).
This page presents the record package schema as tables. You can also download the canonical version of the record package schema as [JSON Schema](../../build/current_lang/record-package-schema.json), or view it in an [interactive browser](packaging/record_package).
```

```{workedexamplelist} The following worked examples are available for records
:tag: record
```

## Package metadata

Records must be published within a [record package](record_package). The record package provides metadata about the record(s) that it contains.

```{jsonschema} ../../build/current_lang/record-package-schema.json
:collapse: records
```

See the [licensing guidance](../guidance/publish.md#license-your-data) for more details on selecting a license, and publishing license information.

See the [publication policy](../guidance/publish.md#finalize-your-publication-policy) guidance for more details on what to include in a publication policy.

The record package metadata has one difference from the release package metadata: instead of a `releases` array, a record package has a `records` array containing one or more records.

The following example demonstrates the package metadata and record fields.

```{jsoninclude} ../examples/merging/updates/versioned.json
:jsonpointer:
:expand: records
:title: package
```

## Record structure

A record **must** contain an [ocid](identifiers.md#open-contracting-process-identifier-ocid) and one or more of:
Expand Down Expand Up @@ -109,6 +89,8 @@ This versioned information is relevant to many use cases relating to contract mo

If the versioned release is not provided, third parties can generate it by processing the record's releases according to the [merging](merging) reference.

A separate, auto-generated [versioned release schema](../../build/current_lang/versioned-release-validation-schema.json) is provided for validating versioned releases within records.

The following example displays a single field's [versioned values](merging.md#versioned-values). This shows that the amount changed between the tender stage and a tender amendment, while the currency did not.

```{jsoninclude} ../examples/merging/updates/versioned.json
Expand Down
Loading

0 comments on commit 910ec05

Please sign in to comment.