From 910ec05964166a5ba1df85db7b2b2bbc1ec84fab Mon Sep 17 00:00:00 2001 From: Duncan Dewhurst Date: Thu, 12 Oct 2023 12:10:55 +1300 Subject: [PATCH] docs/schema: Restructure package reference --- docs/guidance/build.md | 4 +- docs/guidance/build/easy_releases.md | 2 +- docs/guidance/build/hosting.md | 2 +- docs/history/changelog.md | 4 +- docs/schema/index.md | 5 +- docs/schema/packaging/index.md | 10 ++++ docs/schema/packaging/record_package.md | 34 ++++++++++++++ docs/schema/packaging/release_package.md | 34 ++++++++++++++ docs/schema/record_package.md | 17 ------- docs/schema/records_reference.md | 28 ++---------- docs/schema/reference.md | 58 ++++-------------------- docs/schema/release.md | 2 +- docs/schema/release_package.md | 15 ------ 13 files changed, 102 insertions(+), 113 deletions(-) create mode 100644 docs/schema/packaging/index.md create mode 100644 docs/schema/packaging/record_package.md create mode 100644 docs/schema/packaging/release_package.md delete mode 100644 docs/schema/record_package.md delete mode 100644 docs/schema/release_package.md diff --git a/docs/guidance/build.md b/docs/guidance/build.md index 1743ceb38..c36d44c1b 100644 --- a/docs/guidance/build.md +++ b/docs/guidance/build.md @@ -100,8 +100,8 @@ If you have any issues using OCDS tools, contact the [Data Support Team](../../s * If your source data is in **Excel files**, you can alternately transform Excel files to OCDS JSON by using the [Open Contracting Explorer](https://developmentgateway.org/expertise/open-contracting-procurement/), which includes a web interface and web API for users to access and explore the OCDS data. (This tool is authored by Development Gateway.) * If your source data is in **SQL tables**, you can use [Kavure'i](https://gitlab.com/dncp-opendata/opendata-etl/-/blob/master/README_en.md) to transform it to OCDS. To use it, you write SQL queries to extract data from SQL tables, setting the columns for the query results to match the JSON paths in OCDS (for example, `buyer/name`). The query results are saved to CSV files, which are transformed to OCDS JSON using [Flatten Tool](https://flatten-tool.readthedocs.io/en/latest/usage-ocds/). (Kavure'i is authored by Paraguay's Dirección Nacional de Contrataciones Públicas (DNCP).) * To **make OCDS data available via an API**, you can use another component of [Kavure'i](https://gitlab.com/dncp-opendata/opendata-etl/-/blob/master/README_en.md) to load OCDS data into [ElasticSearch](https://www.elastic.co/), and then use [Pitogüé](https://gitlab.com/dncp-opendata/opendata-api-v3/blob/master/README_en.md) to make it available via an API. (Both tools are authored by Paraguay's Dirección Nacional de Contrataciones Públicas (DNCP).) -* If you intend to **publish [record packages](../../schema/record_package)**, [OCDS Merge](https://ocds-merge.readthedocs.io/en/latest/) is the best software library for creating OCDS [records](../../primer/releases_and_records). If you use the [Python](https://www.python.org/) programming language, you can use it directly. If not, you can use its [test cases](https://ocds-merge.readthedocs.io/en/latest/#test-cases) to test your implementation of the [merge routine](../../schema/merging), and you can read its [commented code](https://github.com/open-contracting/ocds-merge) as inspiration for your implementation. -* If you have [release packages](../../schema/release_package) and want to have [record packages](../../schema/record_package), if you have data that follows an older version of OCDS, or if you otherwise need to transform your OCDS data, you can use [OCDS Kit](https://ocdskit.readthedocs.io/en/latest/) as a command-line tool or [Python](https://www.python.org/) library. +* If you intend to **publish [record packages](../../schema/packaging/record_package)**, [OCDS Merge](https://ocds-merge.readthedocs.io/en/latest/) is the best software library for creating OCDS [records](../../primer/releases_and_records). If you use the [Python](https://www.python.org/) programming language, you can use it directly. If not, you can use its [test cases](https://ocds-merge.readthedocs.io/en/latest/#test-cases) to test your implementation of the [merge routine](../../schema/merging), and you can read its [commented code](https://github.com/open-contracting/ocds-merge) as inspiration for your implementation. +* If you have [release packages](../../schema/packaging/release_package) and want to have [record packages](../../schema/packaging/record_package), if you have data that follows an older version of OCDS, or if you otherwise need to transform your OCDS data, you can use [OCDS Kit](https://ocdskit.readthedocs.io/en/latest/) as a command-line tool or [Python](https://www.python.org/) library. If you aren't creating or updating an IT system, but are instead reusing an existing [data collection tool](build/data_collection_tools), you can customize it: diff --git a/docs/guidance/build/easy_releases.md b/docs/guidance/build/easy_releases.md index 36ea28deb..ccd2065ee 100644 --- a/docs/guidance/build/easy_releases.md +++ b/docs/guidance/build/easy_releases.md @@ -21,7 +21,7 @@ These approaches can be useful for most situations but they are not meant to be ### Packaging -Releases in OCDS need to be packaged using a [release package](../../schema/release_package). This is to provide consistency and important metadata. +Releases in OCDS need to be packaged using a [release package](../../schema/packaging/release_package). This is to provide consistency and important metadata. In an 'Easy' releases scenario it is still necessary to package data. Therefore the release needs to be wrapped in a *release package*. It is *not* appropriate to use an OCDS record to contain the release because `record.releases` is a list of all the releases and not just the latest one. diff --git a/docs/guidance/build/hosting.md b/docs/guidance/build/hosting.md index a81ac844b..6f229ff04 100644 --- a/docs/guidance/build/hosting.md +++ b/docs/guidance/build/hosting.md @@ -19,7 +19,7 @@ To meet the [basic criteria for data quality](../publish/quality.md#basic-criter ## Bulk downloads -The [release package](../../schema/release_package) and [record package](../../schema/record_package) can provide **bulk access** to releases and records, respectively. +The [release package](../../schema/packaging/release_package) and [record package](../../schema/packaging/record_package) can provide **bulk access** to releases and records, respectively. However, very large files can be difficult for users to download and process. The following section suggests good practices to assist users in accessing data. These are not requirements of the standard, but are based on experiences of maximizing the number of users able to work with datasets with their existing hardware and software. diff --git a/docs/history/changelog.md b/docs/history/changelog.md index aca703a46..561cdb73f 100644 --- a/docs/history/changelog.md +++ b/docs/history/changelog.md @@ -446,7 +446,7 @@ See the changelogs for: ### Schema * [#251](https://github.com/open-contracting/standard/issues/251) Allow optional field `parties.role` to be `null`. -* [#479](https://github.com/open-contracting/standard/issues/479) Remove `releases` as a required field in [`record-package-schema.json`](../schema/record_package). +* [#479](https://github.com/open-contracting/standard/issues/479) Remove `releases` as a required field in [`record-package-schema.json`](../schema/packaging/record_package). * [#475](https://github.com/open-contracting/standard/issues/475) Add an `enum` property to every field in the schema with a closed codelist. ### Codelists @@ -483,7 +483,7 @@ See the changelogs for: * [#323](https://github.com/open-contracting/standard/issues/323) **[Standard should specify use of UTF-8 for encoding JSON](../guidance/build/serialization.md#json)** - We now recommend use of I-JSON and UTF-8 for JSON publication * [#166](https://github.com/open-contracting/standard/issues/166) **[Indicate encoding for CSV serializations](../guidance/build/serialization.md#multi-table)** - We have added information on CSV encoding to the implementation guidance to recommend UTF-8 or Windows-1252. * [#336](https://github.com/open-contracting/standard/issues/336) **[Codelist properties in schema](https://github.com/open-contracting/standard/commit/ee1db256d7364ee70e3553f4384d9908bff604a6)** - We have included explicit references to the codelist files (CSV versions) in the schema. This will be used in future versions of the validator to support validation against codelists. -* [#301](https://github.com/open-contracting/standard/issues/301) **[Specifying versions and extensions in package metadata](../schema/records_reference.md#package-metadata)** - We have updated the package schemas to require that version is declared, and to provide a means of declaring extensions in use. This will be used in future versions of the validator to support validation against extensions. +* [#301](https://github.com/open-contracting/standard/issues/301) **[Specifying versions and extensions in package metadata](../schema/packaging/index)** - We have updated the package schemas to require that version is declared, and to provide a means of declaring extensions in use. This will be used in future versions of the validator to support validation against extensions. * [#287](https://github.com/open-contracting/standard/issues/287) [#447](https://github.com/open-contracting/standard/pull/447) **[Change how merge strategies are expressed](../schema/merging)** - The `omitWhenMerged` and `wholeListMerge` properties replace the `mergeStrategy` and `mergeOptions` properties, with different semantics. * [#431](https://github.com/open-contracting/standard/pull/431) **Add `minLength` property to required string fields**. diff --git a/docs/schema/index.md b/docs/schema/index.md index d81ee5e6d..8787a5b4d 100644 --- a/docs/schema/index.md +++ b/docs/schema/index.md @@ -2,7 +2,7 @@ The Open Contracting Data Standard is maintained using JSON Schema. -In this section you will find the schema for [releases](release) along with the schemas for [release packages](release_package) and [record packages](record_package), which act as envelopes for releases and records, respectively. +In this section you will find the schema for [releases](release) along with the schemas for [packaging](packaging/index.md), which act as envelopes for releases and records. The [release schema reference](reference) provides guidance on using each of the [sections](reference.md#release-structure) and [building blocks](reference.md#building-block-reference) in the schema, and the [record schema reference](records_reference) provides additional information on publishing records with compiled and versioned releases. @@ -13,9 +13,8 @@ OCDS data must follow the I-JSON (Internet JSON) specification in [RFC7493](http reference release -release_package records_reference -record_package +packaging/index merging identifiers codelists diff --git a/docs/schema/packaging/index.md b/docs/schema/packaging/index.md new file mode 100644 index 000000000..e5da172d4 --- /dev/null +++ b/docs/schema/packaging/index.md @@ -0,0 +1,10 @@ +# Packaging + +In this section you will find the schema for [release packages](release_package.md) and [record packages](record_package.md), which act as envelopes for releases and records, respectively. + +```{toctree} +:hidden: + +release_package +record_package +``` diff --git a/docs/schema/packaging/record_package.md b/docs/schema/packaging/record_package.md new file mode 100644 index 000000000..811bf0cf9 --- /dev/null +++ b/docs/schema/packaging/record_package.md @@ -0,0 +1,34 @@ +# Record Package Schema + +The record package schema describes the structure of the container for publishing records. The package contains important metadata. + +````{admonition} Example +:class: hint + +```{jsoninclude} ../../examples/merging/updates/versioned.json +:jsonpointer: +:title: Record package +``` +```` + +For this version of OCDS, the canonical URL for the record package schema is and for the versioned release schema is . Using the canonical URL guarantees that your software, documentation or other resources will always refer to the specific version of the schema with which they were authored and tested. + +In addition to publishing the structured metadata described by the record package schema, you ought to license your data and provide guidance for data users. For more information, see the [licensing guidance](../../guidance/publish.md#license-your-data) and [publication policy guidance](../../guidance/publish.md#finalize-your-publication-policy). + +```{admonition} Browsing the schema +:class: note + +This page presents the record package schema in an interactive browser and in a table, with additional information in paragraphs. You can also download the canonical version of the record package schema as [JSON Schema](../../../build/current_lang/record-package-schema.json). +``` + +## Schema browser + +Click on schema elements to expand the tree, or use the '+' icon to expand all elements. Use { } to view the underlying schema for any section. Required fields are indicated in **bold**. [Deprecated fields](../../governance/deprecation) and [multilingual fields](../reference.md#language) are omitted. + + + +## Reference table + +```{jsonschema} ../../../build/current_lang/record-package-schema.json +:collapse: records +``` diff --git a/docs/schema/packaging/release_package.md b/docs/schema/packaging/release_package.md new file mode 100644 index 000000000..8f22febff --- /dev/null +++ b/docs/schema/packaging/release_package.md @@ -0,0 +1,34 @@ +# Release Package Reference + +The release package schema describes the structure of the container for publishing releases. The package contains important metadata. + +````{admonition} Example +:class: hint + +```{jsoninclude} ../../examples/release_schema_reference/release_package.json +:jsonpointer: +:title: Release package +``` +```` + +For this version of OCDS, the canonical URL of the release package schema is: . Using the canonical URL guarantees that your software, documentation or other resources will always refer to the specific version of the schema with which they were authored and tested. + +In addition to publishing the structured metadata described by the release package schema, you ought to license your data and provide guidance for data users. For more information, see the [licensing guidance](../../guidance/publish.md#license-your-data) and [publication policy guidance](../../guidance/publish.md#finalize-your-publication-policy). + +```{admonition} Browsing the schema +:class: note + +This page presents the release package schema in an interactive browser and in a table, with additional information in paragraphs. You can also download the canonical version of the release package schema as [JSON Schema](../../../build/current_lang/release-package-schema.json). +``` + +## Schema browser + +Click on schema elements to expand the tree, or use the '+' icon to expand all elements. Use { } to view the underlying schema for any section. Required fields are indicated in **bold**. [Deprecated fields](../../governance/deprecation) and [multilingual fields](../reference.md#language) are omitted. + + + +## Reference table + +```{jsonschema} ../../../build/current_lang/release-package-schema.json +:collapse: releases +``` diff --git a/docs/schema/record_package.md b/docs/schema/record_package.md deleted file mode 100644 index c6b17796b..000000000 --- a/docs/schema/record_package.md +++ /dev/null @@ -1,17 +0,0 @@ -# Record Package Schema - -The record package schema describes the structure of the container for publishing records. The contents of a record are based on the release schema. The package contains important metadata. - -A separate, auto-generated [versioned release schema](../../build/current_lang/versioned-release-validation-schema.json) is provided for validating versioned releases within records. - -For this version of OCDS, the canonical URL for the record package schema is and for the versioned release schema is . Using the canonical URL guarantees that your software, documentation or other resources will always refer to the specific version of the schema with which they were authored and tested. - -Click on schema elements to expand the tree, or use the '+' icon to expand all elements. Use { } to view the underlying schema for any section. Required fields are indicated in **bold**. [Deprecated fields](../governance/deprecation) and [multilingual fields](reference.md#language) are omitted. - -```{admonition} Browsing the schema -:class: note - -This page presents the record package schema in an interactive browser. You can also download the canonical version of the record package schema as [JSON Schema](../../build/current_lang/record-package-schema.json), or view it as [tables](records_reference). -``` - - diff --git a/docs/schema/records_reference.md b/docs/schema/records_reference.md index b3171895f..a01d27061 100644 --- a/docs/schema/records_reference.md +++ b/docs/schema/records_reference.md @@ -6,38 +6,18 @@ There should be a single record per ocid per [distribution](https://www.w3.org/T **Note: If any conflicts are found between this text, and the text within the schema, the schema takes precedence.** +Records must be published within a [record package](packaging/record_package). + ```{admonition} Browsing the schema :class: note -This page presents the record package schema as tables. You can also download the canonical version of the record package schema as [JSON Schema](../../build/current_lang/record-package-schema.json), or view it in an [interactive browser](record_package). +This page presents the record package schema as tables. You can also download the canonical version of the record package schema as [JSON Schema](../../build/current_lang/record-package-schema.json), or view it in an [interactive browser](packaging/record_package). ``` ```{workedexamplelist} The following worked examples are available for records :tag: record ``` -## Package metadata - -Records must be published within a [record package](record_package). The record package provides metadata about the record(s) that it contains. - -```{jsonschema} ../../build/current_lang/record-package-schema.json -:collapse: records -``` - -See the [licensing guidance](../guidance/publish.md#license-your-data) for more details on selecting a license, and publishing license information. - -See the [publication policy](../guidance/publish.md#finalize-your-publication-policy) guidance for more details on what to include in a publication policy. - -The record package metadata has one difference from the release package metadata: instead of a `releases` array, a record package has a `records` array containing one or more records. - -The following example demonstrates the package metadata and record fields. - -```{jsoninclude} ../examples/merging/updates/versioned.json -:jsonpointer: -:expand: records -:title: package -``` - ## Record structure A record **must** contain an [ocid](identifiers.md#open-contracting-process-identifier-ocid) and one or more of: @@ -109,6 +89,8 @@ This versioned information is relevant to many use cases relating to contract mo If the versioned release is not provided, third parties can generate it by processing the record's releases according to the [merging](merging) reference. +A separate, auto-generated [versioned release schema](../../build/current_lang/versioned-release-validation-schema.json) is provided for validating versioned releases within records. + The following example displays a single field's [versioned values](merging.md#versioned-values). This shows that the amount changed between the tender stage and a tender amendment, while the currency did not. ```{jsoninclude} ../examples/merging/updates/versioned.json diff --git a/docs/schema/reference.md b/docs/schema/reference.md index cbd5d1d91..c71684a2e 100644 --- a/docs/schema/reference.md +++ b/docs/schema/reference.md @@ -4,6 +4,8 @@ The [Release Schema](release) provides a detailed specification of the fields an Releases are immutable – presenting information about a particular event in the lifetime of a contracting (or planning) process. Publishers must not edit a release after publication; a new release can be created by changing the release's `id` and `date`. +Releases must be published within a [release package](packaging/release_package). + **Note: If any conflicts are found between this text, and the text within the schema, the schema takes precedence.** ```{admonition} Browsing the schema @@ -64,43 +66,20 @@ A contract for ‘Software consultancy services’ is published in a release wit ## Release structure -The majority of OCDS data is held within a release structure. One or more releases can be published within a release package. Releases are made up of a number of sections, arranged in the following structure. +The majority of OCDS data is held within a release structure. One or more releases can be published within a [release package](packaging/release_package.md). Releases are made up of a number of sections, arranged in the following structure. -* [package](#package-metadata) - * [release](#release) - * [parties](#parties) - * [planning](#planning) - * [tender](#tender) - * [award](#award) - * [contract](#contract) - * [implementation](#implementation) +* [release](#release) + * [parties](#parties) + * [planning](#planning) + * [tender](#tender) + * [award](#award) + * [contract](#contract) + * [implementation](#implementation) A release has a [tag](codelists.md#release-tag) to indicate whether it is about a planning process or a contracting process and, if it is about the latter, to indicate the stage of the contracting process to which it relates. However, there are no formal restrictions on when information about a stage of the process can be provided. For example, a publisher announcing the signing of a contract with a 'contract' tag might also include information in the award and tender blocks in order to provide a comprehensive picture of the contracting process to date which led to that contract being signed. -### Package Metadata - -Releases must be published within a [release package](release_package). The release package provides metadata about the release(s) that it contains. - -````{admonition} Example -:class: hint - -```{jsoninclude} ../examples/release_schema_reference/release_package.json -:jsonpointer: -:expand: publisher -:title: package -``` -```` - -```{jsonschema} ../../build/current_lang/release-package-schema.json -:collapse: releases -``` - -See the [licensing guidance](../guidance/publish.md#license-your-data) for more details on selecting a license, and publishing license information. - -See the [publication policy](../guidance/publish.md#finalize-your-publication-policy) guidance for more details on what to include in a publication policy. - ### Release All new information about a contracting (or planning) process is described within a release. @@ -711,23 +690,6 @@ As well as providing this machine-readable link between processes, publishers ma The [Location](https://extensions.open-contracting.org/en/extensions/location/v1.1.4/) extension can be used to provide location information. -### Publisher - -The publisher block is used in release and record packages to identify the source of a dataset. - -````{admonition} Example -:class: hint - -```{jsoninclude} ../examples/release_schema_reference/release_package.json -:jsonpointer: /publisher -:title: publisher -``` -```` - -```{jsonschema} ../../build/current_lang/release-package-schema.json -:include: publisher -``` - ### Link The entries of the top-level `links` array are `Link` objects: diff --git a/docs/schema/release.md b/docs/schema/release.md index 705498e4a..dbb161b4c 100644 --- a/docs/schema/release.md +++ b/docs/schema/release.md @@ -1,6 +1,6 @@ # Release Schema -The release schema provides the authoritative definition of fields and their structure. It is used to check the structure of the `releases` array of [release packages](release_package), and any `compiledRelease` fields in a [record package](record_package). +The release schema provides the authoritative definition of fields and their structure. It is used to check the structure of the `releases` array of [release packages](packaging/release_package), and any `compiledRelease` fields in a [record package](packaging/record_package). For this version of OCDS, the canonical URL of the release schema is: . Using the canonical URL guarantees that your software, documentation or other resources will always refer to the specific version of the schema with which they were authored and tested. diff --git a/docs/schema/release_package.md b/docs/schema/release_package.md deleted file mode 100644 index 64bc3270a..000000000 --- a/docs/schema/release_package.md +++ /dev/null @@ -1,15 +0,0 @@ -# Release Package Schema - -The release package schema describes the structure of the container for publishing releases. The package contains important metadata. - -For this version of OCDS, the canonical URL of the release package schema is: . Using the canonical URL guarantees that your software, documentation or other resources will always refer to the specific version of the schema with which they were authored and tested. - -Click on schema elements to expand the tree, or use the '+' icon to expand all elements. Use { } to view the underlying schema for any section. Required fields are indicated in **bold**. [Deprecated fields](../governance/deprecation) and [multilingual fields](reference.md#language) are omitted. - -```{admonition} Browsing the schema -:class: note - -This page presents the release package schema in an interactive browser. You can also download the canonical version of the release package schema as [JSON Schema](../../build/current_lang/release-package-schema.json). -``` - -