diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 00000000..163bf4e3 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,6 @@ +## Checklist + +When making changes to YAML files in the [schemas](../python/lsst/sdm_schemas/schemas) directory: + +- [ ] If applicable, incremented the schema version number, following the guidelines in the [contribution guide](../CONTRIBUTING.md) +- [ ] Referred to the [documentation on specific schemas](../CONTRIBUTING.md#specific-schema-documentation) for additional versioning information, change constraints, or tasks that may need to be performed, based on which schema is being updated diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..2824e279 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,70 @@ +Contributing to SDM Schemas +=========================== + +Creating a PR +------------- + +All changes to the repository must be made by [opening a pull request](https://github.com/lsst/sdm_schemas/compare), which will require one approving reviewer. +Any branches used for PRs which contain non-trivial changes should follow the [standard DM naming convention for ticket branches](https://developer.lsst.io/work/flow.html#ticket-branches), or the PR will be rejected and closed. + +The PR title should be formatted like: `[TICKET]: [TITLE]` where `TICKET` is the Jira ticket number and `TITLE` is a short description. + +For example, this is a valid PR title, with a dummy ticket number: + +`DM-12345: Add ra and dec columns to test schema`. + +Including the name of the schema in the PR title is also helpful for the reviewer and provenance, e.g., `test` in this case. + +An appropriate reviewer should be assigned, based on which schema is being changed or added. +If the author does not know who should review their PR, [JeremyMcCormick](https://github.com/JeremyMcCormick) can be assigned as a default, and an appropriate reviewer will be found. + +The author should [resolve the review](https://developer.lsst.io/work/flow.html#resolving-a-review) if changes are requested. +Once the PR is approved, it may then be merged by the author. + +Pull Request Checks +------------------- + +SDM Schemas pull requests have an extensive set of checks that will run automatically. +Aside from special cases, a pull request will have to pass all of these checks before it can be merged. + +These checks will: + +- [Build TAP_SCHEMA and DataLink resources](.github/workflows/build.yaml) +- [Compare Schemas for Changes](.github/workflows/compare.yaml) - Schemas that are in the [deployed list](./python/lsst/sdm_schemas/schemas/deployed-schemas.txt) will fail this check. +- [Build the Schema Browser website](.github/workflows/docs.yaml) +- [Check that `main` is not merged into the branch](.github/workflows/rebase_checker.yaml) +- [Test that the schemas can be used to create databases](.github/workflows/test_databases.yaml) +- [Vaildate the schemas using Felis](.github/workflows/validate.yaml) +- [Lint YAML files](.github/workflows/yamllint.yaml) + +The checks will rerun anytime the PR branch is updated. + +Schema Versioning +----------------- + +Individual schemas may have their own internal [version](https://felis.lsst.io/user-guide/model.html#schema-version) for tracking changes, which is defined at the top of the file, as in [apdb.yaml](./python/lsst/sdm_schemas/schemas/apdb.yaml). +(This is distinct from tags or versions of sdm_schemas itself, and not all schemas may be using them.) +This version may need to be updated when making changes. + +It is recommended to use versions with three numbers in the format `MAJOR.MINOR.PATCH`, such as `1.2.3`, though any scheme may be used in practice. + +The following guidelines can be used for incrementing the version: + +- Different `MAJOR` versions are incompatible. +- Different `MINOR` versions are backward compatible. +- Different `PATCH` versions are completely compatible. + +These suggestions need not be followed strictly, and there may be exceptions. +For instance, some operations which are technically backward-compatible, such as adding a table, should likely trigger a `MAJOR` version increment rather than `MINOR`, as they would constitute an important and significant update, potentially requiring significant changes to client code or APIs. + +The [specific schema documentation](#specific-schema-documentation) should be consulted for more specific rules and guidelines on version incrementing. + +Database Migrations +------------------- + +Database migrations are currently performed with schema-specific tooling in external repositories, so documentation for that particular schema should be consulted on how to perform them and how this may impact the pull request process. + +Specific Schema Documentation +----------------------------- + +- [Alert Production Database (APDB)](docs/APDB.md) diff --git a/docs/APDB.md b/docs/APDB.md new file mode 100644 index 00000000..bb093278 --- /dev/null +++ b/docs/APDB.md @@ -0,0 +1,49 @@ +Alert Production Database (APDB) Schema +======================================= + +The [apdb](../python/lsst/sdm_schemas/schemas/apdb.yaml) schema describes the Alert Production Database (APDB), which contains the results of processing from the [Alert Production Pipeline](https://github.com/lsst/ap_pipe). + +Schema Versioning +----------------- + +The versioning system of the APDB is outlined in [DMTN-269](https://dmtn-269.lsst.io/). +The deployed database schema is a product of the [schema in sdm_schemas](../python/lsst/sdm_schemas/schemas/apdb.yaml) with additional processing by the APDB client library from the [dax_apdb](https://github.com/lsst/dax_apdb) repository. +Compatibility of the client executable with the actual database schema is determined by the version number in the YAML file and the version stored in the database metadata. + +Backward compatibility in the context of the APDB implies that a client using a schema with a higher minor version will be able to read and write into a database with a lower minor version -- in other words, an existing database instance need not be migrated to match the newer client interface. +But a client using a schema with a lower minor version will be incompatible with a database that has a higher version number. +There may be cases where significant schema changes may still be compatible if the client can handle them transparently. + +Typical Schema Changes +---------------------- + +Here are some guidelines for incrementing the version number based on what changes are being performed: + +| Operation | Increment | Notes | +|--------|-----------|-------| +| Add table | `MAJOR` | The client using the new schema will typically want to read or write into that table, which will make it incompatible.| +| Add column | `MAJOR` | The client using the new schema will typically want to read or write into that column, which will make it incompatible.| +| Remove column | `MINOR` or `PATCH` | Clients using the new schema will not care about a removed column, so this should be backward compatible.| +| Change column type | `MINOR` or `PATCH` | This may or may not be completely transparent and so could be either backward compatible or fully compatible. +| Add index | `PATCH` | This should be fully backward compatible.| +| Change object metadata | - | This does not affect the database schema and should not require a version change.| + +Client Compatibility +-------------------- + +Changes to the APDB schema must be propagated to the Avro alert schemas defined in the [alert_packet](https://github.com/lsst/alert_packet) repository, following the instructions under [Adding a new schema](https://github.com/lsst/alert_packet/tree/main?tab=readme-ov-file#adding-a-new-schema). + +Database Migrations +------------------- + +The APDB database schema is upgraded using a special migration tool defined in the [dax_apdb_migrate](https://github.com/lsst-dm/dax_apdb_migrate) repository. +For every version change to `apdb.yaml`, a new migration script needs to be added to that repository. +The details of the migration process are described in the [package documentation](https://github.com/lsst-dm/dax_apdb_migrate/blob/main/doc/lsst.dax.apdb_migrate/index.rst). +The APDB includes a Cassandra-based implementation in addition to a SQL-based one, and the Cassandra implementation may impose additional restrictions on schema changes. +Migration scripts should normally be implemented on the same Jira ticket as the corresponding schema change. + +Reviewers +--------- + +- [Ian Sullivan](https://github.com/isullivan) - primary reviewer (may reroute to another reviewer depending on changes) +- [Andy Salnikov](https://github.com/andy-slac) - for guidance on updating schema versions diff --git a/doc/dp02_dc2.mmd b/docs/dp02_dc2.mmd similarity index 100% rename from doc/dp02_dc2.mmd rename to docs/dp02_dc2.mmd