From 1b51498492ef57c979c1b4cf3b67eecc789ef04f Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Sun, 22 Oct 2023 09:05:27 +0200 Subject: [PATCH 01/11] Add contributing --- docs/development/contributing.md | 290 +++++++++++++++++++++++++++++++ docs/development/versioning.md | 0 mkdocs.yml | 7 + 3 files changed, 297 insertions(+) create mode 100644 docs/development/contributing.md create mode 100644 docs/development/versioning.md diff --git a/docs/development/contributing.md b/docs/development/contributing.md new file mode 100644 index 000000000000..bc892bab8910 --- /dev/null +++ b/docs/development/contributing.md @@ -0,0 +1,290 @@ +# Contributing + +Thanks for taking the time to contribute! We appreciate all contributions, from reporting bugs to implementing new features. +If you're unclear on how to proceed after reading this guide, please contact us on [Discord](https://discord.gg/4UfP5cfBE7). + +## Table of contents + +- [Reporting bugs](#reporting-bugs) +- [Suggesting enhancements](#suggesting-enhancements) +- [Contributing to the codebase](#contributing-to-the-codebase) +- [Contributing to documentation](#contributing-to-documentation) +- [Release flow](#release-flow) +- [License](#license) + +## Reporting bugs + +We use [GitHub issues](https://github.com/pola-rs/polars/issues) to track bugs and suggested enhancements. +You can report a bug by opening a [new issue](https://github.com/pola-rs/polars/issues/new/choose). +Use the appropriate issue type for the language you are using ([Rust](https://github.com/pola-rs/polars/issues/new?labels=bug&template=bug_report_rust.yml) / [Python](https://github.com/pola-rs/polars/issues/new?labels=bug&template=bug_report_python.yml)). + +Before creating a bug report, please check that your bug has not already been reported, and that your bug exists on the latest version of Polars. +If you find a closed issue that seems to report the same bug you're experiencing, open a new issue and include a link to the original issue in your issue description. + +Please include as many details as possible in your bug report. The information helps the maintainers resolve the issue faster. + +## Suggesting enhancements + +We use [GitHub issues](https://github.com/pola-rs/polars/issues) to track bugs and suggested enhancements. +You can suggest an enhancement by opening a [new feature request](https://github.com/pola-rs/polars/issues/new?labels=enhancement&template=feature_request.yml). +Before creating an enhancement suggestion, please check that a similar issue does not already exist. + +Please describe the behavior you want and why, and provide examples of how Polars would be used if your feature were added. + +## Contributing to the codebase + +### Picking an issue + +Pick an issue by going through the [issue tracker](https://github.com/pola-rs/polars/issues) and finding an issue you would like to work on. +Feel free to pick any issue with an [accepted](https://github.com/pola-rs/polars/issues?q=is%3Aopen+is%3Aissue+label%3Aaccepted) label that is not already assigned. +We use the [help wanted](https://github.com/pola-rs/polars/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) label to indicate issues that are high on our wishlist. + +If you are a first time contributor, you might want to look for issues labeled [good first issue](https://github.com/pola-rs/polars/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22). +The Polars code base is quite complex, so starting with a small issue will help you find your way around! + +If you would like to take on an issue, please comment on the issue to let others know. +You may use the issue to discuss possible solutions. + +### Setting up your local environment + +Polars development flow relies on both Rust and Python, which means setting up your local development environment is not trivial. +If you run into problems, please contact us on [Discord](https://discord.gg/4UfP5cfBE7). + +_Note that if you are a Windows user, the steps below might not work as expected; try developing using [WSL](https://learn.microsoft.com/en-us/windows/wsl/install)._ + +Start by [forking](https://docs.github.com/en/get-started/quickstart/fork-a-repo) the Polars repository, then clone your forked repository using `git`: + +```bash +git clone https://github.com//polars.git +cd polars +``` + +In order to work on Polars effectively, you will need [Rust](https://www.rust-lang.org/), [Python](https://www.python.org/), and [dprint](https://dprint.dev/). + +First, install Rust using [rustup](https://www.rust-lang.org/tools/install). +After the initial installation, you will also need to install the nightly toolchain: + +```bash +rustup toolchain install nightly --component miri +``` + +Next, install Python, for example using [pyenv](https://github.com/pyenv/pyenv#installation). +We recommend using the latest Python version (`3.11`). +Make sure you deactivate any active virtual environments or conda environments, as the steps below will create a new virtual environment for Polars. +You will need Python even if you intend to work on the Rust code only, as we rely on the Python tests to verify all functionality. + +Finally, install [dprint](https://dprint.dev/install/). +This is not strictly required, but it is recommended as we use it to autoformat certain file types. + +You can now check that everything works correctly by going into the `py-polars` directory and running the test suite +(warning: this may be slow the first time you run it): + +```bash +cd py-polars +make test +``` + +This will do a number of things: + +- Use Python to create a virtual environment in the `.venv` folder. +- Use [pip](https://pip.pypa.io/) to install all Python dependencies for development, linting, and building documentation. +- Use Rust to compile and install Polars in your virtual environment. _At least 8GB of RAM is recommended for this step to run smoothly._ +- Use [pytest](https://docs.pytest.org/) to run the Python unittests in your virtual environment + +Check if linting also works correctly by running: + +```bash +make pre-commit +``` + +Note that we do not actually use the [pre-commit](https://pre-commit.com/) tool. +We use the Makefile to conveniently run the following formatting and linting tools: + +- [black](https://black.readthedocs.io/) and [blackdoc](https://github.com/keewis/blackdoc) +- [ruff](https://github.com/charliermarsh/ruff) +- [mypy](http://mypy-lang.org/) +- [rustfmt](https://github.com/rust-lang/rustfmt) +- [clippy](https://doc.rust-lang.org/nightly/clippy/index.html) +- [dprint](https://dprint.dev/) + +If this all runs correctly, you're ready to start contributing to the Polars codebase! + +### Working on your issue + +Create a new git branch from the `main` branch in your local repository, and start coding! + +The Rust code is located in the `crates` directory, while the Python codebase is located in the `py-polars` directory. +Both directories contain a `Makefile` with helpful commands. Most notably: + +- `make test` to run the test suite (see the [test suite docs](https://github.com/pola-rs/polars/blob/main/py-polars/tests/README.md) for more info) +- `make pre-commit` to run autoformatting and linting + +Note that your work cannot be merged if these checks fail! +Run `make help` to get a list of other helpful commands. + +Two other things to keep in mind: + +- If you add code that should be tested, add tests. +- If you change the public API, [update the documentation](#api-reference). + +### Pull requests + +When you have resolved your issue, [open a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork) in the Polars repository. +Please adhere to the following guidelines: + +- Start your pull request title with a [conventional commit](https://www.conventionalcommits.org/) tag. This helps us add your contribution to the right section of the changelog. We use the [Angular convention](https://github.com/angular/angular/blob/22b96b9/CONTRIBUTING.md#type). Scope can be `rust` and/or `python`, depending on your contribution. +- Use a descriptive title. This text will end up in the [changelog](https://github.com/pola-rs/polars/releases). +- In the pull request description, [link](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue) to the issue you were working on. +- Add any relevant information to the description that you think may help the maintainers review your code. +- Make sure your branch is [rebased](https://docs.github.com/en/get-started/using-git/about-git-rebase) against the latest version of the `main` branch. +- Make sure all [GitHub Actions checks](https://github.com/pola-rs/polars/blob/main/.github/workflows/README.md) pass. + +After you have opened your pull request, a maintainer will review it and possibly leave some comments. +Once all issues are resolved, the maintainer will merge your pull request, and your work will be part of the next Polars release! + +Keep in mind that your work does not have to be perfect right away! +If you are stuck or unsure about your solution, feel free to open a draft pull request and ask for help. + +## Contributing to documentation + +The most important components of Polars documentation are the [user guide](https://pola-rs.github.io/polars/user-guide/), the API references, and the database of questions on [StackOverflow](https://stackoverflow.com/). + +### User guide + +The user guide is maintained in the `docs/user-guide` folder. Before creating a PR first raise an issue to discuss what you feel is missing or could be improved. + +#### Building and serving the user guide + +The user guide is built using [MkDocs](https://www.mkdocs.org/). You install the dependencies for building the user guide by running `make requirements` in the root of the repo. + +Run `mkdocs serve` to build and serve the user guide, so you can view it locally and see updates as you make changes. + +#### Creating a new user guide page + +Each user guide page is based on a `.md` markdown file. This file must be listed in `mkdocs.yml`. + +#### Adding a shell code block + +To add a code block with code to be run in a shell with tabs for Python and Rust, use the following format: + +```` +=== ":fontawesome-brands-python: Python" + + ```shell + $ pip install fsspec + ``` + +=== ":fontawesome-brands-rust: Rust" + + ```shell + $ cargo add aws_sdk_s3 + ``` +```` + +#### Adding a code block + +The snippets for Python and Rust code blocks are in the `docs/src/python/` and `docs/src/rust/` directories, respectively. To add a code snippet with Python or Rust code to a `.md` page, use the following format: + +``` +{{code_block('user-guide/io/cloud-storage','read_parquet',[read_parquet,read_csv])}} +``` + +- The first argument is a path to either or both files called `docs/src/python/user-guide/io/cloud-storage.py` and `docs/src/rust/user-guide/io/cloud-storage.rs`. +- The second argument is the name given at the start and end of each snippet in the `.py` or `.rs` file +- The third argument is a list of links to functions in the API docs. For each element of the list there must be a corresponding entry in `docs/_build/API_REFERENCE_LINKS.yml` + +If the corresponding `.py` and `.rs` snippet files both exist then each snippet named in the second argument to `code_block` above must exist or the build will fail. An empty snippet should be added to the `.py` or `.rs` file if the snippet is not needed. + +Each snippet is formatted as follows: + +```python +# --8<-- [start:read_parquet] +import polars as pl + +df = pl.read_parquet("file.parquet") +# --8<-- [end:read_parquet] +``` + +The snippet is delimited by `--8<-- [start:]` and `--8<-- [end:]`. The snippet name must match the name given in the second argument to `code_block` above. + +#### Linting + +Before committing, install `dprint` (see above) and run `dprint fmt` from the `docs` directory to lint the markdown files. + +### API reference + +Polars has separate API references for [Rust](https://pola-rs.github.io/polars/docs/rust/dev/polars/) and [Python](https://pola-rs.github.io/polars/docs/python/dev/reference/index.html). +These are generated directly from the codebase, so in order to contribute, you will have to follow the steps outlined in [this section](#contributing-to-the-codebase) above. + +#### Rust + +Rust Polars uses `cargo doc` to build its documentation. Contributions to improve or clarify the API reference are welcome. + +#### Python + +For the Python API reference, we always welcome good docstring examples. +There are still parts of the API that do not have any code examples. +This is a great way to start contributing to Polars! + +Note that we follow the [numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html) convention. +Docstring examples should also follow the [Black](https://black.readthedocs.io/) codestyle. +From the `py-polars` directory, run `make fmt` to make sure your additions pass the linter, and run `make doctest` to make sure your docstring examples are valid. + +Polars uses Sphinx to build the API reference. +This means docstrings in general should follow the [reST](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html) format. +If you want to build the API reference locally, go to the `py-polars/docs` directory and run `make html SPHINXOPTS=-W`. +The resulting HTML files will be in `py-polars/docs/build/html`. + +New additions to the API should be added manually to the API reference by adding an entry to the correct `.rst` file in the `py-polars/docs/source/reference` directory. + +### StackOverflow + +We use StackOverflow to create a database of high quality questions and answers that is searchable and remains up-to-date. +There is a separate tag for each language: + +- [Python Polars](https://stackoverflow.com/questions/tagged/python-polars) +- [Rust Polars](https://stackoverflow.com/questions/tagged/rust-polars) + +Contributions in the form of well-formulated questions or answers are always welcome! +If you add a new question, please notify us by adding a [matching issue](https://github.com/pola-rs/polars/issues/new?&labels=question&template=question.yml) to our GitHub issue tracker. + +## Release flow + +_This section is intended for Polars maintainers._ + +Polars releases Rust crates to [crates.io](https://crates.io/crates/polars) and Python packages to [PyPI](https://pypi.org/project/polars/). + +New releases are marked by an official [GitHub release](https://github.com/pola-rs/polars/releases) and an associated git tag. We utilize [Release Drafter](https://github.com/release-drafter/release-drafter) to automatically draft GitHub releases with release notes. + +### Steps + +The steps for releasing a new Rust or Python version are similar. The release process is mostly automated through GitHub Actions, but some manual steps are required. Follow the steps below to release a new version. + +Start by bumping the version number in the source code: + +1. Check the [releases page](https://github.com/pola-rs/polars/releases) on GitHub and find the appropriate draft release. Note the version number associated with this release. +2. Make sure your fork is up-to-date with the latest version of the main Polars repository, and create a new branch. +3. Bump the version number. + +- _Rust:_ Update the version number in all `Cargo.toml` files in the `polars` directory and subdirectories. You'll probably want to use some search/replace strategy, as there are quite a few crates that need to be updated. +- _Python:_ Update the version number in [`py-polars/Cargo.toml`](https://github.com/pola-rs/polars/blob/main/py-polars/Cargo.toml#L3) to match the version of the draft release. + +4. From the `py-polars` directory, run `make build` to generate a new `Cargo.lock` file. +5. Create a new commit with all files added. The name of the commit should follow the format `release(): Polars `. For example: `release(python): Python Polars 0.16.1` +6. Push your branch and open a new pull request to the `main` branch of the main Polars repository. +7. Wait for the GitHub Actions checks to pass, then squash and merge your pull request. + +Directly after merging your pull request, release the new version: + +8. Go to the release workflow ([Python](https://github.com/pola-rs/polars/actions/workflows/release-python.yml)/[Rust](https://github.com/pola-rs/polars/actions/workflows/release-rust.yml)), click _Run workflow_ in the top right, and click the green button. This will trigger the workflow, which will build all release artifacts and publish them. +9. Wait for the workflow to finish, then check [crates.io](https://crates.io/crates/polars)/[PyPI](https://pypi.org/project/polars/)/[GitHub](https://github.com/pola-rs/polars/releases) to verify that the new Polars release is now available. + +### Troubleshooting + +It may happen that one or multiple release jobs fail. If so, you should first try to simply re-run the failed jobs from the GitHub Actions UI. + +If that doesn't help, you will have to figure out what's wrong and commit a fix. Once your fix has made it to the `main` branch, simply re-trigger the release workflow. + +## License + +Any contributions you make to this project will fall under the [MIT License](https://github.com/pola-rs/polars/blob/main/LICENSE) that covers the Polars project. diff --git a/docs/development/versioning.md b/docs/development/versioning.md new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/mkdocs.yml b/mkdocs.yml index 7734cbd11d5b..1bf7264f892e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,6 +9,7 @@ repo_name: pola-rs/polars # Documentation layout nav: - Home: index.md + - Getting started: - getting-started/intro.md - getting-started/installation.md @@ -16,6 +17,7 @@ nav: - getting-started/reading-writing.md - getting-started/expressions.md - getting-started/joins.md + - User guide: - user-guide/index.md - user-guide/installation.md @@ -81,6 +83,11 @@ nav: - user-guide/misc/alternatives.md - user-guide/misc/reference-guides.md - user-guide/misc/contributing.md + + - Development: + - development/contributing.md + - development/versioning.md + not_in_nav: | /_build/ people.md From 86bdbd251ea8cb53e1334bc13c6550c38d7bc631 Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Sun, 22 Oct 2023 11:58:11 +0200 Subject: [PATCH 02/11] Add releases section --- docs/releases/changelog.md | 4 + docs/releases/migration/0.19.md | 199 ++++++++++++++++++++++++++++++++ mkdocs.yml | 5 + 3 files changed, 208 insertions(+) create mode 100644 docs/releases/changelog.md create mode 100644 docs/releases/migration/0.19.md diff --git a/docs/releases/changelog.md b/docs/releases/changelog.md new file mode 100644 index 000000000000..e2613837f463 --- /dev/null +++ b/docs/releases/changelog.md @@ -0,0 +1,4 @@ +# Changelog + +This page is under construction. +Please refer to our [GitHub releases](https://github.com/pola-rs/polars/releases) in the meantime. diff --git a/docs/releases/migration/0.19.md b/docs/releases/migration/0.19.md new file mode 100644 index 000000000000..914fb972fec9 --- /dev/null +++ b/docs/releases/migration/0.19.md @@ -0,0 +1,199 @@ +# 0.19 + +This document is intended to help you migrate from an older Polars version to Polars version `0.19.*`. + +For the full list of changes, please see the [release notes](https://github.com/pola-rs/polars/releases/tag/py-0.19.0). + +## Breaking changes + +While this section does not contain an exhaustive list of all breaking changes, we think these are most likely to impact your code. + +### Aggregation functions no longer support horizontal computation + +This impacts aggregation functions like `sum`, `min`, and `max`. +These functions were overloaded to support both vertical and horizontal computation. +Recently, new dedicated functionality for horizontal computation was released, and horizontal computation was deprecated. + +Restore the old behavior by using the horizontal variant, e.g. `sum_horizontal`. + +#### Example + +Before: + +```shell +>>> df = pl.DataFrame({'a': [1, 2], 'b': [11, 12]}) +>>> df.select(pl.sum('a', 'b')) # horizontal computation +shape: (2, 1) +┌─────┐ +│ sum │ +│ --- │ +│ i64 │ +╞═════╡ +│ 12 │ +│ 14 │ +└─────┘ +``` + +After: + +```shell +>>> df = pl.DataFrame({'a': [1, 2], 'b': [11, 12]}) +>>> df.select(pl.sum('a', 'b')) # vertical computation +shape: (1, 2) +┌─────┬─────┐ +│ a ┆ b │ +│ --- ┆ --- │ +│ i64 ┆ i64 │ +╞═════╪═════╡ +│ 3 ┆ 23 │ +└─────┴─────┘ +``` + +### Update to `all` / `any` + +`all` will now ignore null values by default, rather than treat them as `False`. + +For both `any` and `all`, the `drop_nulls` parameter has been renamed to `ignore_nulls` and is now keyword-only. +Also fixed an issue when setting this parameter to `False` would erroneously result in `None` output in some cases. + +To restore the old behavior, set `ignore_nulls` to `False` and check for `None` output. + +#### Example + +Before: + +```shell +>>> pl.Series([True, None]).all() +False +``` + +After: + +```shell +>>> pl.Series([True, None]).all() +True +``` + +### Improved error types for many methods + +Improving our error messages is an ongoing effort. +We did a sweep of our Python code base and made many improvements to error messages and error types. +Most notably, many `ValueError`s were changed to `TypeError`s. + +If your code relies on handling Polars exceptions, you may have to make some adjustments. + +#### Example + +Before: + +```shell +>>> pl.Series(values=15) +... +ValueError: Series constructor called with unsupported type; got 'int' +``` + +After: + +```shell +>>> pl.Series(values=15) +... +TypeError: Series constructor called with unsupported type 'int' for the `values` parameter +``` + +### Updates to expression input parsing + +Methods like `select` and `with_columns` accept one or more expressions. +But they also accept strings, integers, lists, and other inputs that we try to interpret as expressions. +We updated our internal logic to parse inputs more consistently. + +#### Example + +Before: + +```shell +>>> pl.DataFrame({'a': [1, 2]}).with_columns(None) +shape: (2, 1) +┌─────┐ +│ a │ +│ --- │ +│ i64 │ +╞═════╡ +│ 1 │ +│ 2 │ +└─────┘ +``` + +After: + +```shell +>>> pl.DataFrame({'a': [1, 2]}).with_columns(None) +shape: (2, 2) +┌─────┬─────────┐ +│ a ┆ literal │ +│ --- ┆ --- │ +│ i64 ┆ null │ +╞═════╪═════════╡ +│ 1 ┆ null │ +│ 2 ┆ null │ +└─────┴─────────┘ +``` + +### `shuffle` / `sample` now use an internal Polars seed + +If you used the built-in Python `random.seed` function to control the randomness of Polars expressions, this will no longer work. +Instead, use the new `set_random_seed` function. + +#### Example + +Before: + +```python +import random + +random.seed(1) +``` + +After: + +```python +import polars as pl + +pl.set_random_seed(1) +``` + +## Deprecations + +Creating a consistent and intuitive API is hard, finding the right name for each function, method, and parameter might be the hardest part. +The new version comes with quite some naming changes, and you will most likely run into deprecation warnings when upgrading to `0.19`. + +If you want to upgrade without worrying about deprecation warnings right now, you can add the following snippet to your code: + +```python +import warnings + +warnings.filterwarnings("ignore", category=DeprecationWarning) +``` + +### `groupby` renamed to `group_by` + +This is not a change we make lightly, as it will impact almost all our users. But "group by" are really two different words, and our naming strategy dictates that these should be separated by an underscore. + +Most likely, a simple search and replace will be enough to take care of this update: + +- Search: `.groupby(` +- Replace: `.group_by(` + +### `apply` renamed to `map_*` + +`apply` is probably the most misused part of our API. Many Polars users come from pandas, where `apply` has a completely different meaning. + +We now consolidate all our functionality for user-defined functions under the name `map`. This results in the following renaming: + +| Before | After | +| --------------------------- | -------------- | +| `Series/Expr.apply` | `map_elements` | +| `Series/Expr.rolling_apply` | `rolling_map` | +| `DataFrame.apply` | `map_rows` | +| `GroupBy.apply` | `map_groups` | +| `pl.apply` | `map_groups` | +| `map` | `map_batches` | diff --git a/mkdocs.yml b/mkdocs.yml index 1bf7264f892e..99de7338d3b1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -88,6 +88,11 @@ nav: - development/contributing.md - development/versioning.md + - Releases: + - releases/changelog.md + - Migration guide: + - releases/migration/0.19.md + not_in_nav: | /_build/ people.md From 51009864cb4377a466633d6ed501c1aa3c6d1fc8 Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Sun, 22 Oct 2023 14:14:09 +0200 Subject: [PATCH 03/11] Add breaking changes policy --- docs/development/versioning.md | 74 ++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/docs/development/versioning.md b/docs/development/versioning.md index e69de29bb2d1..320ecf99ae66 100644 --- a/docs/development/versioning.md +++ b/docs/development/versioning.md @@ -0,0 +1,74 @@ +# Versioning + +## Version changes + +Polars adheres to the [semantic versioning](https://semver.org/) specification. + +As Polars has not released its `1.0.0` version yet, breaking releases lead to a minor version increase (e.g. from `0.18.15` to `0.19.0`), while all other releases increment the patch version (e.g. from `0.18.14` to `0.18.15`) + +## Policy for breaking changes + +Polars takes backwards compatibility seriously, but we are not afraid to change things if it leads to a better product. + +!!! warning Rust users only + + The Rust API for Polars is currently not considered stable. + Functionality can be changed or removed without warning. + +### Philosophy + +We don't always get it right on the first try. +We learn as we go along and get feedback from our users. +Sometimes, we're a little too eager to get out a new feature and didn't ponder all the possible implications. + +If this happens, we correct our mistakes and introduce a breaking change. +Most of the time, this is no big deal. +Users get a deprecation warning, they do a quick search-and-replace in their code base, and that's that. + +At times, we run into an issue requires more effort on our user's part to fix. +A change in the query engine can seriously impact the assumptions in a data pipeline. +We do not make such changes lightly, but we will make them if we believe it makes Polars better. + +Freeing ourselves of past indiscretions is important to keep Polars moving forward. +We know it takes time and energy for our users to keep up with new releases, but in the end, it benefits everyone for Polars to be the best product possible. + +### What qualifies as a breaking change + +**A breaking change occurs when an existing component of the public API is changed or removed.** + +A feature is part of the public API if it is documented in the [API reference](https://pola-rs.github.io/polars/py-polars/html/reference/). + +Examples of breaking changes: + +* A deprecated function or method is removed. +* The default value of a parameter is changed. +* The outcome of a query has changed due to changes to the query engine. + +Examples of changes that are _not_ considered breaking: + +* An undocumented function is removed. +* The module path of a public class is changed. +* An optional parameter is added to an existing method. + +Bug fixes are not considered a breaking change, even though it may impact some users' [workflows](https://xkcd.com/1172/). + +### Deprecation warnings + +If we decide to introduce a breaking change, the existing behavior is deprecated _if possible_. +For example, if we choose to rename a function, the new function is added alongside the old function, and using the old function will result in a deprecation warning. + +Not all changes can be deprecated nicely. +A change to the query engine may have effects across a large part of the API. +Such changes will not be warned for, but *will* be included in the changelog and the migration guide. + +### Deprecation period + +As a rule, deprecated functionality is removed two breaking releases after the deprecation happens. +For example, a function deprecated in version `0.18.3` will be removed in version `0.20.0`. + +This means that if your program does not raise any deprecation warnings, it should be mostly safe to upgrade to the next breaking release. +As breaking releases happen about once every three months, this allows three to six months to adjust to any pending breaking changes. + +**In some cases, we may decide to adjust the deprecation period.** +If retaining the deprecated functionality blocks other improvements to Polars, we may shorten the deprecation period to a single breaking release. This will be mentioned in the warning message. +If the deprecation affects many users, we may extend the deprecation period. From 76f283505b0623bc03a0c7bd02c4b476de5b61bf Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Sun, 22 Oct 2023 14:22:08 +0200 Subject: [PATCH 04/11] Fix contributing --- docs/development/contributing.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/docs/development/contributing.md b/docs/development/contributing.md index bc892bab8910..0e87d1a9ca6e 100644 --- a/docs/development/contributing.md +++ b/docs/development/contributing.md @@ -3,15 +3,6 @@ Thanks for taking the time to contribute! We appreciate all contributions, from reporting bugs to implementing new features. If you're unclear on how to proceed after reading this guide, please contact us on [Discord](https://discord.gg/4UfP5cfBE7). -## Table of contents - -- [Reporting bugs](#reporting-bugs) -- [Suggesting enhancements](#suggesting-enhancements) -- [Contributing to the codebase](#contributing-to-the-codebase) -- [Contributing to documentation](#contributing-to-documentation) -- [Release flow](#release-flow) -- [License](#license) - ## Reporting bugs We use [GitHub issues](https://github.com/pola-rs/polars/issues) to track bugs and suggested enhancements. From 64251ae2b7a50490ec69feb2ce5386612b95162b Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Sun, 22 Oct 2023 14:28:58 +0200 Subject: [PATCH 05/11] dprint --- docs/development/versioning.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/development/versioning.md b/docs/development/versioning.md index 320ecf99ae66..3704afc0f5e8 100644 --- a/docs/development/versioning.md +++ b/docs/development/versioning.md @@ -40,15 +40,15 @@ A feature is part of the public API if it is documented in the [API reference](h Examples of breaking changes: -* A deprecated function or method is removed. -* The default value of a parameter is changed. -* The outcome of a query has changed due to changes to the query engine. +- A deprecated function or method is removed. +- The default value of a parameter is changed. +- The outcome of a query has changed due to changes to the query engine. Examples of changes that are _not_ considered breaking: -* An undocumented function is removed. -* The module path of a public class is changed. -* An optional parameter is added to an existing method. +- An undocumented function is removed. +- The module path of a public class is changed. +- An optional parameter is added to an existing method. Bug fixes are not considered a breaking change, even though it may impact some users' [workflows](https://xkcd.com/1172/). @@ -59,7 +59,7 @@ For example, if we choose to rename a function, the new function is added alongs Not all changes can be deprecated nicely. A change to the query engine may have effects across a large part of the API. -Such changes will not be warned for, but *will* be included in the changelog and the migration guide. +Such changes will not be warned for, but _will_ be included in the changelog and the migration guide. ### Deprecation period From 0b186e2c3424c2b4ce244d68c96fdec0570f08c7 Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Sun, 22 Oct 2023 14:43:45 +0200 Subject: [PATCH 06/11] Fix macro rendering --- docs/development/contributing.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/development/contributing.md b/docs/development/contributing.md index 0e87d1a9ca6e..471e3d7180f3 100644 --- a/docs/development/contributing.md +++ b/docs/development/contributing.md @@ -1,3 +1,7 @@ +--- +render_macros: false +--- + # Contributing Thanks for taking the time to contribute! We appreciate all contributions, from reporting bugs to implementing new features. @@ -177,7 +181,7 @@ To add a code block with code to be run in a shell with tabs for Python and Rust The snippets for Python and Rust code blocks are in the `docs/src/python/` and `docs/src/rust/` directories, respectively. To add a code snippet with Python or Rust code to a `.md` page, use the following format: ``` -{{code_block('user-guide/io/cloud-storage','read_parquet',[read_parquet,read_csv])}} +{{code_block('user-guide/io/cloud-storage','read_parquet',['read_parquet','read_csv'])}} ``` - The first argument is a path to either or both files called `docs/src/python/user-guide/io/cloud-storage.py` and `docs/src/rust/user-guide/io/cloud-storage.rs`. From 93cba87add79a0631ab35b349e40ecd14e61a6db Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Sun, 22 Oct 2023 14:45:45 +0200 Subject: [PATCH 07/11] Rename migration to upgrade --- docs/releases/{migration => upgrade}/0.19.md | 2 +- mkdocs.yml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) rename docs/releases/{migration => upgrade}/0.19.md (98%) diff --git a/docs/releases/migration/0.19.md b/docs/releases/upgrade/0.19.md similarity index 98% rename from docs/releases/migration/0.19.md rename to docs/releases/upgrade/0.19.md index 914fb972fec9..534b6b7d8f36 100644 --- a/docs/releases/migration/0.19.md +++ b/docs/releases/upgrade/0.19.md @@ -1,6 +1,6 @@ # 0.19 -This document is intended to help you migrate from an older Polars version to Polars version `0.19.*`. +This document is intended to help you upgrade from an older Polars version to Polars version `0.19.*`. For the full list of changes, please see the [release notes](https://github.com/pola-rs/polars/releases/tag/py-0.19.0). diff --git a/mkdocs.yml b/mkdocs.yml index 99de7338d3b1..b38c5bd4f41e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -90,8 +90,8 @@ nav: - Releases: - releases/changelog.md - - Migration guide: - - releases/migration/0.19.md + - Upgrade guide: + - releases/upgrade/0.19.md not_in_nav: | /_build/ From fc974e4e3322d7e95b6c09fcca5beae4fa2ffeb9 Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Sun, 22 Oct 2023 15:08:09 +0200 Subject: [PATCH 08/11] Configure MLC --- .github/workflows/docs-global.yml | 1 + docs/mlc-config.json | 7 +++++++ 2 files changed, 8 insertions(+) create mode 100644 docs/mlc-config.json diff --git a/.github/workflows/docs-global.yml b/.github/workflows/docs-global.yml index 823278186535..583c364b6d78 100644 --- a/.github/workflows/docs-global.yml +++ b/.github/workflows/docs-global.yml @@ -19,6 +19,7 @@ jobs: - uses: actions/checkout@v4 - uses: gaurav-nelson/github-action-markdown-link-check@v1 with: + config-file: docs/mlc-config.json folder-path: docs lint: diff --git a/docs/mlc-config.json b/docs/mlc-config.json new file mode 100644 index 000000000000..e77aed6c4d0e --- /dev/null +++ b/docs/mlc-config.json @@ -0,0 +1,7 @@ +{ + "ignorePatterns": [ + { + "pattern": "^https://crates.io/" + } + ] +} From 8c07adb541044b937f8f82ee705d44718c55901d Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Sun, 22 Oct 2023 15:46:52 +0200 Subject: [PATCH 09/11] Update reference --- docs/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/index.md b/docs/index.md index c5c3dfb1bfbf..3be5b9bb7224 100644 --- a/docs/index.md +++ b/docs/index.md @@ -58,9 +58,9 @@ See the results in h2oai's [db-benchmark](https://duckdblabs.github.io/db-benchm --8<-- "docs/people.md" -## Contribute +## Contributing -Thanks for taking the time to contribute! We appreciate all contributions, from reporting bugs to implementing new features. If you're unclear on how to proceed read our [contribution guide](https://github.com/pola-rs/polars/blob/main/CONTRIBUTING.md) or contact us on [discord](https://discord.com/invite/4UfP5cfBE7). +We appreciate all contributions, from reporting bugs to implementing new features. Read our [contributing guide](development/contributing.md) to learn more. ## License From 2f96579cee75b9b2484197eaa0c7b5d7b62af7e0 Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Mon, 23 Oct 2023 14:19:39 +0200 Subject: [PATCH 10/11] Apply suggestions from code review Co-authored-by: Alexander Beedie --- docs/development/versioning.md | 2 +- docs/releases/upgrade/0.19.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/development/versioning.md b/docs/development/versioning.md index 3704afc0f5e8..92bbf81ea1db 100644 --- a/docs/development/versioning.md +++ b/docs/development/versioning.md @@ -30,7 +30,7 @@ A change in the query engine can seriously impact the assumptions in a data pipe We do not make such changes lightly, but we will make them if we believe it makes Polars better. Freeing ourselves of past indiscretions is important to keep Polars moving forward. -We know it takes time and energy for our users to keep up with new releases, but in the end, it benefits everyone for Polars to be the best product possible. +We know it takes time and energy for our users to keep up with new releases but, in the end, it benefits everyone for Polars to be the best product possible. ### What qualifies as a breaking change diff --git a/docs/releases/upgrade/0.19.md b/docs/releases/upgrade/0.19.md index 534b6b7d8f36..2c57654945f5 100644 --- a/docs/releases/upgrade/0.19.md +++ b/docs/releases/upgrade/0.19.md @@ -163,8 +163,8 @@ pl.set_random_seed(1) ## Deprecations -Creating a consistent and intuitive API is hard, finding the right name for each function, method, and parameter might be the hardest part. -The new version comes with quite some naming changes, and you will most likely run into deprecation warnings when upgrading to `0.19`. +Creating a consistent and intuitive API is hard; finding the right name for each function, method, and parameter might be the hardest part. +The new version comes with several naming changes, and you will most likely run into deprecation warnings when upgrading to `0.19`. If you want to upgrade without worrying about deprecation warnings right now, you can add the following snippet to your code: @@ -176,7 +176,7 @@ warnings.filterwarnings("ignore", category=DeprecationWarning) ### `groupby` renamed to `group_by` -This is not a change we make lightly, as it will impact almost all our users. But "group by" are really two different words, and our naming strategy dictates that these should be separated by an underscore. +This is not a change we make lightly, as it will impact almost all our users. But "group by" is really two different words, and our naming strategy dictates that these should be separated by an underscore. Most likely, a simple search and replace will be enough to take care of this update: From 66070caea742fa0b4e2e75630456047742812935 Mon Sep 17 00:00:00 2001 From: Stijn de Gooijer Date: Tue, 24 Oct 2023 13:06:45 +0200 Subject: [PATCH 11/11] Update Rust API description --- docs/development/versioning.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/development/versioning.md b/docs/development/versioning.md index 92bbf81ea1db..de5383b5cc90 100644 --- a/docs/development/versioning.md +++ b/docs/development/versioning.md @@ -4,17 +4,12 @@ Polars adheres to the [semantic versioning](https://semver.org/) specification. -As Polars has not released its `1.0.0` version yet, breaking releases lead to a minor version increase (e.g. from `0.18.15` to `0.19.0`), while all other releases increment the patch version (e.g. from `0.18.14` to `0.18.15`) +As Polars has not released its `1.0.0` version yet, breaking releases lead to a minor version increase (e.g. from `0.18.15` to `0.19.0`), while all other releases increment the patch version (e.g. from `0.18.15` to `0.18.16`) ## Policy for breaking changes Polars takes backwards compatibility seriously, but we are not afraid to change things if it leads to a better product. -!!! warning Rust users only - - The Rust API for Polars is currently not considered stable. - Functionality can be changed or removed without warning. - ### Philosophy We don't always get it right on the first try. @@ -61,6 +56,11 @@ Not all changes can be deprecated nicely. A change to the query engine may have effects across a large part of the API. Such changes will not be warned for, but _will_ be included in the changelog and the migration guide. +!!! warning Rust users only + + Breaking changes to the Rust API are not deprecated first, but _will_ be listed in the changelog. + Supporting deprecated functionality would slow down development too much at this point in time. + ### Deprecation period As a rule, deprecated functionality is removed two breaking releases after the deprecation happens.