Merge branch '3.x' into extended-find-files

.github/workflows/test.yml

@@ -57,7 +57,9 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - run: pip install '.[test]'
       - run: pytest --cov --disable-warnings
-      - uses: codecov/codecov-action@v3
+      - name: Upload coverage report to codecov
+        if: ${{ matrix.os == 'ubuntu-20.04' && matrix.python-version == '3.12' }}
+        uses: codecov/codecov-action@v3
 
   # End-to-end tests
   #

.gitignore

@@ -14,6 +14,7 @@ venv/
 .cache/
 .mypy_cache/
 .pytest_cache/
+.python-version
 
 .vagrant*
 

CHANGELOG.md

@@ -1,3 +1,44 @@
+# v3.1.1
+
+- Improve errors with 2.x style `@decorator` (vs `@decorator()`) functions
+- Document adding custom connectors (@simonhammes)
+- Add basic API example to docs (@pirate)
+- Fix sphinx warnings (@simonhammes)
+- Fix force & pull arguments in `git.worktree` operation
+- Fix `server.reboot` reconnection (@wackou)
+- Fix chroot/local connector non-utf file gets (@evoldstad)
+- Fix `AptSources` fact to parse components in order & with digits (@rsfzi)
+
+# v3.1
+
+Here's pyinfra 3.1 - a release primarily driven by contributors new and old - a HUGE THANK YOU to all of you who dedicate time to work on pushing pyinfra forward. New stuff:
+
+- Add `zfs` operations (`dataset`, `snapshot`, `volume`, `filesystem`) facts (`Pools`, `Datasets`, `Filesystems`, `Snapshots`, `Volumes`) (@taliaferro)
+- Add `flatpak` operations (`packages`) and facts (`FlatpakPackage`, `FlatpakPackages`) (@JustScreaMy)
+- Add `jinja_env_kwargs` argument to `files.template` operation (@DonDebonair)
+- Add using dictionaries as `@terraform` output (map from group -> hosts)
+- Add default `@terraform` output key - `pyinfra_inventory.value`, promote connector to beta
+- Add support for multiple keys in each `server.authorized_keys` file (@matthijskooijman)
+- Add print all dependency versions with `--support` flag (@kytta)
+
+Fixes:
+
+- Fix when `ssh_hostname` is set as override data, don't do inventory hostname check
+- Fix `apt.AptSources` parsing special characters (@CondensedTea)
+- Fix `server.reboot` connection detection (@bauen1 + @lemmi)
+- Fix systemd flagging of sockets running (@bauen1)
+- Fix mysql dump quoting (@simonhammes)
+- Fix tilde expansion in files facts (@simonhammes)
+- Fix host lookup check with SSH alias config (@simonhammes)
+- Fix crontab comparison (@andrew-d)
+
+Docs/internal tweaks:
+
+- Improve operations documentation (@bauen1)
+- Default to local machine if `user_name` set in systecmt (@bauen1)
+- Improve efficiency of Docker operations (@apecnascimento)
+- Shallow copy `host.data` data to mutation
+
 # v3.0.2
 
 - Fix `OperationMeta.did_change`: this is now a function as originally designed

docs/api/connectors.md

@@ -86,3 +86,14 @@ class LocalConnector(BaseConnector):
             bool: indicating success or failure.
         """
 ```
+
+## pyproject.toml
+
+In order for pyinfra to gain knowledge about your connector, you need to add the following snippet to your connector's `pyproject.toml`:
+
+```toml
+[project.entry-points.'pyinfra.connectors']
+# Key = Entry point name
+# Value = module_path:class_name
+custom = 'pyinfra_custom_connector.connector:LoggingConnector'
+```

docs/api/deploys.md

@@ -34,7 +34,7 @@ install_mariadb()
 
 ## Global Arguments
 
-Deploys accept the same [global arguments as as operations](../deploys.html#global-arguments) and they apply to every operation call within that deploy. Taking the above example:
+Deploys accept the same [global arguments as operations](../arguments) and they apply to every operation call within that deploy. Taking the above example:
 
 
 ```py

docs/api/index.rst

@@ -3,4 +3,64 @@ Using the API
 
 In addition to :doc:`the pyinfra CLI <../cli>`, pyinfra provides a full Python API. As of ``v3`` this API can be considered mostly stable. See the :doc:`./reference`.
 
-`An example of how to use the API can be found here <https://github.com/pyinfra-dev/pyinfra-examples/blob/main/.old/api_deploy.py>`_. Note: this is not yet tested and pending future updates.
+You can also reference `pyinfra's own main.py <https://github.com/pyinfra-dev/pyinfra/blob/3.x/pyinfra_cli/main.py>`_, and the `pyinfra API source code <https://github.com/pyinfra-dev/pyinfra/tree/3.x/pyinfra/api>`_.
+
+Full Example
+============
+
+`A full example of how to use the API can be found here <https://github.com/pyinfra-dev/pyinfra-examples/blob/main/.old/api_deploy.py>`_. (Note: this is not yet tested and is pending future updates)
+
+Basic Localhost Example
+=======================
+
+.. code:: python
+
+   from pyinfra.api import Config, Inventory, State
+   from pyinfra.api.connect import connect_all
+   from pyinfra.api.operation import add_op
+   from pyinfra.api.operations import run_ops
+   from pyinfra.api.facts import get_facts
+   from pyinfra.facts.server import Os
+   from pyinfra.operations import server
+
+   # Define your inventory (@local means execute on localhost using subprocess)
+   # https://docs.pyinfra.com/en/3.x/apidoc/pyinfra.api.inventory.html
+   inventory = Inventory((["@local"], {}))
+
+   # Define any config you need
+   # https://docs.pyinfra.com/en/3.x/apidoc/pyinfra.api.config.html
+   config = Config(SUDO=True)
+
+   # Set up the state object
+   # https://docs.pyinfra.com/en/3.x/apidoc/pyinfra.api.state.html
+   state = State(inventory=inventory, config=config)
+
+   # Connect to all the hosts
+   connect_all(state)
+
+   # Start adding operations
+   result1 = add_op(
+       state,
+       server.user,
+       user="pyinfra",
+       home="/home/pyinfra",
+       shell="/bin/bash",
+   )
+   result2 = add_op(
+       state,
+       server.shell,
+       name="Run some shell commands",
+       commands=["whoami", "echo $PATH", "bash --version"]
+   )
+
+   # And finally we run the ops
+   run_ops(state)
+
+   # add_op returns an OperationMeta for each op, letting you access stdout, stderr, etc. after they run
+   host = state.hosts.inventory['@local']
+   print(result1.changed, result1[host].stdout, result1[host].stderr)
+   print(result2.changed, result2[host].stdout, result2[host].stderr)
+
+   # We can also get facts for all the hosts
+   # https://docs.pyinfra.com/en/3.x/apidoc/pyinfra.api.facts.html
+   print(get_facts(state, Os))

docs/cli.md

@@ -46,7 +46,7 @@ By default pyinfra only prints high level information (this host connected, this
 
 ## Inventory
 
-When using pyinfra inventory can be provided directly via the command line or [defined in a file](./deploys.html#inventory). Both support the full range of [connectors](./connectors) and multiple hosts. Some CLI examples:
+When using pyinfra inventory can be provided directly via the command line or [defined in a file](./inventory-data). Both support the full range of [connectors](./connectors) and multiple hosts. Some CLI examples:
 
 ```sh
 # Load inventory hosts from a file

docs/compatibility.md

@@ -39,12 +39,36 @@ pyinfra aims to be compatible with all Unix-like operating systems and is curren
     * HardenedBSD 11
     * DragonflyBSD 5
 + OpenSUSE (leap15 + tumbleweed)
-+ macOS 10.15 (with [`@local` connector](connectors.html#local))
-+ Docker (with [`@docker` connector](connectors.html#docker))
++ macOS 10.15 (with [`@local` connector](./connectors/local))
++ Docker (with [`@docker` connector](./connectors/docker))
 
 In general, the only requirement on the remote side is shell access. POSIX commands are used where possible for facts and operations, so most of the ``server`` and ``files`` operations should work anywhere POSIX.
 
 
+## Upgrading pyinfra from ``2.x`` -> ``3.x``
+
+- Rename `_use_sudo_password` argument to `_sudo_password`
+- Deploy decorator must be called:
+
+```py
+# Old, 2.x decorator
+@deploy
+def mydeploy():
+    ...
+
+# New, 3.x decorator
+@deploy()
+def mydeploy():
+    ...
+```
+
+- Remove `@winrm` connector, will come back as [`pyinfra-windows`](https://github.com/pyinfra-dev/pyinfra-windows)
+
+## Upgrading pyinfra from ``1.x`` -> ``2.x``
+
+- Python 2.7 (finally!), 3.5 support dropped, Python 3.6 is now the minimum required version
+- The "deploy directory" concept has been removed - everything now executes from the current working directory which removes the ambiguous magic v1 used to pick a deploy directory. A new --chdir CLI flag has been added to set the working directory before pyinfra executes
+
 ## Upgrading pyinfra from ``0.x`` -> ``1.x``
 
 The move to `v1` removes a lot of old legacy functionality in pyinfra - there will be warnings written to the user in CLI mode if any of these are encountered. The full list can be seen [on the changelog](https://github.com/Fizzadar/pyinfra/blob/master/CHANGELOG.md#v1). In addition to these removals, `v1` brings a few major changes and deprecates the old methods. Again, pyinfra will output warnings when these are encountered, and support will be removed in `v2`.

docs/conf.py

@@ -46,6 +46,8 @@
     "primary_doc_version": "3.x",
 }
 
+myst_heading_anchors = 3
+
 templates_path = ["templates"]
 
 html_favicon = "static/logo_small.png"

docs/contributing.md

@@ -2,12 +2,12 @@
 
 🎉 Thank you for taking the time to contribute to pyinfra! 🎉
 
-Third party pull requests help expand pyinfra's functionality and are essential to it's continued growth. This guide should help get you started adding additional operations, facts and other functionality to pyinfra.
+Third party pull requests help expand pyinfra's functionality and are essential to its continued growth. This guide should help get you started adding additional operations, facts and other functionality to pyinfra.
 
 ## Guides
 
 + [How to write operations](api/operations)
-+ [How to write facts](api/facts)
++ [How to write facts](api/operations.md#facts)
 + [How to write connectors](api/connectors)
 + [API reference](api/reference)
 
@@ -23,7 +23,7 @@ Third party pull requests help expand pyinfra's functionality and are essential
 # python -m venv / pyenv virtualenv / virtualenv
 
 # Clone the repo
-git clone [email protected]:Fizzadar/pyinfra.git
+git clone [email protected]:pyinfra-dev/pyinfra.git
 
 # Install the package in editable mode with development requirements
 cd pyinfra

docs/install.md

@@ -1,3 +1,8 @@
+---
+# Ignore warning about this page not being included in any toctree
+orphan: true
+---
+
 # Installation
 
 ## Pip

docs/support.md

@@ -5,16 +5,16 @@ Need a hand with using pyinfra or have some questions? You can find official pyi
 + Chat in [the `pyinfra` Matrix room](https://matrix.to/#/#pyinfra:matrix.org)
 + Ask questions on [Stack Overflow](https://stackoverflow.com/questions/tagged/pyinfra) (be sure to use the `pyinfra` tag!)
 
-### I think I've got a bug!
+## I think I've got a bug!
 
 + Please have a quick scan of the [the compatibility page](./compatibility) to see if the issue is referenced there
 + Check [the issue list](https://github.com/Fizzadar/pyinfra/issues) to see if anyone else has this issue
 + Submit [a new bug report](https://github.com/Fizzadar/pyinfra/issues/new/choose)
 
-### I have an idea!
+## I have an idea!
 
 Please consider [submitting a feature request](https://github.com/Fizzadar/pyinfra/issues/new/choose).
 
-### I'd like to contribute!
+## I'd like to contribute!
 
 Fantastic! Please see [the contributing docs](./contributing).

docs/using-operations.rst

@@ -143,10 +143,27 @@ All operations return an operation meta object which provides information about
         user="myuser",
     )
 
+    create_otheruser = server.user(
+        name="Create user otheruser",
+        user="otheruser",
+    )
+
     server.shell(
-        name="Bootstrap user",
+        name="Bootstrap myuser",
         commands=["..."],
-        _if=create_user,
+        _if=create_user.did_change,
+    )
+
+    # A list can be provided to run an operation if **all** functions return true
+    server.shell(
+        commands=["echo 'Both myuser and otheruser changed'"],
+        _if=[create_user.did_change, create_otheruser.did_change],
+    )
+
+    # You can also build your own lamba functions to achieve, e.g. an OR condition
+    server.shell(
+        commands=["echo 'myuser or otheruser changed'"],
+        _if=lambda: create_user.did_change() or create_otheruser.did_change(),
     )
 
 Operation Output

docs/utils.py

@@ -7,7 +7,7 @@ def title_line(char, string):
 
 def format_doc_line(line):
     # Bold the <arg>: part of each line
-    line = re.sub(r"\+ ([0-9a-z_\/]+)(.*)", r"+ **\1**\2", line)
+    line = re.sub(r"\+ ([0-9a-z_\/\*]+)(.*)", r"+ **\1**\2", line)
 
     # Strip the first 4 characters (first indent from docstring)
     return line[4:]

pyinfra/api/arguments.py

@@ -137,7 +137,7 @@ def generate_env(config: "Config", value: dict) -> dict:
     ),
     "_chdir": ArgumentMeta(
         "Directory to switch to before executing the command.",
-        default=lambda _: "",
+        default=lambda _: None,
     ),
     "_env": ArgumentMeta(
         "Dictionary of environment variables to set.",

pyinfra/api/deploy.py

@@ -23,7 +23,7 @@
     from pyinfra.api.state import State
 
 
-def add_deploy(state: "State", deploy_func: Callable[..., Any], *args, **kwargs):
+def add_deploy(state: "State", deploy_func: Callable[..., Any], *args, **kwargs) -> None:
     """
     Prepare & add an deploy to pyinfra.state by executing it on all hosts.
 
@@ -54,12 +54,22 @@ def add_deploy(state: "State", deploy_func: Callable[..., Any], *args, **kwargs)
 P = ParamSpec("P")
 
 
-def deploy(name: Optional[str] = None, data_defaults=None):
+def deploy(
+    name: Optional[str] = None, data_defaults: Optional[dict] = None
+) -> Callable[[Callable[P, Any]], PyinfraOperation[P]]:
     """
     Decorator that takes a deploy function (normally from a pyinfra_* package)
     and wraps any operations called inside with any deploy-wide kwargs/data.
     """
 
+    if name and not isinstance(name, str):
+        raise PyinfraError(
+            (
+                "The `deploy` decorator must be called, ie `@deploy()`, "
+                "see: https://docs.pyinfra.com/en/3.x/compatibility.html#upgrading-pyinfra-from-2-x-3-x"  # noqa
+            )
+        )
+
     def decorator(func: Callable[P, Any]) -> PyinfraOperation[P]:
         func.deploy_name = name or func.__name__  # type: ignore[attr-defined]
         if data_defaults: