feature: learn how to start a worker pool

.gitignore

@@ -16,3 +16,5 @@ tmp
 
 # VS code
 .vscode
+# Doc
+site

docs/guides/index.md

@@ -1,6 +1,9 @@
 # Guides
 - dependency injection
 - test task
+- worker config
+- app config
+- broker configuration
 - perform complex progress update
 - distribute large tasks across multiple workers
 - read/write to/from DS index

docs/index.md

@@ -41,24 +41,11 @@ $ cd datashare-ml-worker-template
 $ curl -LsSf https://astral.sh/uv/install.sh | sh
 ---> 100%
 $ uv run ./scripts/worker_entrypoint.sh
-[INFO][10:26:52.063][icij_worker.backend.backend]: Loading worker configuration from env...
-[INFO][10:26:52.065][icij_worker.backend.backend]: worker config: {
-  "connection_timeout_s": 5.0,
-  "reconnection_wait_s": 5.0,
-  "rabbitmq_host": "localhost",
-  "rabbitmq_password": "*****",
-  "rabbitmq_port": 5672,
-  "rabbitmq_management_port": 15672,
-  "rabbitmq_user": "guest",
-  "rabbitmq_vhost": "%2F",
-  "rabbitmq_is_qpid": false,
-  "app_bootstrap_config_path": null,
-  "inactive_after_s": null,
-  "log_level": "INFO",
-  "task_queue_poll_interval_s": 1.0
+[INFO][icij_worker.backend.backend]: Loading worker configuration from env...
+...
 }
-[INFO][10:26:52.065][icij_worker.backend.mp]: starting 1 worker for app ml_worker.app.app
-... 
+[INFO][icij_worker.backend.mp]: starting 1 worker for app ml_worker.app.app
+...
 ```
 you'll then be able to execute task by starting using our [HTTP client]() (and soon using Datashare's UI).    
 

docs/learn/app.md

@@ -1,5 +1,5 @@
-# Creating an async app featuring multiple tasks
-[//]: # (TODO: put ref to the app concept)
+# Building an app featuring multiple tasks
+
 Let's recap, in the previous [section](tasks.md), we learnt how to:
 
 - transform Python functions into asynchronous tasks
@@ -9,9 +9,9 @@ Let's recap, in the previous [section](tasks.md), we learnt how to:
 
 ## Full async app
 
-If we put everything together, we can build a full async app, featuring multiple tasks:
+If we put everything together, we can build a full [async app](../concepts-basic#app), featuring multiple tasks:
 
-```python
+```python title="my_app.py"
 --8<--
 hello_world_app.py:app
 --8<--

docs/learn/run-worker.md

@@ -1,10 +1,72 @@
-# Running your app with the `icij-worker` CLI
+# Running a worker pool with the [`icij-worker`](https://github.com/ICIJ/icij-python/tree/main/icij-worker) CLI
 
 In the previous [section](app.md), we've created a simple async app featuring multiple tasks.
 
-[//]: # (TODO: put ref to the worker pool concept)
-We now need start a worker pool, in order to run the app. The workers will execute the app tasks when task execution is requested.
+We now need start a [worker pool](./concepts-advanced#worker-pools), in order to run the app.
+Workers will execute the app tasks when new tasks are published by the [task manager](./concepts-basic.md#task-manager) through the [broker](./concepts-advanced.md#broker).
 
 ## Installing the `icij-worker` CLI
 
-## Start a worker pool using the `icij-worker` CLI
+Once your app is created you can use your favorite package manager to install [`icij-worker`](https://github.com/ICIJ/icij-python/tree/main/icij-worker):
+<!-- termynal -->
+```
+$ python -m pip install icij-worker 
+```
+
+
+## Start the worker pool
+
+To start a worker pool we'll need to provide a config. Config can be provided via environment variables or using a config files.
+Check out our [worker config guide](../guides/worker-config) in order to learn about all config parameters as well as env var naming conventions.  
+
+### Using environment variables 
+
+Since we've put all of our app code in a variable named `my_app` in a file named `app.py`, we can start the app as following:
+<!-- termynal -->
+```
+$ ICIJ_WORKER_TYPE=amqp python -m icij-worker workers start -n 2 app.app
+[INFO][icij_worker.backend.backend]: Loading worker configuration from env...
+[INFO][icij_worker.backend.backend]: worker config: {
+  ...
+}
+[INFO][icij_worker.backend.mp]: starting 2 worker for app my_app.app
+... 
+```
+!!! note
+    The above code assumes that an AMQP [broker](./concepts-advanced.md#broker) is running on the default host and ports.
+    Check out the [broker config guide](../guides/broker-config) to make sure your worker can correctly connect to it.
+
+Let's break this down a bit
+```bash
+$ ICIJ_WORKER_TYPE=amqp python -m icij-worker workers start -n 2 app.app
+
+```
+
+### Using a config file
+
+Sometimes it can be convenient to configure the worker pool via a config file rather than env vars,
+in this case you can just drop your config into a file:
+
+```json title="app_config.json"
+{
+  "type": "amqp"
+}
+```
+and start the pool like this:
+<!-- termynal -->
+```
+$ python -m icij-worker workers start -c app_config.json -n 2 app.app
+[INFO][icij_worker.backend.backend]: Loading worker configuration from env...
+[INFO][icij_worker.backend.backend]: worker config: {
+  ...
+}
+[INFO][icij_worker.backend.mp]: starting 2 worker for app my_app.app
+...
+```
+
+## Next
+
+Here, we've built a basic app and run it using a worker pool.
+However, the app we've built is not able yet to speak with Datashare's [Task Manager](concepts-basic.md#task-manager).
+
+Continue this tutorial to learn how implement task which can be integrated inside Datashare !

docs/learn/tasks.md

@@ -2,7 +2,7 @@
 
 ## Turning functions into async tasks using [`icij-worker`](https://github.com/ICIJ/icij-python/tree/main/icij-worker)
 
-Before implementing task for Datashare, let's learn how to transform any Python function into an task which can be run asynchronously by a worker using the [icij-worker](https://github.com/ICIJ/icij-python/tree/main/icij-worker) lib.
+Before implementing task for Datashare, let's learn how to transform any Python function into a [task](concepts-basic.md#tasks) which can be run [asynchronously](concepts-basic.md#asynchronous) by a [worker](concepts-basic.md#workers) using the [icij-worker](https://github.com/ICIJ/icij-python/tree/main/icij-worker) lib.
 
 Given the following function:
 ```python
@@ -53,8 +53,7 @@ hello_world_app.py:hello_user
 The task result is simply the value returned by the decorated function.
 
 ## Supported argument and result types
-[//]: # (TODO: put ref to the async concept)
-[//]: # (TODO: put ref to the task concept)
+
 
 `icij-worker` tasks support any types as arguments and result **as long as they are JSON serializable**.
 

mkdocs.yml

@@ -61,17 +61,18 @@ markdown_extensions:
   - toc:
       permalink: true
 plugins:
+  - search
   - termynal
 nav:
-  - Datashare Python: index.md
+  - Home: index.md
   - Learn:
     - Concepts and definitions:
       - Basic: learn/concepts-basic.md
       - Advanced: learn/concepts-advanced.md
     - From functions to tasks:
       - Creating async tasks: learn/tasks.md
-      - Creating an async app: learn/app.md
-#      - Running your app with the icij-worker CLI: learn/run-worker.md
+      - Building an app: learn/app.md
+      - Running a worker pool with the icij-worker CLI: learn/run-worker.md
 #    - Creating Datashare tasks: learn/datashare-app.md
 #    - Execute your tasks: learn/execute-tasks.md
 #  - Get started: get-started/index.md