README

Zaku, a fast Task Queue for ML Workloads
========================================

To get a quick overview of what you can do with ``zaku``, check out the
following:

-  take a look at the basic tutorial or the tutorial for robotics:

   -  `Zaku Basics <tutorials/basics>`__

-  or try to take a look at the example gallery
   `here <examples/01_simple_queue>`__

Install zaku — the latest version is ``{VERSION}`` on
`pypi <https://pypi.org/project/zaku/%7BVERSION%7D/>`__.

.. code-block:: python

   pip install -U 'zaku[all]=={VERSION}'

**Setting Up Zaku Server**

The server script is installed as the command ``zaku``. First, take a
look at its options by running

.. code-block:: shell

   zaku -h

This should show you the documents on all of the arguments. Now, to
setup a zaku task queue server, run the following: This enables access
from other than localhost.

.. code-block:: shell

   zaku --host 0.0.0.0 --port 9000

**Adding Jobs**:

Supposed you have a TaskServer running at ``localhost:9000``.

.. code-block:: python

   from zaku import TaskQ

   queue = TaskQ(name="my-test-queue", uri="http://localhost:9000")

   for i in range(100):
       queue.add_job({"job_id": i, "seed": i * 100})

**Retrieving Jobs**:

.. code-block:: python

   from zaku import TaskQ

   queue = TaskQ(name="my-test-queue", uri="http://localhost:9000")

   job_id, job = queue.take()

Now, after you have finished the job, you need to mark the job for
completion. The way we do so is by calling

.. code-block:: python

   queue.mark_done(job_id)

Sometimes when you worker responsible for completeing the job encounters
a failure, you need to also put the job back into the queue so that
other workers can retry. You can do so by calling

.. code-block:: python

   queue.mark_reset()

Now, we offer a context manager ``TaskQ.pop``, which automatically
catches exceptions and resets the job (or marks it complete).

.. code-block:: python

   from zaku import TaskQ

   queue = TaskQ(name="my-test-queue", uri="http://localhost:9000")

   with queue.pop() as job:
     if job is None:
       print("No job available")
     
     print("Retrieved job:", job)

Developing Zaku (Optional)
--------------------------

If you want to develop zaku, you can install it in editable mode plus
dependencies relevant for building the documentations:

.. code-block:: shell

   cd zaku
   pip install -e '.[dev]'

To build the documentations, run

.. code-block:: shell

   make docs

Running the tests (specs)
~~~~~~~~~~~~~~~~~~~~~~~~~

To run the tests in the spec folder, first start a zaku task queue
server at local port ``9000``, with a redis-stack-server in the backing.

1. run the ``redis-stack-server``: ``shell     redis-stack-server``
2. run the zaku task queue at port ``9000``:
   ``shell     zaku --port 9000 --verbose``

Now you can run the tests by running

.. code-block:: shell

   pytest

In pycharm, you should see the following:

.. raw:: html

   <p align="center">

.. raw:: html

   </p>