Feat: makes AWSbatch the default executor (#161)

* modify changelog and version

* changes

* small fixes

* change warning

* use cloudos queue default

* change default queue

* fix pytest

* changelog

* changelog improvement

* adding badge

* adding badge

* adding badge

* adding badge

* adding badge2

* adding badge3

* fail CI

* adding proper badge

* correct CI

* remove batch from CI test

* fix message string

* fixing queue selection

.github/workflows/ci.yml

@@ -1,4 +1,4 @@
-name: CI tests
+name: CI_tests
 
 on:
   pull_request:
@@ -86,7 +86,7 @@ jobs:
         INSTANCE_TYPE: "c5.large"
       run: |
         JOB_NAME="$JOB_NAME_BASE""|GitHubCommit:""${COMMIT_HASH:0:6}""|PR-NUMBER:""$PR_NUMBER"
-        cloudos job run --cloudos-url $CLOUDOS_URL --apikey $CLOUDOS_TOKEN --workspace-id $CLOUDOS_WORKSPACE_ID --project-name "$PROJECT_NAME" --workflow-name "$WORKFLOW" --job-config $JOB_CONFIG --job-name "$JOB_NAME" --instance-type $INSTANCE_TYPE --spot --batch 2>&1 | tee out.txt
+        cloudos job run --cloudos-url $CLOUDOS_URL --apikey $CLOUDOS_TOKEN --workspace-id $CLOUDOS_WORKSPACE_ID --project-name "$PROJECT_NAME" --workflow-name "$WORKFLOW" --job-config $JOB_CONFIG --job-name "$JOB_NAME" --instance-type $INSTANCE_TYPE --spot 2>&1 | tee out.txt
         JOB_ID=$(grep -e "Your assigned job id is:" out.txt | rev | cut -f1 -d " " | rev)
         cloudos job status --cloudos-url $CLOUDOS_URL --apikey $CLOUDOS_TOKEN --job-id $JOB_ID
   workflow_list:

CHANGELOG.md

@@ -1,5 +1,12 @@
 ## lifebit-ai/cloudos-cli: changelog
 
+## v2.7.0 (2024-03-21)
+
+### Feature
+
+- Changes the default Nextflow executor to be AWSbatch. Now, the `--batch` flag is no longer necessary (although it's maintained for backwards compatibility) and a new `--ignite` flag is created to support ignite if available.
+- Selects the CloudOS workspace default queue, when no valid `--job-queue` is provided.
+
 ## v2.6.3 (2024-03-19)
 
 ### Fix

README.md

@@ -1,4 +1,6 @@
-# cloudos
+# cloudos-cli
+
+[![CI_tests](https://github.com/lifebit-ai/cloudos-cli/actions/workflows/ci.yml/badge.svg)](https://github.com/lifebit-ai/cloudos-cli/actions/workflows/ci.yml)
 
 Python package for interacting with CloudOS
 
@@ -103,8 +105,14 @@ Options:
   --job-name TEXT                 The name of the job. Default=new_job.
   --resumable                     Whether to make the job able to be resumed
                                   or not.
-  --batch                         Whether to make use the batch executor
-                                  instead of the default ignite.
+  --batch                         [Deprecated in 2.7.0] Since v2.7.0, the
+                                  default executor is AWSbatch so there is no
+                                  need to use this flag. It is maintained for
+                                  backwards compatibility.
+  --ignite                        This flag allows running ignite executor if
+                                  available. Please, note that ignite executor
+                                  is being deprecated and may not be available
+                                  in your CloudOS.
   --job-queue TEXT                Name of the job queue to use with a batch
                                   job.
   --instance-type TEXT            The type of execution platform compute
@@ -263,10 +271,8 @@ Executing run...
 
 #### Executor support
 
-CloudOS supports Apache [ignite](https://www.nextflow.io/docs/latest/ignite.html#apache-ignite) and
-[AWS batch](https://www.nextflow.io/docs/latest/executor.html?highlight=executors#aws-batch) executors.
-When using `cloudos job run` command, the default executor will be ignite.
-To choose AWS batch, you can use the flag `--batch`. In addition, you can specify the AWS batch queue to
+CloudOS supports [AWS batch](https://www.nextflow.io/docs/latest/executor.html?highlight=executors#aws-batch) executor by default.
+You can specify the AWS batch queue to
 use, from the ones available in your workspace (see [here](#get-a-list-of-the-available-job-queues))
 by specifying its name with the `--job-queue` parameter.
 If none is specified, the most recent suitable queue in your workspace will be selected by default.
@@ -280,18 +286,22 @@ cloudos job run \
     --project-name "$PROJECT_NAME" \
     --workflow-name $WORKFLOW_NAME \
     --job-config $JOB_PARAMS \
-    --resumable \
-    --batch
+    --resumable
 ```
 
-#### Execution platform support
+> Note: from cloudos-cli 2.7.0, the default executor is AWS batch. The previous Apache [ignite](https://www.nextflow.io/docs/latest/ignite.html#apache-ignite)
+> executor is being removed progressively from CloudOS, so most likely will not be available in your CloudOS. Cloudos-cli is still supporting ignite during this
+> period, by adding the `--ignite` flag to the `cloudos job run` command. Please, note that if you use `--ignite` flag in a CloudOS without ignite support,
+> the command fail.
+
+#### Azure execution platform support
 
-CloudOS can be configured to use AWS (Amazon Web Services) or Microsoft Azure compute platforms.
-By default `cloudos-cli` is configured to use AWS execution platform. If your CloudOS is configured to
+CloudOS can also be configured to use Microsoft Azure compute platforms.
+If your CloudOS is configured to
 use Azure, you will need to take into consideration the following:
 
 - When sending jobs to CloudOS using `cloudos job run` or `cloudos job run-curated-examples` commands, please use the option `--execution-platform azure`.
-- Options `--batch` and `--spot` are not taking effect when using Azure execution platform.
+- Option `--spot` is not taking effect when using Azure execution platform.
 - Due to the lack of AWS batch queues in Azure, `cloudos queue list` command is not working.
 
 Other than that, `cloudos-cli` will work very similarly. For instance, this is a typical send job command:

cloudos/__main__.py

@@ -138,7 +138,14 @@ def queue():
               help='Whether to make the job able to be resumed or not.',
               is_flag=True)
 @click.option('--batch',
-              help='Whether to make use the batch executor instead of the default ignite.',
+              help=('[Deprecated in 2.7.0] Since v2.7.0, the default executor is AWSbatch ' +
+                    'so there is no need to use this flag. It is maintained for ' +
+                    'backwards compatibility.'),
+              is_flag=True)
+@click.option('--ignite',
+              help=('This flag allows running ignite executor if available. Please, note ' +
+                    'that ignite executor is being deprecated and may not be available in your ' +
+                    'CloudOS.'),
               is_flag=True)
 @click.option('--job-queue',
               help='Name of the job queue to use with a batch job.')
@@ -216,6 +223,7 @@ def run(apikey,
         job_name,
         resumable,
         batch,
+        ignite,
         job_queue,
         nextflow_profile,
         instance_type,
@@ -246,6 +254,13 @@ def run(apikey,
     if execution_platform == 'azure':
         batch = None
         spot = None
+    elif ignite:
+        batch = None
+        print('\n[Warning] You have specified ignite executor. Please, note that ignite is being ' +
+              'removed from CloudOS, so the command may fail. Check ignite availability in your ' +
+              'CloudOS\n')
+    else:
+        batch = True
     if verbose:
         print('\t...Detecting workflow type')
     cl = Cloudos(cloudos_url, apikey, cromwell_token)
@@ -364,7 +379,14 @@ def run(apikey,
               help='Whether to make the job able to be resumed or not.',
               is_flag=True)
 @click.option('--batch',
-              help='Whether to make use the batch executor instead of the default ignite.',
+              help=('[Deprecated in 2.7.0] Since v2.7.0, the default executor is AWSbatch ' +
+                    'so there is no need to use this flag. It is maintained for ' +
+                    'backwards compatibility.'),
+              is_flag=True)
+@click.option('--ignite',
+              help=('This flag allows running ignite executor if available. Please, note ' +
+                    'that ignite executor is being deprecated and may not be available in your ' +
+                    'CloudOS.'),
               is_flag=True)
 @click.option('--instance-type',
               help=('The type of execution platform compute instance to use. ' +
@@ -423,6 +445,7 @@ def run_curated_examples(apikey,
                          project_name,
                          resumable,
                          batch,
+                         ignite,
                          instance_type,
                          instance_disk,
                          spot,
@@ -455,6 +478,13 @@ def run_curated_examples(apikey,
     if execution_platform == 'azure':
         batch = None
         spot = None
+    elif ignite:
+        batch = None
+        print('\n[Warning] You have specified ignite executor. Please, note that ignite is being ' +
+              'removed from CloudOS, so the command may fail. Check ignite availability in your ' +
+              'CloudOS\n')
+    else:
+        batch = True
     for workflow in runnable_curated_workflows:
         workflow_name = workflow['name']
         j = jb.Job(cloudos_url, apikey, None, workspace_id, project_name, workflow_name,

cloudos/_version.py

@@ -1 +1 @@
-__version__ = '2.6.3'
+__version__ = '2.7.0'

cloudos/jobs/job.py

@@ -231,7 +231,7 @@ def convert_nextflow_to_json(self,
         resumable: bool
             Whether to create a resumable job or not.
         batch: bool
-            Whether to create a batch job instead of the default ignite.
+            Whether to create a batch job or an ignite one.
         job_queue_id : string
             Job queue Id to use in the batch job.
         nextflow_profile: string
@@ -423,7 +423,7 @@ def send_job(self,
                  git_tag=None,
                  job_name='new_job',
                  resumable=False,
-                 batch=False,
+                 batch=True,
                  job_queue_id=None,
                  nextflow_profile=None,
                  instance_type='c5.xlarge',
@@ -461,7 +461,7 @@ def send_job(self,
         resumable: bool
             Whether to create a resumable job or not.
         batch: bool
-            Whether to create a batch job instead of the default ignite.
+            Whether to create a batch job or an ignite one.
         job_queue_id : string
             Job queue Id to use in the batch job.
         nextflow_profile: string

cloudos/queue/queue.py

@@ -70,6 +70,7 @@ def process_queue_list(r, all_fields=False):
                    'name',
                    'label',
                    'description',
+                   'isDefault',
                    'resourceType',
                    'executor',
                    'status'
@@ -81,20 +82,21 @@ def process_queue_list(r, all_fields=False):
             df = df_full.loc[:, COLUMNS]
         return df
 
-    def fetch_job_queue_id(self, workflow_type, batch=False, job_queue=None):
+    def fetch_job_queue_id(self, workflow_type, batch=True, job_queue=None):
         """Fetches CloudOS ID for a given job queue.
 
-        When batch=True and isinstance(job_queue, str) this method will try to find the
+        This method will try to find the
         corresponding CloudOS ID for the job_queue in a given workspace. If
-        job_queue=None, this method will select the newest "ready" job queue
-        for the selected workflow type.
+        job_queue=None, this method will select the available default queue in
+        the workspace, or the newest "ready" job queue if no default queues are
+        available.
 
         Parameters
         ----------
         workflow_type : str ['wdl'|'cromwell'|'nextflow']
             The type of workflow to run.
         batch: bool
-            Whether to create a batch job instead of the default ignite.
+            Whether to create a batch job or an ignite one.
         job_queue : str or None
             The name of the job queue to search. If None, a default one will be selected.
 
@@ -116,15 +118,22 @@ def fetch_job_queue_id(self, workflow_type, batch=False, job_queue=None):
         if len(available_queues) == 0:
             raise Exception(f'[ERROR] There are no available job queues for {workflow_type} ' +
                             'workflows. Consider creating one using CloudOS UI.')
-        default_queue_id = available_queues[-1]['id']
-        default_queue_name = available_queues[-1]['label']
+        default_queue = [q for q in available_queues if q['isDefault']]
+        if len(default_queue) > 0:
+            default_queue_id = default_queue[0]['id']
+            default_queue_name = default_queue[0]['label']
+            queue_as_default = 'CloudOS default'
+        else:
+            default_queue_id = available_queues[-1]['id']
+            default_queue_name = available_queues[-1]['label']
+            queue_as_default = 'most recent suitable'
         if job_queue is None:
-            print('\tNo job_queue was specified, using the most recent suitable one: ' +
+            print(f'\tNo job_queue was specified, using the {queue_as_default} queue: ' +
                   f'{default_queue_name}.')
             return default_queue_id
         selected_queue = [q for q in available_queues if q['label'] == job_queue]
         if len(selected_queue) == 0:
-            print(f'\tQueue \'{job_queue}\' you specified was not found, using the most recent ' +
-                  f'suitable one instead: {default_queue_name}.')
+            print(f'\tQueue \'{job_queue}\' you specified was not found, using the {queue_as_default} ' +
+                  f'queue instead: {default_queue_name}.')
             return default_queue_id
         return selected_queue[0]['id']

tests/test_data/queue/process_queue_list_results.csv

@@ -1,2 +1,2 @@
-id,name,label,description,resourceType,executor,status
-xxxxx,test_queue,test_queue_label,,,nextflow,Ready
+id,name,label,description,isDefault,resourceType,executor,status
+xxxxx,test_queue,test_queue_label,,True,,nextflow,Ready

tests/test_data/queue/process_queue_list_results_FULL.csv

@@ -1,2 +1,2 @@
-id,resource,name,label,description,resourceType,executor,arn,computeEnvironments,status
-xxxxx,xxxx,test_queue,test_queue_label,,,nextflow,,"[{'label': 'Default', 'environment': {}, 'status': 'Ready', 'createdAt': '2023-04-13T12:23:47.339Z', 'updatedAt': '2023-04-13T12:23:54.032Z'}]",Ready
+id,resource,name,label,description,isDefault,resourceType,executor,arn,computeEnvironments,status
+xxxxx,xxxx,test_queue,test_queue_label,,True,,nextflow,,"[{'label': 'Default', 'environment': {}, 'status': 'Ready', 'createdAt': '2023-04-13T12:23:47.339Z', 'updatedAt': '2023-04-13T12:23:54.032Z'}]",Ready

tests/test_data/queue/queues.json

@@ -1 +1 @@
-[{"id":"xxxxx","resource":"xxxx","name":"test_queue","label":"test_queue_label","description":"","resourceType":"","executor":"nextflow","arn":"","computeEnvironments":[{"label":"Default","environment":{},"status":"Ready","createdAt":"2023-04-13T12:23:47.339Z", "updatedAt":"2023-04-13T12:23:54.032Z"}],"status":"Ready"}]
+[{"id":"xxxxx","resource":"xxxx","name":"test_queue","label":"test_queue_label","description":"","isDefault": true,"resourceType":"","executor":"nextflow","arn":"","computeEnvironments":[{"label":"Default","environment":{},"status":"Ready","createdAt":"2023-04-13T12:23:47.339Z", "updatedAt":"2023-04-13T12:23:54.032Z"}],"status":"Ready"}]