Allowed for multiple service handling and fixed in_progress update

CHANGELOG.md

@@ -48,20 +48,23 @@ Code contributions to the new version:
 - Fixed singularity-images path when updating pangolin database in lablog_viralrecon. Added line break after prompted input. [#282](https://github.com/BU-ISCIII/buisciii-tools/pull/282)
 - Updated characterization and snippy templates to fit bacass pipeline. Corrected path in 05-iqtree in snippy template. [#283](https://github.com/BU-ISCIII/buisciii-tools/pull/283)
 - Included multiqc_report.html in RESULTS folder in every service, where necessary [#265] (https://github.com/BU-ISCIII/buisciii-tools/pull/265)
+- Added MAG tempalte and removed MAG from other templates [#288](https://github.com/BU-ISCIII/buisciii-tools/pull/288)
 
 ### Modules
 
 #### Added enhancements
 
 - PR [#274](https://github.com/BU-ISCIII/buisciii-tools/pull/274): added `--dev` option, configuration dev and test folder structure.
 - PR [#276](https://github.com/BU-ISCIII/buisciii-tools/pull/276): wkhtmlpdf does not need absolute path to executable. Added better error handling when executable does not exists.
+- PR [#288](https://github.com/BU-ISCIII/buisciii-tools/pull/288) Allowed to handle more than one service at a time, related to issue [#217](https://github.com/BU-ISCIII/buisciii-tools/issues/217)
 
 #### Fixes
 
 - Fixed archive module. Updated correct header for scout tsv [#258](https://github.com/BU-ISCIII/buisciii-tools/pull/258).
 - Fixed clean module. Corrected purge_files function. Renaming stage moved from clean to rename_nocopy option. Updated services.json file with correct paths for some services. [#280](https://github.com/BU-ISCIII/buisciii-tools/pull/280)
 - Fixed autoclean-sftp function. [#281](https://github.com/BU-ISCIII/buisciii-tools/pull/281)
 - Fixed bioinfo_doc.py. Modified it so that this module creates a .pdf file including new-line characters, without merging lines into one single line [#259](https://github.com/BU-ISCIII/buisciii-tools/pull/259).
+- PR [#288](https://github.com/BU-ISCIII/buisciii-tools/pull/288) Fixed updating service's state to in_progress multiple times, related with issue [#285](https://github.com/BU-ISCIII/buisciii-tools/issues/285)
 
 #### Changed
 

bu_isciii/archive.py

@@ -224,7 +224,7 @@ def __init__(
             try:
                 for service in rest_api.get_request(
                     request_info="services",
-                    safe=False,
+                    safe=True,
                     state="delivered",
                     date_from=str(self.date_from),
                     date_until=str(self.date_until),
@@ -259,7 +259,7 @@ def __init__(
                 if isinstance(
                     (
                         service_data := rest_api.get_request(
-                            request_info="service-data", safe=False, service=service
+                            request_info="service-data", safe=True, service=service
                         )
                     ),
                     int,

bu_isciii/bioinfo_doc.py

@@ -77,7 +77,7 @@ def __init__(
             conf_api["server"], conf_api["api_url"], api_user, api_password
         )
         self.resolution_info = self.rest_api.get_request(
-            request_info="service-data", safe=False, resolution=self.resolution_id
+            request_info="service-data", safe=True, resolution=self.resolution_id
         )
         if self.resolution_info == 404:
             print("Received Error 404 from Iskylims API. Aborting")
@@ -92,7 +92,7 @@ def __init__(
             else:
                 self.post_delivery_info()
         self.resolution_info = self.rest_api.get_request(
-            request_info="service-data", safe=False, resolution=self.resolution_id
+            request_info="service-data", safe=True, resolution=self.resolution_id
         )
         self.services_requested = self.resolution_info["resolutions"][0][
             "available_services"

bu_isciii/clean.py

@@ -47,7 +47,7 @@ def __init__(
             conf_api["server"], conf_api["api_url"], api_user, api_password
         )
         self.resolution_info = rest_api.get_request(
-            request_info="service-data", safe=False, resolution=self.resolution_id
+            request_info="service-data", safe=True, resolution=self.resolution_id
         )
         self.service_folder = self.resolution_info["resolutions"][0][
             "resolution_full_number"
@@ -129,10 +129,16 @@ def get_clean_items(self, services_ids, type="files"):
         for service in services_ids:
             try:
                 items = service_conf.get_find_deep(service, type)
-                if len(clean_items_list) == 0 and len(items) > 0:
-                    clean_items_list = items
-                elif len(items) > 0:
-                    clean_items_list.append(items)
+                if items:
+                    for item in items:
+                        if item not in clean_items_list:
+                            clean_items_list.append(item)
+                else:
+                    stderr.print(
+                        "[red]ERROR: Service type %s not found in services json file for service %s."
+                        % (type, service)
+                    )
+                    sys.exit()
             except KeyError as e:
                 stderr.print(
                     "[red]ERROR: Service id %s not found in services json file."

bu_isciii/config_json.py

@@ -24,7 +24,6 @@ def get_configuration(self, topic):
 
     def get_find(self, topic, found):
         """
-        Owner: Pablo
         Description:
             Obtain from topic any forward items from json data
         """

bu_isciii/copy_sftp.py

@@ -50,7 +50,7 @@ def __init__(
         )
 
         self.resolution_info = rest_api.get_request(
-            request_info="service-data", safe=False, resolution=self.resolution_id
+            request_info="service-data", safe=True, resolution=self.resolution_id
         )
         if sftp_folder is None:
             self.sftp_folder = bu_isciii.utils.get_sftp_folder(
@@ -112,8 +112,9 @@ def get_last_folders(self, services_ids, type="last_folder"):
         last_folders_list = []
         for service in services_ids:
             try:
-                items = service_conf.get_find_deep(service, type)
-                last_folders_list.append(items)
+                item = service_conf.get_find_deep(service, type)
+                if item not in last_folders_list:
+                    last_folders_list.append(item)
             except KeyError as e:
                 stderr.print(
                     "[red]ERROR: Service id %s not found in services json file."

bu_isciii/new_service.py

@@ -54,7 +54,7 @@ def __init__(
             conf_api["server"], conf_api["api_url"], api_user, api_password
         )
         self.resolution_info = self.rest_api.get_request(
-            request_info="service-data", safe=False, resolution=self.resolution_id
+            request_info="service-data", safe=True, resolution=self.resolution_id
         )
         self.service_folder = self.resolution_info["resolutions"][0][
             "resolution_full_number"
@@ -125,13 +125,13 @@ def copy_template(self):
         )
         services_ids = bu_isciii.utils.get_service_ids(self.services_requested)
         services_json = bu_isciii.service_json.ServiceJson()
-        if len(services_ids) == 1:
+        for service_id in services_ids:
             try:
-                service_template = services_json.get_find(services_ids[0], "template")
+                service_template = services_json.get_find(service_id, "template")
             except KeyError as e:
                 stderr.print(
                     "[red]ERROR: Service id %s not found in services json file."
-                    % services_ids[0]
+                    % service_id
                 )
                 stderr.print("traceback error %s" % e)
                 sys.exit()
@@ -153,13 +153,6 @@ def copy_template(self):
                 stderr.print("[red]ERROR: Copying template failed.")
                 stderr.print("traceback error %s" % e)
                 sys.exit()
-        else:
-            stderr.print(
-                "[red] ERROR: I'm not already prepared for handling more than one error at the same time, sorry!"
-                "Please re-run and select one of the service ids."
-            )
-            sys.exit(1)
-            return False
         return True
 
     def create_samples_id(self):
@@ -214,11 +207,9 @@ def create_symbolic_links(self):
                     )
                 except OSError as e:
                     stderr.print(
-                        "[red]ERROR: Symbolic links creation failed for sample %s."
-                        % sample["sampleName"]
+                        "[red]ERROR: Symbolic links creation failed for file %s." % file
                     )
                     stderr.print("Traceback: %s" % e)
-                    sys.exit()
 
     def samples_json(self):
         json_samples = json.dumps(self.service_samples, indent=4)
@@ -237,23 +228,30 @@ def create_new_service(self):
             self.create_samples_id()
             self.create_symbolic_links()
             self.samples_json()
-            self.rest_api.put_request(
-                "update-state", "resolution", self.resolution_id, "state", "in_progress"
-            )
-        else:
-            stderr.print(
-                "[yellow]WARN: No samples recorded in service: " + self.resolution_id
-            )
-            if bu_isciii.utils.prompt_yn_question("Do you want to proceed?: "):
-                self.create_folder()
-                self.copy_template()
+            if self.resolution_info["service_state"] != "in_progress":
                 self.rest_api.put_request(
                     "update-state",
                     "resolution",
                     self.resolution_id,
                     "state",
                     "in_progress",
                 )
+
+        else:
+            stderr.print(
+                "[yellow]WARN: No samples recorded in service: " + self.resolution_id
+            )
+            if bu_isciii.utils.prompt_yn_question("Do you want to proceed?: "):
+                self.create_folder()
+                self.copy_template()
+                if self.resolution_info["service_state"] != "in_progress":
+                    self.rest_api.put_request(
+                        "update-state",
+                        "resolution",
+                        self.resolution_id,
+                        "state",
+                        "in_progress",
+                    )
             else:
                 stderr.print("Directory not created. Bye!")
                 sys.exit(1)

bu_isciii/scratch.py

@@ -64,7 +64,7 @@ def __init__(
         self.conf = conf.get_configuration("scratch_copy")
 
         self.resolution_info = rest_api.get_request(
-            request_info="service-data", safe=False, resolution=self.resolution_id
+            request_info="service-data", safe=True, resolution=self.resolution_id
         )
         self.service_folder = self.resolution_info["resolutions"][0][
             "resolution_full_number"

bu_isciii/service_json.py

@@ -38,7 +38,6 @@ def get_service_configuration(self, service):
 
     def get_find(self, service, found):
         """
-        Owner: Pablo
         Description:
             Obtain from service any forward items from json data
         """

bu_isciii/templates/IRMA/ANALYSIS/lablog_irma

@@ -1,5 +1,4 @@
 #ls ../RAW/* | tr '\/' '\t' | cut -f3 | cut -d "_" -f 1 | sort -u | grep -v "md5" > samples_id.txt
 mkdir -p 00-reads
 mv ANALYSIS01_FLU_IRMA $(date '+%Y%m%d')_ANALYSIS01_FLU_IRMA
-mv ANALYSIS02_MET $(date '+%Y%m%d')_ANALYSIS02_MET
 cd 00-reads; cat ../samples_id.txt | xargs -I % echo "ln -s ../../RAW/%_*R1*.fastq.gz %_R1.fastq.gz" | bash; cat ../samples_id.txt | xargs -I % echo "ln -s ../../RAW/%_*R2*.fastq.gz %_R2.fastq.gz" | bash; cd -

bu_isciii/templates/IRMA/RESULTS/lablog_irma_results

@@ -3,7 +3,6 @@ cd $(date '+%Y%m%d')_entrega01
 
 #Create symbolic links depending on the analysis
 #Individual files
-ln -s ../../ANALYSIS/*_MET/99-stats/multiqc_report.html ./krona_results.html
 ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/all_samples_completo.txt .
 ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/A_H* .
 ln -s ../../ANALYSIS/*FLU_IRMA/04-irma/B .

bu_isciii/templates/IRMA/ANALYSIS/ANALYSIS02_MET/lablog → bu_isciii/templates/mag/ANALYSIS/ANALYSIS02_MAG/lablog

@@ -18,7 +18,7 @@ cat <<EOF > mag.sbatch
 
 export NXF_OPTS="-Xms500M -Xmx4G"
 
-nextflow run /scratch/bi/pipelines/nf-core-mag-2.1.1/workflow/main.nf \\
+nextflow run /data/bi/pipelines/nf-core-mag/nf-core-mag-2.1.1/workflow/main.nf \\
           -c ../../DOC/mag.config \\
           --input '00-reads/*_R{1,2}.fastq.gz' \\
           --outdir $(date '+%Y%m%d')_mag \\

bu_isciii/templates/mag/ANALYSIS/lablog_mag

@@ -0,0 +1 @@
+mv ANALYSIS02_MAG $(date '+%Y%m%d')_ANALYSIS02_MAG

bu_isciii/templates/mag/RAW/README

@@ -0,0 +1 @@
+##Folder to hold raw reads to analyze in the service

bu_isciii/templates/mag/REFERENCES/README

@@ -0,0 +1 @@
+##Folder to hold references of the service

bu_isciii/templates/mag/RESULTS/lablog_mag_results

@@ -0,0 +1,6 @@
+mkdir $(date '+%Y%m%d')_entrega01
+cd $(date '+%Y%m%d')_entrega01
+
+#Create symbolic links depending on the analysis
+#Individual files
+ln -s ../../ANALYSIS/*_MAG/99-stats/multiqc_report.html ./krona_results.html

bu_isciii/templates/mag/TMP/README

@@ -0,0 +1 @@
+##Folder to hold temporary files

bu_isciii/templates/services.json

@@ -222,7 +222,7 @@
     },
     "mag_met": {
       "label": "",
-      "template": "mag_met",
+      "template": "mag",
       "order": 2,
       "begin": "base",
       "end": "",

bu_isciii/templates/viralrecon/ANALYSIS/lablog_viralrecon

@@ -401,7 +401,6 @@ rm create_summary_report.sh
 rm deduplicate_long_table.sh
 rm percentajeNs.py
 rm _02_create_run_percentage_Ns.sh
-mv DATE_ANALYSIS0X_MAG $(date '+%Y%m%d')_ANALYSIS0X_MAG
 cd 00-reads; cat ../samples_id.txt | xargs -I % echo "ln -s ../../RAW/%_*R1*.fastq.gz %_R1.fastq.gz" | bash; cat ../samples_id.txt | xargs -I % echo "ln -s ../../RAW/%_*R2*.fastq.gz %_R2.fastq.gz" | bash; cd ..
 
 echo_green "\nLablog_viralrecon execution has been completed successfully!"

bu_isciii/templates/viralrecon/RESULTS/lablog_viralrecon_results

@@ -21,7 +21,6 @@ cat references.tmp | while read in; do cd excel_files_${in}; ln -s ../../../ANAL
 cat references.tmp | while read in; do ln -s ../../ANALYSIS/*/*${in}*/variants/ivar/variants_long_table.csv ${in}_variants_long_table.csv; done
 
 for ref in $(cut -f2 ../../ANALYSIS/samples_ref.txt | sort | uniq); do for organism in $(cut -f3 ../../ANALYSIS/samples_ref.txt | tr '[:lower:]' '[:upper:]' | sort | uniq); do report="../../ANALYSIS/*_${organism}/${ref}_*_viralrecon_mapping/multiqc/multiqc_report.html"; if [ -e $report ]; then ln -s ${report} ./multiqc_report_${ref}_${organism}.html; else echo "Multiqc report not found for reference $ref and organism $organism"; fi; done; done
-ln -s ../../ANALYSIS/*_MAG/99-stats/multiqc_report.html ./krona_results.html
 ln -s ../../ANALYSIS/*/mapping_illumina*.tab ./mapping_illumina.csv
 ln -s ../../ANALYSIS/*/assembly_stats.csv ./assembly_stats.csv
 ln -s ../../ANALYSIS/*/01-PikaVirus-results/all_samples_virus_table_filtered.tsv ./pikavirus_table.tsv