Code

NERSC SFAPI Reconstruction Flow#44

Code

Merged

dylanmcreynolds merged 25 commits intoals-computing:mainals-computing/splash_flows_globus:mainfrom davramov:nersc_sfapidavramov/splash_flows_globus:nersc_sfapiCopy head branch name to clipboardJan 29, 2025

+1,753-22

Commits

Commits on Nov 12, 2024

• WIP: Oct 3rd update for nersc recon with podman-hpc container

  rajasriramoju

  committed
  b4c26f5

  Copy full SHA for b4c26f5

Commits on Nov 13, 2024

• Adding Raja's nersc.py script

  davramov

  committed
  37632c7

  Copy full SHA for 37632c7

Commits on Dec 4, 2024

• able to get nersc reconstruction running in this configuration

  davramov

  committed
  b128b2c

  Copy full SHA for b128b2c

• Added an sfapi slurm call to run tiff_to_zarr on nersc

  davramov

  committed
  0463739

  Copy full SHA for 0463739

• Merge branch 'als-computing:main' into nersc_sfapi

  davramov

  authored
  91356bd

  Copy full SHA for 91356bd

Commits on Dec 6, 2024

• Migrating nersc tomography flow code to an abstract class structure (orchestration/flows/bl832/tomography_hpc.py). Linting orchestration/nersc.py

  davramov

  committed
  e1e7add

  Copy full SHA for e1e7add

Commits on Dec 7, 2024

• Addressing dylan's comments. Moved image names to config.yml and config.py, renamed tomography_hpc.py to job_controller.py. Added back nersc.py, which now just uses job_controller (need to add back…

  Show description for 8c5d20a
  davramov

  committed
  8c5d20a

  Copy full SHA for 8c5d20a

Commits on Dec 9, 2024

• Moved NERSC specific NERSCTomographyHPCController to flows/bl832/nersc.py. Updated get_controller() in job_controller.py to take in an HPC enum that stores the corresponding Controller implementati…

  Show description for 235acda
  davramov

  committed
  235acda

  Copy full SHA for 235acda

Commits on Dec 11, 2024

• Configured both reconstruction and multires steps to take in a file path and run respective tasks. Revert to debug queue (at least for now) since the queue time is orders of magnitude short than th…

  Show description for 9afefa3
  davramov

  committed
  9afefa3

  Copy full SHA for 9afefa3

Commits on Dec 12, 2024

• Resolved circular import issue, NERSC controller is not defined in flows/bl832/nersc.py. Removed the ALCF controlelr from job_controller, in anticipation of that being defined in alcf.py when I ref…

  Show description for 6304fe9
  davramov

  committed
  6304fe9

  Copy full SHA for 6304fe9

Commits on Dec 13, 2024

• Catch sfapi errors after submitting jobs, such that we are able to wait for the job to complete before moving onto the next step. An OK workaround for this: NERSC/sfapi_client#93

  davramov

  committed
  8e16fcc

  Copy full SHA for 8e16fcc

Commits on Dec 14, 2024

• Adding test_sfapi_flow.py pytest script.

  davramov

  committed
  c6601be

  Copy full SHA for c6601be

• Addressing prefect secret patch issue in pytest script

  davramov

  committed
  6057aae

  Copy full SHA for 6057aae

Commits on Dec 17, 2024

• Removing the NerscClient dependency, and using sfapi_client module directyly in NERSCTomographyHPCController

  davramov

  committed
  f9548a8

  Copy full SHA for f9548a8

• Adding deployment configuration for the nersc_recon_flow

  davramov

  committed
  3fe95b5

  Copy full SHA for 3fe95b5

Commits on Dec 19, 2024

• Added a comments in indicating that NerscClient is deprecated and will be removed. Removed init of COnfig832 and sfapi_client from NERSCTomographyHPCCOntroller, and instead inject them (no longer o…

  Show description for 6a4d2bf
  davramov

  committed
  6a4d2bf

  Copy full SHA for 6a4d2bf

Commits on Dec 20, 2024

• Updated NERSC job script to pull from ghcr.io image instead of Harbor with up to date python installations

  davramov

  committed
  223c3b4

  Copy full SHA for 223c3b4

Commits on Jan 14, 2025

• Updated config to include location of reconstruction scripts on NERSC. Updated nersc.py to use the realtime queue, which now works with minimal delay. Updated recon/multires jobs to use pscratch fo…

  Show description for 79f128c
  davramov

  committed
  79f128c

  Copy full SHA for 79f128c

Commits on Jan 15, 2025

• Update pruning code in nersc.py, although it still needs testing. Added nersc832_alsdev_pscratch endpoint in the config, and added a nersc_prune_pool and prefect deployment.

  davramov

  committed
  fae190f

  Copy full SHA for fae190f

Commits on Jan 17, 2025

• Added transfer_controller.py, and use the GlobusTransferController to move data within the nersc reconstruction flow.

  davramov

  committed
  1b2ae5c

  Copy full SHA for 1b2ae5c

Commits on Jan 21, 2025

• Included new controller class for data transfers and refactored the NERSC flow to use those. Updated the Prefect deployment script for the NERSC implementation. Updated endpoints in config.yml to i…

  Show description for e2963f0
  davramov

  committed
  e2963f0

  Copy full SHA for e2963f0

• Added placeholders for ALCFTomographyHPCController in alcf.py and OLCFTomographyHPCController in olcf.py, to be addressed in future git issues.

  davramov

  committed
  f30dd28

  Copy full SHA for f30dd28

Commits on Jan 28, 2025

• Added a pytest script for the new transfer_controller and globus/simple implementations. Replaced BaseEndpoint(Protocol) with a more generic TransferEndpoint(ABC). Updated FileSystemEndpoint to inc…

  Show description for fd62200
  davramov

  committed
  fd62200

  Copy full SHA for fd62200

• Updated unittests

  davramov

  committed
  f74a7a3

  Copy full SHA for f74a7a3

Commits on Jan 29, 2025

• removed @DataClass decorator because it is redundant with an __init__ block

  davramov

  committed
  087db2d

  Copy full SHA for 087db2d