DM-46307: execute butler housekeeping scripts at remote DF for multisite processing #186
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
✅ All tests successful. No failed tests found. Additional details and impacted files@@ Coverage Diff @@
## main #186 +/- ##
==========================================
- Coverage 82.58% 82.57% -0.02%
==========================================
Files 41 44 +3
Lines 3492 3672 +180
Branches 355 359 +4
==========================================
+ Hits 2884 3032 +148
- Misses 518 549 +31
- Partials 90 91 +1 ☔ View full report in Codecov by Sentry. |
mxk62
force-pushed
the
tickets/DM-46307
branch
from
cbcc4c7 to
9ad298d
Compare
The code responsible for creating the submit directory was rather generic. I factored it out from the _init_submission_driver() and added it as a general purpose function to bps_utils.
Moved the "business logic" out of the _init_submission_driver() to a separate function to make the driver its structure more similar to other drivers.
Validating a BPS configuration for a regular workflow and for a workflow executing a script remotely requires different sets of validation tests. However, the validation tests for BPS config performed in init_submission() were "hard-coded" and there was no mechanism allowing for customizing them. Modified this function so the validation tests it performs could be explicitly specified when called.
Added a new command that will allow the user to submit custom scripts for execution.
Some unit tests for _init_submission_driver() stopped working after refactoring. Made some modifications to make them work again.
Added a new workflow attribute, bps_iscustom, so any BPS plugin can easily distinguish between regular payload workflows and custom ones intended to run ad hoc scripts.
Added a validator that will ensure 'customJob" is defined in BPS config when submitting a custom jobs.
mxk62
force-pushed
the
tickets/DM-46307
branch
2 times, most recently
from
ced56c0 to
03b03c7
Compare
MichelleGower
approved these changes
Nov 28, 2024
Based on the reviewer input, I fixed some spelling errors in the documentation and included additional pieces of information that the user might be interested in.
Some new docstrings had typos and some shortcomings (e.g. missing return value description). Other were missing entirely. Fixed all of those.
While the custom job validator was checking if the required section in BPS config exists, it wasn't checking if the required entry, in this section, i.e., ``executable``, is present. Made sure it does it as well.
Made some minor edits to the log messages displayed during BPS submission to make them either more specific or stylistically consistent.
The same snippet responsible for logging memory usage was repeated multiple times in ``drivers.py`` module. Factord it out as an auxiliary function to avoid repetitions.
mxk62
force-pushed
the
tickets/DM-46307
branch
from
03b03c7 to
7fb8495
Compare
Added an option the will allow the user to specify compute site when calling ``bps submitcmd``.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
doc/changes