Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

App Runner B-Fabric integration #126

Draft
wants to merge 209 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
209 commits
Select commit Hold shift + click to select a range
13ce417
initial code
leoschwarz Jan 16, 2025
a2cd886
rename
leoschwarz Jan 16, 2025
e4c6fac
add some implementation for the submitters_spec.py
leoschwarz Jan 16, 2025
6d2f997
create submitter module
leoschwarz Jan 16, 2025
30778ee
be a bit more pydantic
leoschwarz Jan 16, 2025
6a0328c
also ensure the user params get validated
leoschwarz Jan 16, 2025
86c407a
more explicit names
leoschwarz Jan 16, 2025
9057229
specify the slurm root
leoschwarz Jan 16, 2025
7bd6b38
implement the submitter
leoschwarz Jan 16, 2025
a10f3b0
use StringConstraints
leoschwarz Jan 16, 2025
b70c952
add input classes for the submitter
leoschwarz Jan 17, 2025
d536125
add some very early draft on the submitter design
leoschwarz Jan 20, 2025
8613af2
implement app_runner wrapper creator
leoschwarz Jan 20, 2025
f4bd898
integrate initial entrypoint
leoschwarz Jan 20, 2025
3347b08
initial executable definition
leoschwarz Jan 20, 2025
a8de5ef
directly set masterexecutableid
leoschwarz Jan 20, 2025
b5df77e
necessary flags
leoschwarz Jan 20, 2025
54e6865
update
leoschwarz Jan 20, 2025
7cfe16f
update
leoschwarz Jan 20, 2025
bd9500a
argparse
leoschwarz Jan 20, 2025
7a885b4
use the executable
leoschwarz Jan 20, 2025
75b4bcc
define default
leoschwarz Jan 20, 2025
21a4e4f
don't pass parameters
leoschwarz Jan 20, 2025
352e0cd
pass workunitid
leoschwarz Jan 20, 2025
25dfa2b
give it a name
leoschwarz Jan 20, 2025
d07767d
add a new line
leoschwarz Jan 20, 2025
029681d
add definition
leoschwarz Jan 20, 2025
5f47cd0
add skeleton submitter
leoschwarz Jan 20, 2025
206ae62
start implementing
leoschwarz Jan 20, 2025
286f5b1
already resolve the app version in wrapper creator
leoschwarz Jan 20, 2025
13e1d51
submitter
leoschwarz Jan 20, 2025
df9e9a4
adapt
leoschwarz Jan 20, 2025
a02731d
combine together
leoschwarz Jan 20, 2025
bbcc75f
debug
leoschwarz Jan 20, 2025
5145cf0
set status available when finished
leoschwarz Jan 20, 2025
f01556e
pass the yaml
leoschwarz Jan 20, 2025
ce25cd1
filter locally
leoschwarz Jan 20, 2025
d9d1d0b
fix
leoschwarz Jan 20, 2025
e64e8c1
fix
leoschwarz Jan 20, 2025
975b2d1
fix
leoschwarz Jan 20, 2025
9a4ec11
fix
leoschwarz Jan 20, 2025
4277f1b
implement more
leoschwarz Jan 20, 2025
192f8cc
adjust
leoschwarz Jan 20, 2025
1cfda5e
refactor
leoschwarz Jan 20, 2025
71a1275
fix
leoschwarz Jan 20, 2025
70aabcd
make the script executable
leoschwarz Jan 20, 2025
30129ab
consistent
leoschwarz Jan 20, 2025
c7d059e
app runner version
leoschwarz Jan 20, 2025
37b9804
log the result
leoschwarz Jan 20, 2025
4d6c5b2
drop
leoschwarz Jan 20, 2025
7c08be9
remove todo
leoschwarz Jan 20, 2025
3f27b89
update yamlfmt
leoschwarz Jan 21, 2025
6627def
format
leoschwarz Jan 21, 2025
3300607
submitter parameter parsing
leoschwarz Jan 21, 2025
a384d0e
submitter parameter parsing
leoschwarz Jan 21, 2025
eb778fe
add placeholder
leoschwarz Jan 21, 2025
5503f8d
add todo
leoschwarz Jan 21, 2025
502a413
store
leoschwarz Jan 21, 2025
1b5fa87
refactor
leoschwarz Jan 21, 2025
c8b00e5
working directory
leoschwarz Jan 21, 2025
3c384fc
allow interpolating workunit information
leoschwarz Jan 21, 2025
3e20359
remove todo
leoschwarz Jan 21, 2025
1bc3576
refactor config handling
leoschwarz Jan 21, 2025
23b7b42
adapt
leoschwarz Jan 21, 2025
0e8ec43
fix
leoschwarz Jan 21, 2025
f1bf893
log the variables
leoschwarz Jan 21, 2025
d23f7ea
fix
leoschwarz Jan 21, 2025
80821b3
fix?
leoschwarz Jan 21, 2025
2791647
fix
leoschwarz Jan 21, 2025
6889458
fix
leoschwarz Jan 21, 2025
6bc40cf
fix
leoschwarz Jan 21, 2025
7ac94c2
make app name safe
leoschwarz Jan 21, 2025
f8f4f8c
make app name safe
leoschwarz Jan 21, 2025
b1c7656
activate the command
leoschwarz Jan 21, 2025
5aa3887
remove todo
leoschwarz Jan 21, 2025
779bae0
remove todo
leoschwarz Jan 21, 2025
a09d16d
refactor
leoschwarz Jan 21, 2025
52c609f
use slurm arguments directly
leoschwarz Jan 21, 2025
551f325
migrate
leoschwarz Jan 21, 2025
406646d
remove the todo
leoschwarz Jan 21, 2025
372e5d5
fix
leoschwarz Jan 21, 2025
5607ad8
set status to done
leoschwarz Jan 21, 2025
a5646ac
try creating the workunit externaljob
leoschwarz Jan 21, 2025
9c69002
Revert "try creating the workunit externaljob"
leoschwarz Jan 21, 2025
a8008da
correctly use annotated
leoschwarz Jan 21, 2025
4a6b7c6
pass shell=True
leoschwarz Jan 21, 2025
1316ba8
Revert "pass shell=True"
leoschwarz Jan 21, 2025
e6184cb
better template
leoschwarz Jan 21, 2025
9fa5f37
fix rename
leoschwarz Jan 21, 2025
a9ae51c
print the config
leoschwarz Jan 21, 2025
436e862
fix a problematic typo
leoschwarz Jan 21, 2025
231f7ea
diagnostic info
leoschwarz Jan 21, 2025
d428b3a
force version field
leoschwarz Jan 22, 2025
3041f76
Merge branch 'main' into refactor-submitter
leoschwarz Feb 7, 2025
064bbe8
fix import
leoschwarz Feb 7, 2025
df9acd7
adapt tests
leoschwarz Feb 7, 2025
ba11224
adapt more imports
leoschwarz Feb 7, 2025
59c524c
Merge branch 'main' into refactor-submitter
leoschwarz Feb 10, 2025
1e3a3cf
Merge branch 'main' into refactor-submitter
leoschwarz Feb 10, 2025
59bb4ea
update app_runner command generation
leoschwarz Feb 10, 2025
9307621
clean the script a bit
leoschwarz Feb 10, 2025
1e5a317
remove todos
leoschwarz Feb 10, 2025
a0708b1
only set workunit to None when explicit
leoschwarz Feb 10, 2025
82648de
interpolate the submitter template
leoschwarz Feb 10, 2025
684807c
adapt test
leoschwarz Feb 10, 2025
a5e52a5
simplify
leoschwarz Feb 10, 2025
7f76b2b
correct type
leoschwarz Feb 10, 2025
b17e629
doc
leoschwarz Feb 10, 2025
56ece36
Merge branch 'main' into refactor-submitter
leoschwarz Feb 10, 2025
ce86200
fix imports
leoschwarz Feb 10, 2025
890d9a3
diagnostics
leoschwarz Feb 10, 2025
49c1466
fix
leoschwarz Feb 10, 2025
539eb71
log the file being parsed
leoschwarz Feb 10, 2025
e8a60eb
add a field for configuring log viewer url
leoschwarz Feb 10, 2025
c9e75f4
correct dependency
leoschwarz Feb 10, 2025
5e31452
fix
leoschwarz Feb 10, 2025
e97941e
ensure python
leoschwarz Feb 10, 2025
c70469b
Merge branch 'main' into refactor-submitter
leoschwarz Feb 10, 2025
0fe595f
integrate force_storage option
leoschwarz Feb 10, 2025
b620a5c
integrate force_storage further
leoschwarz Feb 10, 2025
17de037
try add timestamps
leoschwarz Feb 10, 2025
1f88621
try again
leoschwarz Feb 10, 2025
6811f9f
fail on error
leoschwarz Feb 10, 2025
54065c8
debugging
leoschwarz Feb 10, 2025
97fdfce
fix
leoschwarz Feb 10, 2025
9a89b3e
forward errors
leoschwarz Feb 10, 2025
6917c04
remove
leoschwarz Feb 10, 2025
7cb30f0
try again
leoschwarz Feb 10, 2025
77008ae
multiline
leoschwarz Feb 10, 2025
8f4fe50
try again
leoschwarz Feb 10, 2025
3ed689c
Merge branch 'main' into refactor-submitter
leoschwarz Feb 10, 2025
a49b648
syntax
leoschwarz Feb 10, 2025
ccdf367
escaping
leoschwarz Feb 10, 2025
77d6156
pretty
leoschwarz Feb 10, 2025
0922623
add the link to logs if appropriate
leoschwarz Feb 10, 2025
121fa49
actually call add_links
leoschwarz Feb 10, 2025
127ed68
Merge branch 'main' into refactor-submitter
leoschwarz Feb 11, 2025
8b72e8c
add links
leoschwarz Feb 11, 2025
f1b38c3
diagnose
leoschwarz Feb 11, 2025
cb579c3
correct type
leoschwarz Feb 11, 2025
05988c6
Merge branch 'main' into refactor-submitter
leoschwarz Feb 11, 2025
9f51a07
consistent
leoschwarz Feb 11, 2025
c27ef69
default
leoschwarz Feb 11, 2025
6a99c71
add tests
leoschwarz Feb 11, 2025
c1efe32
implement support for time limit
leoschwarz Feb 12, 2025
d4f933f
missing conversion
leoschwarz Feb 12, 2025
f994ddc
python compat
leoschwarz Feb 12, 2025
57cf788
Merge branch 'main' into refactor-submitter
leoschwarz Feb 13, 2025
d77d532
ensure dir exists
leoschwarz Feb 13, 2025
c2094ee
Merge branch 'main' into refactor-submitter
leoschwarz Feb 13, 2025
c73ac52
use importlib
leoschwarz Feb 13, 2025
1db21db
use importlib
leoschwarz Feb 13, 2025
48b68a8
Merge branch 'main' into refactor-submitter
leoschwarz Feb 13, 2025
8bf67af
correctly specify resource files
leoschwarz Feb 13, 2025
1be65cd
correctly specify resource files
leoschwarz Feb 13, 2025
99bf79a
Merge branch 'main' into refactor-submitter
leoschwarz Feb 13, 2025
31055ca
read the correct file
leoschwarz Feb 13, 2025
8e27f27
read the correct file
leoschwarz Feb 13, 2025
2647b90
Merge branch 'main' into refactor-submitter
leoschwarz Feb 13, 2025
ec03eb6
Merge branch 'main' into refactor-submitter
leoschwarz Feb 13, 2025
f1fac37
Merge branch 'main' into refactor-submitter
leoschwarz Feb 13, 2025
0670d95
use the `app_version.yml` file now
leoschwarz Feb 13, 2025
a82a0d5
Merge branch 'main' into refactor-submitter
leoschwarz Feb 13, 2025
1d09b47
add file_token
leoschwarz Feb 17, 2025
af20f65
Merge branch 'main' into refactor-submitter
leoschwarz Feb 17, 2025
93595df
Merge branch 'main' into refactor-submitter (manual fixes)
leoschwarz Feb 17, 2025
24cd97b
Merge branch 'main' into refactor-submitter
leoschwarz Feb 18, 2025
53450d7
fix problem
leoschwarz Feb 18, 2025
b42bbd4
Merge branch 'main' into refactor-submitter
leoschwarz Feb 18, 2025
2c20a6f
force python 3.13
leoschwarz Feb 18, 2025
ccc2275
Merge branch 'main' into refactor-submitter
leoschwarz Feb 19, 2025
81c62e0
Merge branch 'main' into refactor-submitter
leoschwarz Feb 19, 2025
aa528b2
Merge branch 'main' into refactor-submitter
leoschwarz Feb 24, 2025
eb3835a
Merge branch 'main' into refactor-submitter
leoschwarz Feb 28, 2025
01ab962
create resource instead of link
leoschwarz Feb 28, 2025
ee9bec8
Merge branch 'main' into refactor-submitter
leoschwarz Feb 28, 2025
8974a9b
missing workunit_id
leoschwarz Feb 28, 2025
154175a
improve state logic
leoschwarz Feb 28, 2025
dad2d02
Handle problematic characters in Workunit.store_output_folder
leoschwarz Feb 28, 2025
531b198
try again
leoschwarz Feb 28, 2025
ecd6e36
Merge branch 'main' into refactor-submitter
leoschwarz Feb 28, 2025
779caf3
it's still good enough to use half of it
leoschwarz Feb 28, 2025
0e16fa8
fix
leoschwarz Feb 28, 2025
d8f71de
start preparing for new impl
leoschwarz Feb 28, 2025
9910f58
setup slurm_submitter package
leoschwarz Feb 28, 2025
1841dfe
better template
leoschwarz Feb 28, 2025
5e2c2b1
basically merge the two components
leoschwarz Feb 28, 2025
8720fc7
deactivate
leoschwarz Feb 28, 2025
c514bff
fix imports
leoschwarz Feb 28, 2025
551f5f8
clean
leoschwarz Feb 28, 2025
6b75950
cleaning
leoschwarz Feb 28, 2025
78de0d2
integrate
leoschwarz Feb 28, 2025
cf49f2f
wrong arg
leoschwarz Feb 28, 2025
466bc95
fix
leoschwarz Feb 28, 2025
a31e95e
fix
leoschwarz Feb 28, 2025
74ec001
path
leoschwarz Feb 28, 2025
b484541
path
leoschwarz Feb 28, 2025
b0a550d
typo
leoschwarz Feb 28, 2025
0e70db2
fix param
leoschwarz Feb 28, 2025
38be212
move
leoschwarz Feb 28, 2025
8dd0ef3
refactor
leoschwarz Feb 28, 2025
d14585b
configurable name
leoschwarz Feb 28, 2025
ff5d40a
fix
leoschwarz Mar 4, 2025
1a5d276
Merge branch 'main' into refactor-submitter
leoschwarz Mar 5, 2025
f06c583
Merge branch 'main' into refactor-submitter
leoschwarz Mar 7, 2025
ec2b85b
Merge branch 'main' into refactor-submitter
leoschwarz Mar 7, 2025
64d133f
deactivate python output buffering to ensure correct log alignment of…
leoschwarz Mar 7, 2025
330e43f
handle fgcz_slurm_app_runner_compat.bash
leoschwarz Mar 7, 2025
516479d
return as Path
leoschwarz Mar 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
already resolve the app version in wrapper creator
  • Loading branch information
leoschwarz committed Jan 28, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 286f5b1ee9c5144201df7fe13757415bb3e2956d
Original file line number Diff line number Diff line change
@@ -2,12 +2,10 @@

from pydantic import BaseModel

from app_runner.specs.app.app_spec import AppSpecTemplate # noqa: TC001
from app_runner.specs.app.app_version import AppVersion # noqa: TC001
from bfabric.experimental.workunit_definition import WorkunitDefinition # noqa: TC002


class WorkunitWrapperData(BaseModel):
# TODO this could be revisited in the future, e.g. whether we want to merge this part into the app_definition
# TODO should the template be substituted already?
workunit: WorkunitDefinition
app: AppSpecTemplate
workunit_definition: WorkunitDefinition
app_version: AppVersion
8 changes: 6 additions & 2 deletions app_runner/src/app_runner/bfabric_app/wrapper_creator.py
Original file line number Diff line number Diff line change
@@ -8,6 +8,7 @@
import yaml
from loguru import logger

from app_runner.app_runner.resolve_app import resolve_app
from app_runner.bfabric_app.workunit_wrapper_data import WorkunitWrapperData
from app_runner.specs.app.app_spec import AppSpecTemplate
from bfabric import Bfabric # noqa: TC002
@@ -43,8 +44,11 @@ def get_data(self) -> WorkunitWrapperData:
workunit_definition = WorkunitDefinition.from_workunit(workunit=self._workunit)
# TODO could this be made more robust in the future, e.g. by specifying the root path somehow?
path = Path(self._workunit.application.executable["program"])
application_definition = AppSpecTemplate.model_validate(yaml.safe_load(path.read_text()))
return WorkunitWrapperData(workunit=workunit_definition, app=application_definition)
app_spec_template = AppSpecTemplate.model_validate(yaml.safe_load(path.read_text()))
app = self._workunit.application
app_spec = app_spec_template.evaluate(app_id=str(app.id), app_name=app["name"])
app_version = resolve_app(versions=app_spec, workunit_definition=workunit_definition)
return WorkunitWrapperData(workunit_definition=workunit_definition, app_version=app_version)

def run(self) -> None:
data = self.get_data()