Skip to content

Commit

Permalink
adding support for alias files (#557)
Browse files Browse the repository at this point in the history
* adding support for alias files
an alias file is a yaml file "lookup" alongside a container.yaml
that can be used to provide tag specific commands for a particular
container tag
* updating alias-files -> overrides

Signed-off-by: vsoch <[email protected]>
Co-authored-by: Matthieu Muffato <[email protected]>
  • Loading branch information
vsoch and muffato authored Jul 8, 2022
1 parent b1de8ee commit fc59d9f
Show file tree
Hide file tree
Showing 28 changed files with 2,911 additions and 12 deletions.
7 changes: 5 additions & 2 deletions .github/workflows/test-container.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,15 @@ jobs:
run: |
# Continue if we have a changed recipe file
if [[ "${{ matrix.changed_recipe }}" = registry* ]]; then
echo "keepgoing=true" >> $GITHUB_ENV
recipe_basename=$(basename "${{ matrix.changed_recipe }}")
if [[ "${recipe_basename}" == "container.yaml" ]]; then
echo "keepgoing=true" >> $GITHUB_ENV
fi
fi
- name: Checkout
if: ${{ env.keepgoing == 'true' }}
uses: actions/checkout@v2
uses: actions/checkout@v3

- name: Make Space For Build
if: ${{ env.keepgoing == 'true' }}
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ and **Merged pull requests**. Critical items to know are:
The versions coincide with releases on pip. Only major versions will be released as tags on Github.

## [0.0.x](https://github.com/singularityhub/singularity-hpc/tree/main) (0.0.x)
- adding support for container.yaml overrides (0.0.56)
- `SINGULARITY_CONTAINER` and `PODMAN_CONTAINER` support (0.0.55)
- support for views create, delete, edit, install, uninstall (0.0.54)
- also including cleanup of module in views on uninstall
Expand Down
47 changes: 47 additions & 0 deletions docs/getting_started/developer-guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,50 @@ For each of the above, depending on the prefix of options that you choose, it wi
This means that if you design a new registry recipe, you should consider how to run it for both kinds of technology. Also note that ``docker_options`` are
those that will also be used for Podman.

Overrides
---------

It might be the case that as your containers change over time, the set of any of:

- commands (aliases)
- docker_script
- singularity_script
- environment (env)
- features
- description

does too! Or it be the case that you have hundreds of aliases, and want to better organize them separately from the container.yaml. To support this, shpc
(as of version 0.0.56) has support for an ``overrides`` section in the container.yaml, meaning that you can define pairs of container
tags and relative path lookups to external files with any of the stated sections. A simple example might look like this:

.. code-block:: yaml
docker: python
url: https://hub.docker.com/_/python
maintainer: '@vsoch'
description: An interpreted, high-level and general-purpose programming language.
latest:
3.9.5-alpine: sha256:f189f7366b0d381bf6186b2a2c3d37f143c587e0da2e8dcc21a732bddf4e6f7b
tags:
3.9.2-alpine: sha256:f046c06388c0721961fe5c9b6184d2f8aeb7eb01b39601babab06cfd975dae01
overrides:
3.9.2-alpine: aliases/3.9.2-alpine.yaml
aliases:
python: /usr/local/bin/python
Since this file only has aliases, we chose to use a subdirectory called "aliases" to make that clear, however
the file can have any of the fields mentioned above, and can be organized in any relative path to the container directory that you deem apppropriate.
Here is what this corresponding file with relative path ``aliases/3.9.2-alpine.yaml`` might look like this:

.. code-block:: yaml
aliases:
python: /alias/path/to/python
Finally, for all fields mentioned above, the format is expected to follow the same convention as above (and it will be validated again on update).


Wrapper Script
--------------

Expand Down Expand Up @@ -514,6 +558,9 @@ Fields include:
* - aliases
- Named entrypoints for container (dict) as described above
- false
* - overrides
- Key value pairs to override container.yaml defaults.
- false
* - url
- Documentation or other url for the container uri
- false
Expand Down
247 changes: 247 additions & 0 deletions registry/quay.io/pawsey/openfoam-org/aliases/2.2.0.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,247 @@
aliases:
Co: Co
LTSInterFoam: LTSInterFoam
LTSReactingParcelFoam: LTSReactingParcelFoam
Lambda2: Lambda2
MRFInterFoam: MRFInterFoam
MRFMultiphaseInterFoam: MRFMultiphaseInterFoam
Mach: Mach
PDRFoam: PDRFoam
PDRMesh: PDRMesh
Pe: Pe
Q: Q
R: R
SRFPimpleFoam: SRFPimpleFoam
SRFSimpleFoam: SRFSimpleFoam
XiFoam: XiFoam
adiabaticFlameT: adiabaticFlameT
adjointShapeOptimizationFoam: adjointShapeOptimizationFoam
ansysToFoam: ansysToFoam
applyBoundaryLayer: applyBoundaryLayer
applyWallFunctionBoundaryConditions: applyWallFunctionBoundaryConditions
attachMesh: attachMesh
autoPatch: autoPatch
autoRefineMesh: autoRefineMesh
blockMesh: blockMesh
boundaryFoam: boundaryFoam
boxTurb: boxTurb
buoyantBoussinesqPimpleFoam: buoyantBoussinesqPimpleFoam
buoyantBoussinesqSimpleFoam: buoyantBoussinesqSimpleFoam
buoyantPimpleFoam: buoyantPimpleFoam
buoyantSimpleFoam: buoyantSimpleFoam
cavitatingDyMFoam: cavitatingDyMFoam
cavitatingFoam: cavitatingFoam
cfx4ToFoam: cfx4ToFoam
changeDictionary: changeDictionary
checkMesh: checkMesh
chemFoam: chemFoam
chemkinToFoam: chemkinToFoam
chtMultiRegionFoam: chtMultiRegionFoam
chtMultiRegionSimpleFoam: chtMultiRegionSimpleFoam
coalChemistryFoam: coalChemistryFoam
coldEngineFoam: coldEngineFoam
collapseEdges: collapseEdges
combinePatchFaces: combinePatchFaces
compressibleInterDyMFoam: compressibleInterDyMFoam
compressibleInterFoam: compressibleInterFoam
compressibleTwoPhaseEulerFoam: compressibleTwoPhaseEulerFoam
createBaffles: createBaffles
createPatch: createPatch
createTurbulenceFields: createTurbulenceFields
datToFoam: datToFoam
decomposePar: decomposePar
deformedGeom: deformedGeom
dnsFoam: dnsFoam
dsmcFieldsCalc: dsmcFieldsCalc
dsmcFoam: dsmcFoam
dsmcInitialise: dsmcInitialise
electrostaticFoam: electrostaticFoam
engineCompRatio: engineCompRatio
engineFoam: engineFoam
engineSwirl: engineSwirl
enstrophy: enstrophy
equilibriumCO: equilibriumCO
equilibriumFlameT: equilibriumFlameT
execFlowFunctionObjects: execFlowFunctionObjects
expandDictionary: expandDictionary
extrude2DMesh: extrude2DMesh
extrudeMesh: extrudeMesh
extrudeToRegionMesh: extrudeToRegionMesh
faceAgglomerate: faceAgglomerate
financialFoam: financialFoam
fireFoam: fireFoam
flattenMesh: flattenMesh
flowType: flowType
fluent3DMeshToFoam: fluent3DMeshToFoam
fluentMeshToFoam: fluentMeshToFoam
foamCalc: foamCalc
foamDataToFluent: foamDataToFluent
foamDebugSwitches: foamDebugSwitches
foamFormatConvert: foamFormatConvert
foamHelp: foamHelp
foamInfoExec: foamInfoExec
foamListTimes: foamListTimes
foamMeshToFluent: foamMeshToFluent
foamToEnsight: foamToEnsight
foamToEnsightParts: foamToEnsightParts
foamToGMV: foamToGMV
foamToStarMesh: foamToStarMesh
foamToSurface: foamToSurface
foamToTetDualMesh: foamToTetDualMesh
foamToVTK: foamToVTK
foamUpgradeCyclics: foamUpgradeCyclics
foamUpgradeFvSolution: foamUpgradeFvSolution
gambitToFoam: gambitToFoam
gmshToFoam: gmshToFoam
icoFoam: icoFoam
icoUncoupledKinematicParcelDyMFoam: icoUncoupledKinematicParcelDyMFoam
icoUncoupledKinematicParcelFoam: icoUncoupledKinematicParcelFoam
ideasUnvToFoam: ideasUnvToFoam
insideCells: insideCells
interDyMFoam: interDyMFoam
interFoam: interFoam
interMixingFoam: interMixingFoam
interPhaseChangeFoam: interPhaseChangeFoam
kivaToFoam: kivaToFoam
laplacianFoam: laplacianFoam
magneticFoam: magneticFoam
mapFields: mapFields
mdEquilibrationFoam: mdEquilibrationFoam
mdFoam: mdFoam
mdInitialise: mdInitialise
mergeMeshes: mergeMeshes
mergeOrSplitBaffles: mergeOrSplitBaffles
mhdFoam: mhdFoam
mirrorMesh: mirrorMesh
mixtureAdiabaticFlameT: mixtureAdiabaticFlameT
modifyMesh: modifyMesh
moveDynamicMesh: moveDynamicMesh
moveEngineMesh: moveEngineMesh
moveMesh: moveMesh
mshToFoam: mshToFoam
multiphaseEulerFoam: multiphaseEulerFoam
multiphaseInterFoam: multiphaseInterFoam
netgenNeutralToFoam: netgenNeutralToFoam
nonNewtonianIcoFoam: nonNewtonianIcoFoam
objToVTK: objToVTK
pPrime2: pPrime2
particleTracks: particleTracks
patchAverage: patchAverage
patchIntegrate: patchIntegrate
patchSummary: patchSummary
pdfPlot: pdfPlot
pimpleDyMFoam: pimpleDyMFoam
pimpleFoam: pimpleFoam
pisoFoam: pisoFoam
plot3dToFoam: plot3dToFoam
polyDualMesh: polyDualMesh
porousInterFoam: porousInterFoam
porousSimpleFoam: porousSimpleFoam
postChannel: postChannel
potentialFoam: potentialFoam
potentialFreeSurfaceFoam: potentialFreeSurfaceFoam
probeLocations: probeLocations
ptot: ptot
reactingFoam: reactingFoam
reactingParcelFilmFoam: reactingParcelFilmFoam
reactingParcelFoam: reactingParcelFoam
reconstructPar: reconstructPar
reconstructParMesh: reconstructParMesh
redistributePar: redistributePar
refineHexMesh: refineHexMesh
refineMesh: refineMesh
refineWallLayer: refineWallLayer
refinementLevel: refinementLevel
removeFaces: removeFaces
renumberMesh: renumberMesh
rhoCentralDyMFoam: rhoCentralDyMFoam
rhoCentralFoam: rhoCentralFoam
rhoLTSPimpleFoam: rhoLTSPimpleFoam
rhoPimpleFoam: rhoPimpleFoam
rhoPimplecFoam: rhoPimplecFoam
rhoPorousSimpleFoam: rhoPorousSimpleFoam
rhoReactingBuoyantFoam: rhoReactingBuoyantFoam
rhoReactingFoam: rhoReactingFoam
rhoSimpleFoam: rhoSimpleFoam
rhoSimplecFoam: rhoSimplecFoam
rotateMesh: rotateMesh
sammToFoam: sammToFoam
sample: sample
scalarTransportFoam: scalarTransportFoam
selectCells: selectCells
setFields: setFields
setSet: setSet
setsToZones: setsToZones
settlingFoam: settlingFoam
shallowWaterFoam: shallowWaterFoam
simpleFoam: simpleFoam
simpleReactingParcelFoam: simpleReactingParcelFoam
singleCellMesh: singleCellMesh
smapToFoam: smapToFoam
snappyHexMesh: snappyHexMesh
solidDisplacementFoam: solidDisplacementFoam
solidEquilibriumDisplacementFoam: solidEquilibriumDisplacementFoam
sonicDyMFoam: sonicDyMFoam
sonicFoam: sonicFoam
sonicLiquidFoam: sonicLiquidFoam
splitCells: splitCells
splitMesh: splitMesh
splitMeshRegions: splitMeshRegions
sprayEngineFoam: sprayEngineFoam
sprayFoam: sprayFoam
star3ToFoam: star3ToFoam
star4ToFoam: star4ToFoam
steadyParticleTracks: steadyParticleTracks
stitchMesh: stitchMesh
streamFunction: streamFunction
stressComponents: stressComponents
subsetMesh: subsetMesh
surfaceAdd: surfaceAdd
surfaceAutoPatch: surfaceAutoPatch
surfaceBooleanFeatures: surfaceBooleanFeatures
surfaceCheck: surfaceCheck
surfaceClean: surfaceClean
surfaceCoarsen: surfaceCoarsen
surfaceConvert: surfaceConvert
surfaceFeatureConvert: surfaceFeatureConvert
surfaceFeatureExtract: surfaceFeatureExtract
surfaceFind: surfaceFind
surfaceInertia: surfaceInertia
surfaceMeshConvert: surfaceMeshConvert
surfaceMeshConvertTesting: surfaceMeshConvertTesting
surfaceMeshExport: surfaceMeshExport
surfaceMeshImport: surfaceMeshImport
surfaceMeshInfo: surfaceMeshInfo
surfaceMeshTriangulate: surfaceMeshTriangulate
surfaceOrient: surfaceOrient
surfacePointMerge: surfacePointMerge
surfaceRedistributePar: surfaceRedistributePar
surfaceRefineRedGreen: surfaceRefineRedGreen
surfaceSmooth: surfaceSmooth
surfaceSplitByPatch: surfaceSplitByPatch
surfaceSplitByTopology: surfaceSplitByTopology
surfaceSplitNonManifolds: surfaceSplitNonManifolds
surfaceSubset: surfaceSubset
surfaceToPatch: surfaceToPatch
surfaceTransformPoints: surfaceTransformPoints
temporalInterpolate: temporalInterpolate
tetgenToFoam: tetgenToFoam
topoSet: topoSet
transformPoints: transformPoints
twoLiquidMixingFoam: twoLiquidMixingFoam
twoPhaseEulerFoam: twoPhaseEulerFoam
uncoupledKinematicParcelFoam: uncoupledKinematicParcelFoam
uprime: uprime
viewFactorsGen: viewFactorsGen
vorticity: vorticity
vtkUnstructuredToFoam: vtkUnstructuredToFoam
wallFunctionTable: wallFunctionTable
wallGradU: wallGradU
wallHeatFlux: wallHeatFlux
wallShearStress: wallShearStress
wdot: wdot
writeCellCentres: writeCellCentres
writeMeshObj: writeMeshObj
yPlusLES: yPlusLES
yPlusRAS: yPlusRAS
zipUpMesh: zipUpMesh
Loading

0 comments on commit fc59d9f

Please sign in to comment.