feat(ci): build and publish docker image

[JaeAeich]

CI to build and publish docker image to docker hub.

[JaeAeich]

@uniqueg Kanitz I propose releases to be based on git tags and git tags alone, this improves trackability of releases but raise a question of versioning. Different "artifact" have different way of versioning schema, for example dockerhub might have -slim, -ubuntu etc tags attached to version number (say x.y.z) ie x.y.z-slim while on the other hand pypi might have x.y.z format only.

Initially I thought of validating all the git tags with the version mentioned in pyproject.toml which will gel well with pypi but I not sure about dockerhub and down the line with artifacthub ie version diverging from the "single source of truth" (pyporject). Any thoughts on it or how you picture the release cycle to look like.

This is currently for https://github.com/elixir-cloud-aai/cookiecutter-python/issues, given I mostly see python libraries (pypi only) and API (pypi, dockerhub, artifacthub etc) we have some constraint ie its not very general and for the limited scope of the problem we might be better off with git tag based release only.

So should I validate the git tag with pyproject and then create a release with that name to pypi and dockerhub.

Yes it will add an overhead of doing poetry version major|minor|path before git tag .

We can always give another way to create release using workflow_dispatch or released in gh actions, but again the more leeway we give the more are the chances of version divergence.

[uniqueg]

Different "artifact" have different way of versioning schema, for example dockerhub might have -slim, -ubuntu etc tags attached to version number (say x.y.z) ie x.y.z-slim while on the other hand pypi might have x.y.z format only.

I don't see a problem here. The different tags of the format you describe generally are for different versions of the underlying operating system, which is obviously not a concern for PyPI releases. The same version number x.y.z across both PyPI and (potentially) multiple Docker containers would still be expected to refer to the same version of the packaged software.

Initially I thought of validating all the git tags with the version mentioned in pyproject.toml which will gel well with pypi [...]

What do you mean? Please be more precise. What are "all the git tags" and how (and why) do you want to "validate" them?

[...] but I not sure about dockerhub and down the line with artifacthub ie version diverging from the "single source of truth" (pyporject). Any thoughts on it or how you picture the release cycle to look like.

Again, I fail to see the problem. For a given release, automated or not, wouldn't you just want to bump the version in pyproject.toml (according to SemVer rules) and then use that increased version for creating your different releases (GitHub, on PyPI, on the container registry, maybe other places like Artifact Hub), based on a specific tagged commit? If so, pyproject.toml remains the single source of truth, no? So what exactly is the problem that you are foreseeing?

This is currently for https://github.com/elixir-cloud-aai/cookiecutter-python/issues, given I mostly see python libraries (pypi only) and API (pypi, dockerhub, artifacthub etc) we have some constraint ie its not very general and for the limited scope of the problem we might be better off with git tag based release only.

This is really confusing. Are you referring to a specific issue? Which one? What is it that you "mostly see", and where? What exactly are the constraints we have? And compared to what other approach do you think we might be better off with a "git tag based release only"? I feel that I really shouldn't have such a hard time deciphering what you mean - please try to write more clearly and precisely 🙏

But with the help of AI (!), I will give it a shot: Is your concern that it is hard to implement a one size fits all release process for all possible use cases and therefore we should focus on something that works for our use cases? If so, I agree 100%. This is not a project to market and distribute and convince other people to use it. It's for us and our use cases, period. And for that I think we can come up with something reasonable that is at least a good starting point. We can still always update the template or fine tune according to particular constraints/requirements in each project implementing the template.

So should I validate the git tag with pyproject and then create a release with that name to pypi and dockerhub.

I still don't know what "validate the git tag with pyproject" means - but yes, please go ahead with implementing releases for GitHub, PyPI and Docker Hub (that's enough for now!), based on the version specified in pyproject.toml and a corresponding Git tag as a trigger. But perhaps you want to check out python-semantic-release first, because it uses the same strategy (takes Python version from some file, e.g., pyproject.toml, then creates various releases based on that) and so it's possible that it will save you some time to have it take care of these things (compared to trying to do all of these manually first and then changing to fully automated releases later).

Yes it will add an overhead of doing poetry version major|minor|path before git tag .

Do you mean you need to run a command poetry version major|minor|patch? What does it do and why is it needed?

We can always give another way to create release using workflow_dispatch or released in gh actions, but again the more leeway we give the more are the chances of version divergence.

As said before, we are not trying to build a "God template". It's just a strongly opinionated stack of tools and boilerplate config that is supposed to help us quickly get to where we want to go, given that we are using a pretty consistent stack of tools for building Web API-based microservices. Nothing more, nothing less. If people (and that includes ourselves) are not happy with the defaults for a given project, it will be easy to change things up according to our needs. And if we find that we consistently need to do the same changes in our releases, we know that we set the wrong defaults in the template and update it accordingly.

That being said, I'm reiterating my conviction that having different builds for the same major/minor/patch version and things of that nature do not qualify as "version divergence".

[JaeAeich]

Yeah understandable, and sorry for being a little lazy/bad with the context. But yeah we don't wanna make a "God template", I don't know how to (yet!). But the question was mostly how do we see ourself releasing the artifacts of the projects, mostly from standardization's perspective, While going though the explanation for the abysmal sonnet I wrote above, I found realized I was stupidly missing the simple fact which makes this conversion a little silly and solution down right obvious.

Lets first start with validation, ie the dev (say Bob with privilege to push tags to main), pushes a wrong tag by mistake, we need to validate this tag first from single source of truth so that releases can be systematically traced back to changes in code, what I mean is if the python code has 1.1.4 as version, it would be a little stupid to publish its helm chart with a version of 2.1.3. Hence the "validation", we would want the release number of different artifact to be linked to the single source of truth (in our case pyporject.toml).

About divergence, for some reason I thought that all the releases have to happen from one tag, I could put my fingers on the fact that we can have multiple tags for different artifact release, ie tf-1.1.3 for terraform, py-1.1.3 for pypi release, img-1.1.3 for docker release. With this we can run actions separately.

So on main only the pypi releases (with vx.y.z) are triggered, while on another locked branch (say develop) all other release published like testpypi with pypi-v1.0.0 or image/docker release with docker-v1.0.0 so on and so forth.

Or we can have main branch based model as well where all the tags are on main and releases are triggered from there.

Note: The separation of concerns with regards to branch name is important to be discussed before hand because firstly they will a pain to change if the template changes and secondly these are hardcoded in GH actions.

With that we would be able to write "validation" and git tag "sanitization" for each "artifact" and add them to their metadata and publish.

So for me the obvious missing piece was that for some reason I thought all the releases should come from the same tag as I mentioned, which is rather difficult. But given the discussion now, it would be good to discuss that how we would like our release to be on.

Do we go with main branch model, ie only python releases will happen on main, this will make it so that main branch only has python releases ie less cluttered main and then develop branch has all other tags. Or we go with the all tags on main branch model.

[JaeAeich]

I think this is a broader discussion on releases so I have created a discussion on the same.

[uniqueg]

I still don't quite understand this 🙈🤣

Two questions/points:

1. Why do you want to create the tag manually? As a trigger to start the release? But that would indeed open up the possibility of having two sources of truth - the version info from pyproject.toml and the tag. Why don't we use another trigger and stick with pyproject.toml - which is also the single source of truth that python-semantic-release will consider. The Git tag and release can than be automated from that as well.
2. I think a particular version of a tool/project should always refer to the same state of the codebase. In terms of SemVer, that means that a version string of the form x.y.z should always mean the same thing, in terms of the project that is versioned. And in terms of version control with Git, what defines a particular state of a codebase is a commit. So anything that is referring to/based on/built upon a particular version x.y.z should reflect the codebase at commit som3ehash (or the corresponding tag x.y.z). How or when we trigger different releases (i.e., publish different artifacts built from the codebase at state x.y.z) from that code is a different story and free for us to decide - as long as we make sure that they are triggered based on that commit. Again, I tend to think that we should figure this out with python-semantic-release - which has some options there, and also makes it relatively easy to configure what should happen when (and on what branch). Personally, I think in the default state, we should just publish all artifacts (as pre-configured via this template) upon merging to main. If we or others want a more complex model for a particular project (e.g., pushing Docker images for all PRs, pushing deployment artifacts upon merging into dev or similar) - they can modify the python-semantic-release config to do so.

Does that work for you? And seem reasonable for you?