Semantic versioning and cache management for mixins and plugins

[sestegra]

Problem

Current implementation cannot manage 2 projects with different versions of mixins and plugins.

Suggestions

I suggest following updates:

• Implement the semantic versioning for mixins and plugins
• In porter.yamlfile, add version constraint in mixins collection elements (for instance similar to Terraform - Version Constraints).
• In porter.yamlfile, add version constraint in plugins configuration in config.toml
• In project directory, create a dependency lock file that store resolved version from mixins and plugins after launching porter build command
• Create mixins local cache in PORTER_HOME/cache/mixins
• Create plugins local cache in PORTER_HOME/cache/plugins
• Move config.toml to project directory to manage plugins configuration per project (not global).
• Add init command to porter to resolve mixins and plugins dependencies

During installation procedure, only porter and porter-runtime are installed.

When porter init is launched, porter will request cache manager to get resolved version.
If the mixin is present in cache, copy mixin in .cnab folder as usual.
If the mixin isn't present in cache, download mixin in PORTER_HOME/cache/mixins/<name>/<version>, then copy mixin in .cnab folder as usual.
Same behavior could be applied to plugin as well.

Examples

porter.yaml

mixins:
  - helm: 
    version: "~> 0.13.0"
  - terraform:
    version: "~> 0.6.0"

config.toml

[[azure]]
version = "~> 0.9.1"

PORTER_HOME

.porter/mixins
└── terraform
    ├── 0.5.3
    │   ├── terraform
    │   └── terraform-runtime
    ├── 0.6.0
    │   ├── terraform
    │   └── terraform-runtime
    └── 0.6.1
        ├── terraform
        └── terraform-runtime

References

• Semantic Versioning
• Terraform - Version Constraints

[sestegra]

I will update reference issues from the backlog that could be linked to this idea.

• What should be installed by default with porter? What should be installed by default with porter? #1192
• Enable listing available mixins and plugins that can be installed from the packages index Enable listing available mixins and plugins that can be installed from the packages index #1112
• Improve installation of mixins from the search results Improve installation of mixins from the search results #1109
• Epic: Install mixins defined in bundle Epic: Install mixins defined in bundle #975
• Design: Mixin Distribution Design: Mixin Distribution #150
• porter mixin update command porter mixin update command #30

[carolynvs]

I'd say the most relevant issue to start with is: Epic: Install mixins defined in bundle. That covers installing multiple versions of a mixin, pulling in the right version during build, updating the mixin commands to support versions, etc. I like the idea of a lockfile, so that if they didn't specify a version in porter.yaml we consistently resolve the same one. I've added that to the requirements for that epic.

I have some questions/feedback about some of the other changes below:

In porter.yamlfile, add version constraint in plugins configuration in config.toml

As for supported plugin versions, I'd like to know if you included plugins for completeness (if mixins why not plugins too), or if there was a specific use case you are targeting with that change. I ask because as plugins are designed today, they are a client only feature, providing different implementations for functionality that is only used on the client system. For example: storage of installation records, or resolution of secrets before injection into a bundle. Plugins are not distributed or used inside the bundles. So they do not need to be mentioned in the porter.yaml or the accompanying bundle lock file that tracks mixin versions used in the bundle.

Move config.toml to project directory to manage plugins configuration per project (not global).

I wanted to get a dig a bit deeper into the use cases for having porter client configuration (config.toml) inside bundle source code directories. If we need that, we can do it but I want to make sure we understand why some people would want it and if it's sufficient for those people's needs. Porter's config file currently controls (mostly) settings used during the execution of bundles. For example, storage of installation data, and resolution of secrets (through the plugin settings). The only relevant setting we currently have for buildtime porter commands is debug.

So if someone wanted to have bundle-level settings per bundle, it would only help them when developing their bundle. This is when the bundle source directory is available, i.e. locally testing your bundle before publishing it. After publishing a bundle, people are using --tag and running commands that use porter's settings from any directory. The proposed change doesn't cover having bundle-level settings in that case.

Add init command to porter to resolve mixins and plugins dependencies

I assume that the proposed change to have the init command be used within a bundle directory. If I'm off, let me know! 😀 Sounds like you would like a way for other people who are developing a bundle (or just for a dev with multiple machines), to have a way to install any buildtime dependencies of a bundle on their local environment. I'm going to just talk about mixins because of the unresolved question above about plugins.

This proposal sounds like what I was suggesting in #975, installing mixins used by a bundle during porter build. We could have that be a separate user-facing command if necessary, but I'd like to know if having build handle that would be sufficient, or if there are other use cases you had in mind? Having build automatically handle installing missing mixins is something that we plan on doing, so I'm mostly looking for use cases to have the separate init command someone could optionally call to only install mixins.

Create mixins local cache in PORTER_HOME/cache/mixins

The original plan was to have the mixins directory in PORTER_HOME/mixins act as the cache since it will have multiple versions of each mixin there already. I think that would be sufficient and we would not need a second directory, PORTER_HOME/cache/mixins, that also caches mixin versions.

With that one directory, and changing how Porter looks for the mixin client binary so that it looks in BUNDLE_DIR/.cnab/mixins, we can make a relatively small change for how porter calls mixins (just change the base directory to look in, but it doesn't need to be aware of the version when calling the helm mixin), and the version resolution changes only need to happen for the porter mixins commands and the process immediately before build when we copy mixins into the .cnab directory.

[sestegra]

Sorry, I hadn't got well the difference between plugins and mixins. I better understand it now.
Most of my suggestion mistakes come from this lack of knowledge.

You're right, Epic: Install mixins defined in bundle is the first Epic to work on.

In porter.yamlfile, add version constraint in plugins configuration in config.toml

In fact, I've put this statement for completness. So, I removed it and related points from my suggestion.

Move config.toml to project directory to manage plugins configuration per project (not global).

I don't have any use case in mind, it's not relevant anymore. So, I removed it from my suggestion.

Add init command to porter to resolve mixins and plugins dependencies
Create mixins local cache in PORTER_HOME/cache/mixins

I agree, no extra command is required to download mixins and extra cache directory is useless.
These points are addressed in #975.

[carolynvs]

I have written up a PEP and submitted a draft getporter/proposals#7.

This may not include everything that was proposed here, for example changes to the configuration system should be in its own PEP because there are other outstanding issues that also want to make changes to that area. Also because the PEP is huge and I wanted to limit scope. 😁

@sestegra I put your name down as an author of the PEP as well. Let me know if you are okay with that! If you have changes you'd like to suggest or discuss, let's do that on that PR so that we have it all in one place.

[sestegra]

No problem, I'm okay with that.