RFC - Support safe subdirectory-based plugin conf loading

[bleggett]

Designed to address #928 (and related issues such as #878), largely building off the ideas in #928.

The basic problem is that since plugin config is stored in a centralized, non-locked config file on the node, if multiple node agents install or remove CNI plugins in the same chain, they are both mutating a shared node resource with no locking, which unavoidably leads to unfixable TOCTOU errors.

This resolves that by moving the plugin configs out of the main shared node config, and into subdirectories, Unix init-system style.

One thing that is TBD is whether we rev the network config object spec for this or not.

• Proposal: drop-in directories #928 describes a mechanism where the subdir-based loading is dynamic and not explicit in the top-level config.

• This approach revisions the network config object spec to remove the old plugins array, more clearly indicating that subdirectories will be used instead, and forcing a hard version bump so it is not ambiguous as to what runtimes support this.

This adds a new config flag loadPluginsFromFolder - if present, for a given named network bar, plugin configuration objects will be loaded from <path-to-bar-network-config-file>/bar/xxx.conf

If this flag is not present, inlined plugin configs will be handled as they were previously. If this flag is true and there are inlined plugin config objects in the primary network config, the libcni config loading will combine everything into one plugin list.

Current things to note:

• Impl is open for debate but here to spark debate.
• This will rev the config spec from 1.0.0
• Given that we still keep support for pre-1.0.0 spec formats in libcni, and this proposal will add yet another post-1.0.0 spec format, and we should support both 1.0.0 and post-1.0.0 formats in libcni for transitional purposes, that leaves us with libcni's API surface supporting 3 different versions of the spec, 2 of which would be non-current.
• I have marked all pre-1.0.0 conf loading functions as deprecated.
• Related to that, we have a lot of typenames that no longer accurately represent what they are, in this PR I have renamed them while keeping backwards-compatible aliases (marked as deprecated) around for the time being.

[bleggett]

One question here - we are still carrying non-conflist format support in libcni - it's been ages since we dropped that from the spec, and I would suggest we actively drop support for that old format entirely as part of this, and cutting a new semver.

Otherwise things get quite messy with essentially 3 generations of config file support implemented in libcni, which I don't think we need to carry.

[bleggett]

Basic impl up for people to concretely argue with, which should help. Note that there are no tests yet, I want to get a feeling for what people think before I add tests.

[s1061123]

One question here - we are still carrying non-conflist format support in libcni - it's been ages since we dropped that from the spec, and I would suggest we actively drop support for that old format entirely as part of this, and cutting a new semver.

Does this means CNI no longer support old CNI version which supports conf file? If so, all version will be deprecated other than 1.0.0.

[bleggett]

One question here - we are still carrying non-conflist format support in libcni - it's been ages since we dropped that from the spec, and I would suggest we actively drop support for that old format entirely as part of this, and cutting a new semver.

Does this means CNI no longer support old CNI version which supports conf file? If so, all version will be deprecated other than 1.0.0.

I'm soliciting good arguments around why we should keep libcni support for parsing a discontinued format that has been deprecated for 3 years :D

In semver, you increase the major version to indicate to consumers of your {spec, library} that you are breaking backwards compatibility. We did that three years ago with the spec, but kept back compat parsing in libcni to ease the transition to 1.0.0.

Anyone still using the non-list format hasn't been following the spec for the last 3 years. Enabling that makes libcni harder to understand, and results in all implementation consumers having to juggle different types of files and manage upconverts (all of them have code for this, it is AFAICT never invoked once they take/use 1.0.0, but since our API surface still indicates this is a requirement they have all implemented it).

No one is writing conf files in the old format AFAICT, and shouldn't have been since 1.0.0. We only support spec version 1.0.0 in libcni. So I don't know why we still have support for parsing and upconverting pre-1.0.0 spec formats in our API, and I'd like to drop it unless someone has a good argument for keeping it.

[bleggett]

@squeed & others: this should be closer to what you wanted, PTAL.

As per discussion in the Jan 29, 2024 CNI WG:

• We are not dropping support for pre-1.0.0 conf formats. I have marked functions that handle them as deprecated, and where they are still used but the names make no sense, I have type-aliased and marked the aliases as deprecated.
• Instead of removing plugins as a key allowing inlining plugin objects (current behavior) the PR here now allows an either/or approach, and unless people enable plugin loading from subdirs in their config with an explicit, they will see no changes.
• Consumers/implementors using libcni to load and parse CNI config files should not see a difference. Implementers doing their own loading logic will need to support folder-based plugin loading in their code.
• libcni test cases added which should describe the functionality here and (most) corner cases.

[coveralls]

coverage: 64.219% (+0.5%) from 63.69%
when pulling 4254258 on bleggett:bleggett/multi-dir-pluginconfig
into e82d996 on containernetworking:main.

[bleggett]

@squeed PTAL - inverted flag as discussed.

[bleggett]

Split above into 3 commits:

1. SPEC update for loading non-inlined plugin configs: 99181ae
2. Use type aliases to let us use godoc to mark certain pre-1.0 types as deprecated with dochints: 5d96e63
3. libcni update for loading non-inlined plugin configs: 9a1d54e

[bleggett]

Also threw up #1081 based on some of the discuss here, as a start.

[MikeZappa87]

One question here - we are still carrying non-conflist format support in libcni - it's been ages since we dropped that from the spec, and I would suggest we actively drop support for that old format entirely as part of this, and cutting a new semver.

Otherwise things get quite messy with essentially 3 generations of config file support implemented in libcni, which I don't think we need to carry.

This would need to be coordinated :-/ However we should do this unless people have a case for a CNI network configuration that executes a single CNI binary.

[squeed]

Looks basically good, but please split in to 3-or-so commits (spec changes, go renames, and implementation). Thanks!

[bleggett]

Looks basically good, but please split in to 3-or-so commits (spec changes, go renames, and implementation). Thanks!

Yep, had it like that at one point. Fixed up, ty!

[squeed]

@bleggett this looks good to go, we just want to merge #1090 and cut v1.2.1 first. Then we can merge this!

[bleggett]

I'll resync this once #1090 + #1097 is in

[bleggett]

Resynced - since this already adds a type alias (NetConf -> PluginConf) it replaces some of #1097 (but continues to follow the same basic approach).

[bleggett]

@squeed @s1061123 PTAL - should be good from my perspective to merge when you wish.