Replies: 9 comments 38 replies
-
|
One thing to clarify here: This is just for new resources, not for new fields on graduated resources, right? |
Beta Was this translation helpful? Give feedback.
-
|
If you had an experimental API in the |
Beta Was this translation helpful? Give feedback.
-
|
@JoelSpeed that's correct. I'll rephrase this to clarify that it would avoid those errors when anyone was trying to install standard channel CRDs. Edit: Actually already had "when upgrading from experimental to standard", open to other ways to clarify this. |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure I understand how this proposal solves that problem. The schema cannot change between versions without a conversion webhook...? |
Beta Was this translation helpful? Give feedback.
-
|
This proposal ensures that installing standard channel CRDs will always be safe and smooth at the cost of removing any theoretical in-place upgrade path from experimental to standard channel. They would be entirely different resources with different API groups and thus the upgrade path would be to create equivalent standard channel resources. |
Beta Was this translation helpful? Give feedback.
-
|
This does not solve the "breaking changes need a conversion web hook to not be breaking changes" problem, and isn't intended to. As Rob says, this is currently only working towards making sure that Standard is always safe, and that using Experimental can't affect that. This is not the last step, to be clear, it's the first. But we want to see if this approach is worth the cost to implementation and users before pushing too hard. |
Beta Was this translation helpful? Give feedback.
-
|
@robscott would be good to also gather feedback from implementations on who will support this new |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, thanks for the clarifications here and on chat. I am good with this provided we do as much as possible to make experimental truly unstable, notably the expectation that implementations should not support experimental and v1 concurrently (I am not sure if we can somehow enforce this, since its kind of a prisoners dilemma -- see companies selling LTS, once one does it (for $$$ usually) there is an expectation others do it too). I do, however, have concerns about the proposals that use this as a stepping stone, around "Kubernetes comes with the CRDs enabled by default and you cannot get experimental fields". That certainly doesn't need to be linked to this decision though. |
Beta Was this translation helpful? Give feedback.
-
Do you mean concurrently in the sense of "actively for a single installation within a cluster" (this feels like it might be possible to enforce with conformance tests, and could possibly encourage safer patterns like using dev/staging clusters for testing experimental functionality), or "within a project's codebase"? |
Beta Was this translation helpful? Give feedback.
-
|
@mikemorris the big concern with splitting groups (from a maintainer POV) is the cost to support 2 groups is orders of magnitude larger than to support 2 versions of the same group (since versions are identical). So I really don't want to get into a world where an implementation has to deal with that. The easy way out is... say GW v1.5 promotes FooRoute. So v1.4 had xFooRoute. v1.5 should remove xFooRoute and add FooRoute (stable). Implementations supporting v1.5 should replace xFooRoute support with FooRoute support. tl;dr "within a projects codebase" |
Beta Was this translation helpful? Give feedback.
-
I remember this coming up before and I think there was some consideration that it might be possible to work with SIG API Machinery or improve tooling to make this easier?
That wouldn't solve for new experimental fields on existing stable resources though I think? |
Beta Was this translation helpful? Give feedback.
-
Correct -- this proposal is not covering that though |
Beta Was this translation helpful? Give feedback.
-
|
Posted in slack but I'll surface this here Circling back to the discussion on which group |
Beta Was this translation helpful? Give feedback.
-
|
Yeah this specific proposal doesn't really provide isolation for experimental fields. My previous proposal that did (#3106) was not particularly popular, so I'm trying to start smaller here. |
Beta Was this translation helpful? Give feedback.
-
|
Agreed this is a bit awkward for features spanning across new resources and additions to existing resources, where perhaps either resources from the x-k8s.io group would be available but only alongside standard channel resources, or new fields on existing experimental channel resources in the newest release of the main group may become available without support in the implementation for x-k8s.io resources. This could lead to "dead fields" as @shaneutt describes in #3576 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
To clarify, the changes I'm proposing here are quite tiny. The only difference from today is that new resources in experimental channel would start in a different experimental API group. Everything else about experimental channel would stay the same. In practice that means that installing experimental channel would continue to get you resources were fully experimental, and resources that were GA with experimental fields. Although I'd like to pursue broader changes like a full separation between release channels to eliminate problems like dead fields, this specific proposal is really small. |
Beta Was this translation helpful? Give feedback.
-
|
Gotcha, that makes sense - I was jumping ahead a bit to thinking about which resources a cloud provider might allow or install by default on a cluster. |
Beta Was this translation helpful? Give feedback.
-
|
The APIs represent a common way to solve a specific problem. If 2
implementations use the same API - and 15 don't implement it - it is still
better than the 2 implementations using different API, and if it is a
rarely used feature - it is better than attempting to
force the 15 others to implement it, and a good signal that the feature is
not important enough to be part of the standard.
My main concern is with calling this 'experimental' - like we called a lot
of APIs 'alpha' before but still encouraged
users to use in production. Should be treated just like vendor-specific v1
APIs, with the main difference that more than
1 vendor is using the same API.
I used to be very enthusiastic about this proposal - but right now I think
a better approach is to do nothing,
and let each implementation define their own v1 API - perhaps with some
communication and discussion on
a common behavior. Let them try different things, get feedback, wait to see
adoption and convergence - and
only then attempt to standardize. Defining any standard API (or protocol)
by committee, before having real
world experience is never working well.
…On Wed, Jan 29, 2025 at 5:55 PM Rob Scott ***@***.***> wrote:
Yeah this specific proposal doesn't really provide isolation for
experimental *fields*. My previous proposal that did (#3106
<#3106>) was
not particularly popular, so I'm trying to start smaller here.
—
Reply to this email directly, view it on GitHub
<#3497 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAUR2S67EXLO4WRTOHXS432NGBBRAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBQGE4TENY>
.
You are receiving this because you commented.Message ID:
<kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12001927@
github.com>
|
Beta Was this translation helpful? Give feedback.
-
|
Not sure how supporting 2 groups is 'orders of magnitude' - I would expect
at worst double, and no worse than what Istio does with HttpRoute and
VirtualService.
All implementation I know have vendor extensions - and I would expect they
would remain supported, not thrown away
even if a subset or variant of the extension is adopted ( and this is
likely to be on a permanent basis for most
implementations - expecting that upstream implements every single features
that any vendor needs is not viable).
Maybe the confusion is 'experiments' ( which should not be used by anyone
in production ) versus features
like FooRoute - that some vendors may adopt and maintain on their own for
their customers. There is
no expectation that a vendor FooRoute will go away and be replaced by an
upstream FooRoute, just like
there was no expectation that VirtualService will be deprecated when
HttpRoute was v1.
Since 'experiments' should never be in production - I am not at all
concerned with their fate, as long as we avoid
the istio confusion between alpha and v1.
The (single or multi-vendor) FooRoute can be v1 and can have more fields or
behave differently - and may
have its own vendor-defined support lifecycle, and since common APIs are
expected to be implementable
by everyone - we can't expect implementations that can support more
features to drop that.
…On Fri, Jan 31, 2025 at 9:26 AM John Howard ***@***.***> wrote:
@mikemorris <https://github.com/mikemorris> the big concern with
splitting groups (from a maintainer POV) is the cost to support 2 groups is
orders of magnitude larger than to support 2 versions of the same group
(since versions are identical).
So I really don't want to get into a world where an implementation has to
deal with that.
The easy way out is... say GW v1.5 promotes FooRoute. So v1.4 had
xFooRoute. v1.5 should remove xFooRoute and add FooRoute (stable).
Implementations supporting v1.5 should *replace* xFooRoute support with
FooRoute support.
tl;dr "within a projects codebase"
—
Reply to this email directly, view it on GitHub
<#3497 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAUR2XLEC7TXQXEIZJ4N5L2NOW5BAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBSGA2DKOI>
.
You are receiving this because you commented.Message ID:
<kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12020459@
github.com>
|
Beta Was this translation helpful? Give feedback.
-
|
supporting 2 versions is literally 0 effort so orders of magnitude was incorrect - more like infinite 🙂 I agree vendor Foo and experimental Foo are different, and this proposal is for experimental. The difference being that as (for example) the Istio project there is a ~0% chance I am going to implement GKEFoo and a very high chance I will implement ExperimentalFoo |
Beta Was this translation helpful? Give feedback.
-
|
I get where you are coming from and would agree a lot more if that is what was being proposed - but it doesn't seem to be. |
Beta Was this translation helpful? Give feedback.
-
|
In other words: I read x-k8s.io as "eXtended" - not 'eXperimental'.
A collection of gateway API extensions that have common definition across
few vendors - and are as v1
as the core APIs - but can't be supported by majority/all vendors, or are
too corner cases.
If some of those extensions become common or popular - the same or slightly
different API can be added
to core, of course - and the vendors will have a 2x effort to keep both.
…On Fri, Jan 31, 2025 at 5:14 PM Costin Manolache ***@***.***> wrote:
Not sure how supporting 2 groups is 'orders of magnitude' - I would expect
at worst double, and no worse than what Istio does with HttpRoute and
VirtualService.
All implementation I know have vendor extensions - and I would expect they
would remain supported, not thrown away
even if a subset or variant of the extension is adopted ( and this is
likely to be on a permanent basis for most
implementations - expecting that upstream implements every single features
that any vendor needs is not viable).
Maybe the confusion is 'experiments' ( which should not be used by anyone
in production ) versus features
like FooRoute - that some vendors may adopt and maintain on their own for
their customers. There is
no expectation that a vendor FooRoute will go away and be replaced by an
upstream FooRoute, just like
there was no expectation that VirtualService will be deprecated when
HttpRoute was v1.
Since 'experiments' should never be in production - I am not at all
concerned with their fate, as long as we avoid
the istio confusion between alpha and v1.
The (single or multi-vendor) FooRoute can be v1 and can have more fields
or behave differently - and may
have its own vendor-defined support lifecycle, and since common APIs are
expected to be implementable
by everyone - we can't expect implementations that can support more
features to drop that.
On Fri, Jan 31, 2025 at 9:26 AM John Howard ***@***.***>
wrote:
> @mikemorris <https://github.com/mikemorris> the big concern with
> splitting groups (from a maintainer POV) is the cost to support 2 groups is
> orders of magnitude larger than to support 2 versions of the same group
> (since versions are identical).
>
> So I really don't want to get into a world where an implementation has to
> deal with that.
>
> The easy way out is... say GW v1.5 promotes FooRoute. So v1.4 had
> xFooRoute. v1.5 should remove xFooRoute and add FooRoute (stable).
> Implementations supporting v1.5 should *replace* xFooRoute support with
> FooRoute support.
>
> tl;dr "within a projects codebase"
>
> —
> Reply to this email directly, view it on GitHub
> <#3497 (reply in thread)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAUR2XLEC7TXQXEIZJ4N5L2NOW5BAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBSGA2DKOI>
> .
> You are receiving this because you commented.Message ID:
> <kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12020459@
> github.com>
>
|
Beta Was this translation helpful? Give feedback.
-
|
On Fri, Jan 31, 2025 at 5:23 PM John Howard ***@***.***> wrote:
supporting 2 versions is literally 0 effort so orders of magnitude was incorrect - more like infinite 🙂
I agree vendor Foo and experimental Foo are different, and this proposal is for experimental. The difference being that as (for example) the Istio project there is a ~0% chance I am going to implement GKEFoo and a very high chance I will implement ExperimentalFoo
Right, most vendors will implement APIs defined by another vendor -
and that is the problem,
each vendor defining a XXXFoo API that is slightly different from each
other - all v1 and supported but
subtly different.
Having a common gateway-extensions.io/Foo is far better for users, if
2-3 vendors agree on a common Foo.
From the 'core' API perspective, each single vendor extension and each
multi-vendor extension are 'experimental',
in the sense that there is no clear evidence it is a universal problem
or consensus across all vendors. I think that
is the main purpose (and benefit) of vendor extensions, it allows
vendor to 'experiment' by releasing features
and APIs.
If x-net.io/Foo (v1) was defined - maybe both Istio and GKE and other
could implement it, users can adopt
such a feature with confidence it will be supported as a v1 API, and
when a core Foo API is defined we can
use the lessons learned from x-net.io/Foo and make changes as needed.
Istio implementing ExperimentalFoo - and dealing with migrations and
users who (as usual) adopt
ExperimentalFoo in production ignoring the alpha and experimental
labels - is indeed order of magnitudes
harder than support an x-net.io/FooV1 or an istio.io/FooV1 API long
term and not dealing with any migration.
|
Beta Was this translation helpful? Give feedback.
-
|
I feel very strongly that we need to move to having all APIs copied to a new experimental group, rather than just new ones. In the past we've lived in a world where an implementation manages the CRDs. Increasingly since GA we live in a world where these APIs are deployed and managed by platforms, and more and more treated like core APIs as multiple implementations need to be using the same APIs. In this future where platforms need to serve Gateway API to multiple implementations, I do not see how we can effectively "overlay" experimental fields with the standard APIs. This means that fields may or may not be present on a "core-like" API and this is not a good situation to be in. Ultimately too, I think the complexity of one API potentially being one, or a very different experimental version on the same production cluster will drive platforms to block experimental to control API surface. I think we should:
And I think this has a huge impact on how #3576 plays out. |
Beta Was this translation helpful? Give feedback.
-
If you could please link all discussions (and maybe the individual comments) across discussions, Slack, etc that you're thinking of that would be helpful to help me better understand perspectives, and I will read them all. |
Beta Was this translation helpful? Give feedback.
-
|
A couple of the things driving me with this alternative proposal are:
I wanted to think through what I see as the pros and cons of each approach. I'll call the approach I'm proposing "full copy" and the other "new only". Full CopyPros
Cons
New OnlyPros
Cons
Note that the implementations that are already heavily invested in experimental have work to do in either case, which is an unfortunate (but anticipated) cost of investing in something labeled |
Beta Was this translation helpful? Give feedback.
-
|
imo there are 3 types of experimental features
If Gateway API goes ahead with the What happens with the current state of Experimental APIs and fields in Standard APIs ? My hope is, the decision to either graduate them to Standard (like TCPRoute) or drop them, is prioritized, so the impact of rewriting configs is minimized for the user. |
Beta Was this translation helpful? Give feedback.
-
|
@arkodg, do you see a substantive difference in the costs to Envoy Gateway of the full-copy vs new-only proposals? or are you saying that the costs to Envoy Gateway are prohibitive for either? More than that, though, I'm curious how you see the costs to Ana (and Chihiro, but Ana is Envoy Gateway's primary user, so I think focusing on her first is relevant)? I tend to agree with you that experimental features come in three flavors, FTR, though I tend to think that they all mostly look the same to Gateway API itself: either they're a thing that we think would benefit multiple implementations and we should be thinking about a path to standard, or they're not and we should think about whether they should be an implementation-specific extension rather than a mainline Gateway API thing. I also tend to agree that we should prioritize getting existing experimental elements out of experimental - one way or the other! - before adding new ones. |
Beta Was this translation helpful? Give feedback.
-
|
@kflynn the cost is high for both cases - full-copy and new only proposals Regarding Ana
im just trying to highlight how this situation we are in, is not a win-win one for the API group and implementations |
Beta Was this translation helpful? Give feedback.
-
|
I think we are still using the word 'experiment' in a very confusing way.
In your '3 kinds of experiments' -
for 1 you mention 'are used in production', so perhaps you mean the same
thing as I do, a v1 vendor
extensions that are considered as an 'experiment' for the core API, in the
sense that other vendors
and the WG waits to get feedback on how the v1 vendor feature behaves and
what adoptions it gets.
Assuming we mean the same thing - vendor v1 APIs, that multiple vendors
support but in slightly
different ways - having a common API, even if not part of this WG - is a
good thing for users, and it
doesn't really matter if Gateaway API is planning or not to adopt it in
core, so (2) is the same as (1),
stable v1 APIs that users can safely rely on with no risks or problems, but
is outside of this WG but
still cross-vendor.
(3) is the only tricky one - if Gateway API decides to adopt an API from
(1) or (2) category, there
will be some pain for the user in using both the original (1/2) API and (3)
at the same time - but it
is reasonable to assume the vendor API will remain a super-set of the
Gateway API and remain
supported long-term by the vendors.
The only source of pain is if a user touches any 'alpha'/experimental CRD
in prod, and that is something
every vendor and K8S community should do a better job of preventing and
educating users about.
…On Fri, Feb 7, 2025 at 11:20 AM Arko Dasgupta ***@***.***> wrote:
@kflynn <https://github.com/kflynn> the cost is high for both cases -
full-copy and new only proposals
Regarding Ana
1. if envoy gateway doesnt implement the full copy or new-only
proposal, because it already supports the feature using 1. or 2. then
Ana shouldnt mind, because her use case is solved
2. If many users voice the fact that they want to use the Gateway API
version of the feature, that may be a consideration for Envoy Gateway but
its a stronger signal for Gateway API to move the feature into standard
im just trying to highlight how this situation we are in, is not a win-win
one for the API group and implementations
—
Reply to this email directly, view it on GitHub
<#3497 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAUR2SP6FOK34FCXDQMFPT2OUBQLAVCNFSM6AAAAABTOYPLEKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBZHAYDMMI>
.
You are receiving this because you commented.Message ID:
<kubernetes-sigs/gateway-api/repo-discussions/3497/comments/12098061@
github.com>
|
Beta Was this translation helpful? Give feedback.
-
As a follow up to #3106, I'd like to propose a smaller subset of that to address many of the same problems:
Importantly this means that when a new resource graduates to GA, users would need to manually copy their config over from x-k8s.io (alpha) to k8s.io (v1) CRs. Although this is rather onerous, I'd argue that it's preferable to the current experience which can result in confusing error messages when trying to install Gateway API standard channel. This new approach would ensure that it was always safe to install standard channel CRDs. For example, this could allow Kubernetes to include Gateway API standard channel by default (x-ref #3576).
I don't really like 3) here as it means we're not solving the full set of problems that fully separate API groups for experimental channel and standard channel would, but this more limited approach does come with some advantages. Notably, controllers would not need to support both experimental and standard channel API groups at the same time for the same resource. Instead, they could transition to the
k8s.ioAPI group at the same time the API does. If they want to support a broader set of versions, they can support both API groups for a limited period of time, but importantly any Gateway API release would only ever include a single API group (and version) per resource.Note: All mentions of API groups here are just focused on the suffix. In all cases, they would also include the
gateway.networking.prefix they already do.Beta Was this translation helpful? Give feedback.
All reactions