KEP-4794: initial proposal for deprecating v1.Endpoints

[danwinship]

The `v1.Endpoints` API has been essentially deprecated since
EndpointSlices became GA in 1.21. Several new Service features (such
as dual-stack and topology, not to mention "services with more than
1000 endpoints") are implemented only for EndpointSlice, not for
Endpoints. Kube-proxy no longer uses Endpoints ever, for anything, and
the Gateway API conformance tests also require implementations to use
EndpointSlices.

Despite this, kube-controller-manager still does all of the work of
managing Endpoints objects for all Services, and a cluster cannot pass
the conformance test suite unless the Endpoints and EndpointSlice
Mirroring controllers are running, even though in many cases nothing
will ever look at the output of the Endpoints controller (and the
EndpointSlice Mirroring controller will never output anything).

While Kubernetes's API guarantees make it essentially impossible to
ever actually fully remove Endpoints, we should at least move toward a
world where most users run Kubernetes with the Endpoints and
EndpointSlice Mirroring controllers disabled.

#4974

/sig network
/cc @aojea @robscott

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/sig-network/OWNERS [danwinship]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sftim]

Some feedback

[aojea]

I think this makes a lot of sense, we need to find a way to move the ecosystem to EndpointSlices in a way that Endpoints will not even be required ... that I think is a good litmus test.
There still be cluster that will need endpoints, kub-apiserver aggregated-routing, kube-dns or knative project comes to mind, but I'm sure there will be a lot of projects or custom users using custom Endpoints, leader election used to use an Endpoints as lock IIRC.

One important thing is that we can not regress on quality, we can not drop just everything that uses Endpoints from e2e or we miss a lot of coverage so SIG Testing should review carefully the test plan cc: @BenTheElder @pohly

I also think this is a new challenge in Kubernetes, as v1.Endpoints is a GA API, and some of the proposals like removing them from conformance are tricky or AFAIK we never did it, I also think we have this target of 100% API endpoints coverage that will fail if we remove v1.Endpoints of comformance ... we need SIG Architecture input here @johnbelamaric @dims

Independently, definitive a topic we should work toward it, though not one of the highest priority at this point IMHO
/cc @thockin

[danwinship]

I also think this is a new challenge in Kubernetes, as v1.Endpoints is a GA API, and some of the proposals like removing them from conformance are tricky or AFAIK we never did it

The KEP does not propose to demote the Endpoints API from conformance; it only proposes to demote the Endpoints controller from conformance.

I also think we have this target of 100% API endpoints coverage that will fail if we remove v1.Endpoints of comformance

I think you're referring to tests like "framework.ConformanceIt("should test the lifecycle of an Endpoint"", and that would remain a conformance test under this plan, because it doesn't depend on the Endpoints controller at all, it just confirms that you can create/update/get/watch Endpoints objects as expected.

[robscott]

The KEP does not propose to demote the Endpoints API from conformance; it only proposes to demote the Endpoints controller from conformance.

I'm generally supportive of this direction, but it does raise the question of "what even is an API"? The API would technically still be there, but this change would effectively force any system that had been reading Endpoints managed by the Endpoints controller to transition to the EndpointSlice API which does feel pretty close to a deprecation even if the API does technically still exist.

As long as the EndpointSlice Mirroring controller is still running, people will still be able to create Endpoints and have them automatically translated to EndpointSlices, so this is primarily a question of systems that are reading from Endpoints.

I do think that we need to have an answer for what we do if/when we migrate functionality from one v1 API to another. In practice, a rather small portion of components are still relying on the Endpoints API, and an even smaller portion are relying on the Endpoints controller. Given that most of our existing e2e test coverage is focused on functionality that is now provided by the EndpointSlice controller, it's possible that a regression in the Endpoints controller could go unnoticed for a very long time.

There's also the practical matter that running the Endpoints controller is likely wasting resources on >90% of existing Kubernetes clusters. More clearly communicating that running the Endpoints controller is optional could be a good step here.

To refer to the Kubernetes deprecation policy, making the Endpoints controller optional may fall under "Deprecating a feature or behavior", which has the following guidelines:

1. Deprecated behaviors must function for no less than 1 year after their announced deprecation
2. The feature of behavior must not be deprecated in favor of an alternative implementation that is less stable

1 just requires lots of advance notice, and I think it's safe to say that the EndpointSlice controller is more widely tested and likely more reliable than the Endpoints controller at this point.

[BenTheElder]

To refer to the Kubernetes deprecation policy, making the Endpoints controller optional may fall under "Deprecating a feature or behavior", which has the following guidelines:

Yeah, I think we need to give a lot of heads up, even if we're not using it in SIG-Network owned projects/tools anymore aside from these controller we don't know how users are still relying on the endpoints being populated.

The v1.Endpoints API has been essentially deprecated since
EndpointSlices became GA in 1.21.

But not actually documented as deprecated, right?
https://kubernetes.io/docs/reference/kubernetes-api/service-resources/endpoints-v1/

Several new Service features (such
as dual-stack and topology, not to mention "services with more than
1000 endpoints") are implemented only for EndpointSlice, not for
Endpoints. Kube-proxy no longer uses Endpoints ever, for anything, and
the Gateway API conformance tests also require implementations to use
EndpointSlices.

"Clusters with <= 1000 endpoints per service that don't use dual-stack / topology" is probably still a LOT of clusters that may be using it successfully with some other controller / tooling, even if we aren't with kube-proxy?

I don't think gateway-api conformance tests are super relevant in this context?

The KEP does not propose to demote the Endpoints API from conformance; it only proposes to demote the Endpoints controller from conformance.

Important distinction that still still seems like a big expectation change, I don't think we have a good precedent for this yet.

This seems relevant to @kubernetes/api-approvers in addition to SIG-Arch.

[danwinship]

Given that most of our existing e2e test coverage is focused on functionality that is now provided by the EndpointSlice controller, it's possible that a regression in the Endpoints controller could go unnoticed for a very long time.

I had assumed this too, and that's true to the extent that lots of tests depend on the behavior of kube-proxy, and kube-proxy only looks at EndpointSlices. But there are still a surprising number of tests that look at Endpoints explicitly, and even some that look at Endpoints and not EndpointSlices (including, disturbingly, the dual-stack Service tests!)

My plan here was to ensure that everything outside of test/e2e/network/ only looks at slices, but to preserve all of the Endpoints-specific and "Endpoints match EndpointSlices" tests in test/e2e/network (with all of the Endpoints stuff eventually feature-tagged and skippable).

1 just requires lots of advance notice

Right, I should have been clearer in the KEP but I was expecting this might take a while. (Antonio mentioned deprecating the in-tree cloud providers as a comparison point.)

On that note, the KEP currently claims that demoting the Endpoints controller tests from conformance is part of "step 1", which I was thinking would sort of constitute the official announcement that we were doing this, but Antonio suggested that maybe it should come later in the process, so providers don't end up disabling the controller while there are still lots of third-party components that aren't ready for it.

But not actually documented as deprecated, right?

Yeah... I had kinda been thinking it was, though it's obvious in hindsight that it's not (I look at those API docs all the time and should have noticed this).

"Clusters with <= 1000 endpoints per service that don't use dual-stack / topology" is probably still a LOT of clusters

Yes; I was thinking more in terms of "how much third-party software is still using Endpoints rather than EndpointSlices?". Even if most individual users of FooBarProxy™ have small single-stack clusters, the fact that some don't should mean that the authors would have been under pressure to port away from Endpoints for a while now.

I don't think gateway-api conformance tests are super relevant in this context?

I was mentioning it because it means we don't have to worry that there are Gateway implementations that are based on Endpoints rather than EndpointSlice.

[sftim]

Nothing in our stability guarantees stops us from, eg, making and promoting a ValidatingAdmissionPolicy that blocks Endpoints creation.

[BenTheElder]

On that note, the KEP currently claims that demoting the Endpoints controller tests from conformance is part of "step 1", which I was thinking would sort of constitute the official announcement that we were doing this, but Antonio suggested that maybe it should come later in the process, so providers don't end up disabling the controller while there are still lots of third-party components that aren't ready for it.

+1, I think it should look closer to cloud provider removal:

1. Formally deprecated. Release note highlight, blog, etc.
2. Wait
3. Disable controllers + drop from conformance. New release note highlight, blog, etc.
4. Wait
5. Remove controllers?