#1967 Customize K8s services creation in Kube service discovery pro…

…vider (#2052)

* Initial refactoring

* Interfaces namespace

* `IKubeServiceBuilder` interface vs `KubeServiceBuilder` class

* `IKubeServiceCreator` interface vs `KubeServiceCreator` class

* Customize K8s services creation

* Add logger

* namespace Ocelot.AcceptanceTests.ServiceDiscovery

* Add `KubernetesServiceDiscoveryTests`

* Unit tests

* AAA pattern

* Acceptance tests

* Update kubernetes.rst

* Check docs

docs/features/kubernetes.rst

@@ -73,7 +73,11 @@ The example here shows a typical configuration:
     }
 
 Service deployment in **Namespace** ``Dev``, **ServiceDiscoveryProvider** type is ``Kube``, you also can set :ref:`k8s-pollkube-provider` type.
-Note: **Host**, **Port** and **Token** are no longer in use.
+
+  **Note 1**: ``Host``, ``Port`` and ``Token`` are no longer in use.
+
+  **Note 2**: The ``Kube`` provider searches for the service entry using ``ServiceName`` and then retrieves the first available port from the ``EndpointSubsetV1.Ports`` collection.
+  Therefore, if the port name is not specified, the default downstream scheme will be ``http``; 
 
 .. _k8s-pollkube-provider:
 
@@ -99,10 +103,10 @@ This really depends on how volatile your services are.
 We doubt it will matter for most people and polling may give a tiny performance improvement over calling Kubernetes per request.
 There is no way for Ocelot to work these out for you. 
 
-Global vs Route levels
-^^^^^^^^^^^^^^^^^^^^^^
+Global vs Route Levels
+----------------------
 
-If your downstream service resides in a different namespace, you can override the global setting at the Route-level by specifying a **ServiceNamespace**:
+If your downstream service resides in a different namespace, you can override the global setting at the Route-level by specifying a ``ServiceNamespace``:
 
 .. code-block:: json
 
@@ -113,7 +117,45 @@ If your downstream service resides in a different namespace, you can override th
     }
   ]
 
+Downstream Scheme vs Port Names [#f3]_
+--------------------------------------
+
+Kubernetes configuration permits the definition of multiple ports with names for each address of an endpoint subset.
+When binding multiple ports, you assign a name to each subset port.
+To allow the ``Kube`` provider to recognize the desired port by its name, you need to specify the ``DownstreamScheme`` with the port's name;
+if not, the collection's first port entry will be chosen by default.
+
+For instance, consider a service on Kubernetes that exposes two ports: ``https`` for **443** and ``http`` for **80**, as follows:
+
+.. code-block:: text
+
+  Name:         my-service
+  Namespace:    default
+  Subsets:
+    Addresses:  10.1.161.59
+    Ports:
+      Name   Port  Protocol
+      ----   ----  --------
+      https  443   TCP
+      http   80    TCP
+
+**When** you need to use the ``http`` port while intentionally bypassing the default ``https`` port (first one),
+you must define ``DownstreamScheme`` to enable the provider to recognize the desired ``http`` port by comparing ``DownstreamScheme`` with the port name as follows:
+
+.. code-block:: json
+
+  "Routes": [
+    {
+      "ServiceName": "my-service",
+      "DownstreamScheme": "http", // port name -> http -> port is 80
+    }
+  ]
+
+**Note**: In the absence of a specified ``DownstreamScheme`` (which is the default behavior), the ``Kube`` provider will select **the first available port** from the ``EndpointSubsetV1.Ports`` collection.
+Consequently, if the port name is not designated, the default downstream scheme utilized will be ``http``.
+
 """"
 
 .. [#f1] `Wikipedia <https://en.wikipedia.org/wiki/Kubernetes>`_ | `K8s Website <https://kubernetes.io/>`_ | `K8s Documentation <https://kubernetes.io/docs/>`_ | `K8s GitHub <https://github.com/kubernetes/kubernetes>`_
 .. [#f2] This feature was requested as part of `issue 345 <https://github.com/ThreeMammals/Ocelot/issues/345>`_ to add support for `Kubernetes <https://kubernetes.io/>`_ :doc:`../features/servicediscovery` provider. 
+.. [#f3] *"Downstream Scheme vs Port Names"* feature was requested as part of `issue 1967 <https://github.com/ThreeMammals/Ocelot/issues/1967>`_ and released in version `23.3 <https://github.com/ThreeMammals/Ocelot/releases/tag/23.3.0>`_

src/Ocelot.Provider.Kubernetes/EndPointClientV1.cs

@@ -1,6 +1,7 @@
 using HTTPlease;
 using KubeClient.Models;
 using KubeClient.ResourceClients;
+using Ocelot.Provider.Kubernetes.Interfaces;
 
 namespace Ocelot.Provider.Kubernetes
 {

src/Ocelot.Provider.Kubernetes/IEndPointClient.cs → src/Ocelot.Provider.Kubernetes/Interfaces/IEndPointClient.cs

@@ -1,7 +1,7 @@
 using KubeClient.Models;
 using KubeClient.ResourceClients;
 
-namespace Ocelot.Provider.Kubernetes;
+namespace Ocelot.Provider.Kubernetes.Interfaces;
 
 public interface IEndPointClient : IKubeResourceClient
 {

src/Ocelot.Provider.Kubernetes/Interfaces/IKubeServiceBuilder.cs

@@ -0,0 +1,9 @@
+using KubeClient.Models;
+using Ocelot.Values;
+
+namespace Ocelot.Provider.Kubernetes.Interfaces;
+
+public interface IKubeServiceBuilder
+{
+    IEnumerable<Service> BuildServices(KubeRegistryConfiguration configuration, EndpointsV1 endpoint);
+}

src/Ocelot.Provider.Kubernetes/Interfaces/IKubeServiceCreator.cs

@@ -0,0 +1,10 @@
+using KubeClient.Models;
+using Ocelot.Values;
+
+namespace Ocelot.Provider.Kubernetes.Interfaces;
+
+public interface IKubeServiceCreator
+{
+    IEnumerable<Service> Create(KubeRegistryConfiguration configuration, EndpointsV1 endpoint, EndpointSubsetV1 subset);
+    IEnumerable<Service> CreateInstance(KubeRegistryConfiguration configuration, EndpointsV1 endpoint, EndpointSubsetV1 subset, EndpointAddressV1 address);
+}

src/Ocelot.Provider.Kubernetes/Kube.cs

@@ -1,5 +1,6 @@
 using KubeClient.Models;
 using Ocelot.Logging;
+using Ocelot.Provider.Kubernetes.Interfaces;
 using Ocelot.Values;
 
 namespace Ocelot.Provider.Kubernetes;
@@ -9,47 +10,44 @@ namespace Ocelot.Provider.Kubernetes;
 /// </summary>
 public class Kube : IServiceDiscoveryProvider
 {
-    private readonly KubeRegistryConfiguration _kubeRegistryConfiguration;
+    private readonly KubeRegistryConfiguration _configuration;
     private readonly IOcelotLogger _logger;
     private readonly IKubeApiClient _kubeApi;
-
-    public Kube(KubeRegistryConfiguration kubeRegistryConfiguration, IOcelotLoggerFactory factory, IKubeApiClient kubeApi)
+    private readonly IKubeServiceBuilder _serviceBuilder;
+    private readonly List<Service> _services;
+
+    public Kube(
+        KubeRegistryConfiguration configuration,
+        IOcelotLoggerFactory factory,
+        IKubeApiClient kubeApi,
+        IKubeServiceBuilder serviceBuilder)
     {
-        _kubeRegistryConfiguration = kubeRegistryConfiguration;
+        _configuration = configuration;
         _logger = factory.CreateLogger<Kube>();
         _kubeApi = kubeApi;
+        _serviceBuilder = serviceBuilder;
+        _services = new();
     }
 
-    public async Task<List<Service>> GetAsync()
+    public virtual async Task<List<Service>> GetAsync()
     {
         var endpoint = await _kubeApi
             .ResourceClient(client => new EndPointClientV1(client))
-            .GetAsync(_kubeRegistryConfiguration.KeyOfServiceInK8s, _kubeRegistryConfiguration.KubeNamespace);
+            .GetAsync(_configuration.KeyOfServiceInK8s, _configuration.KubeNamespace);
 
-        var services = new List<Service>();
-        if (endpoint != null && endpoint.Subsets.Any())
+        _services.Clear();
+        if (endpoint?.Subsets.Count != 0)
         {
-            services.AddRange(BuildServices(endpoint));
+            _services.AddRange(BuildServices(_configuration, endpoint));
         }
         else
         {
-            _logger.LogWarning(() => $"namespace:{_kubeRegistryConfiguration.KubeNamespace}service:{_kubeRegistryConfiguration.KeyOfServiceInK8s} Unable to use ,it is invalid. Address must contain host only e.g. localhost and port must be greater than 0");
+            _logger.LogWarning(() => $"K8s Namespace:{_configuration.KubeNamespace}, Service:{_configuration.KeyOfServiceInK8s}; Unable to use: it is invalid. Address must contain host only e.g. localhost and port must be greater than 0!");
         }
 
-        return services;
+        return _services;
     }
 
-    private static List<Service> BuildServices(EndpointsV1 endpoint)
-    {
-        var services = new List<Service>();
-
-        foreach (var subset in endpoint.Subsets)
-        {
-            services.AddRange(subset.Addresses.Select(address => new Service(endpoint.Metadata.Name,
-                new ServiceHostAndPort(address.Ip, subset.Ports.First().Port),
-                endpoint.Metadata.Uid, string.Empty, Enumerable.Empty<string>())));
-        }
-
-        return services;
-    }
+    protected virtual IEnumerable<Service> BuildServices(KubeRegistryConfiguration configuration, EndpointsV1 endpoint)
+        => _serviceBuilder.BuildServices(configuration, endpoint);
 }

src/Ocelot.Provider.Kubernetes/KubeRegistryConfiguration.cs

@@ -1,8 +1,8 @@
-namespace Ocelot.Provider.Kubernetes
+namespace Ocelot.Provider.Kubernetes;
+
+public class KubeRegistryConfiguration
 {
-    public class KubeRegistryConfiguration
-    {
-        public string KubeNamespace { get; set; }
-        public string KeyOfServiceInK8s { get; set; }
-    }
+    public string KubeNamespace { get; set; }
+    public string KeyOfServiceInK8s { get; set; }
+    public string Scheme { get; set; }
 }

src/Ocelot.Provider.Kubernetes/KubeServiceBuilder.cs

@@ -0,0 +1,36 @@
+using KubeClient.Models;
+using Ocelot.Logging;
+using Ocelot.Provider.Kubernetes.Interfaces;
+using Ocelot.Values;
+
+namespace Ocelot.Provider.Kubernetes;
+
+public class KubeServiceBuilder : IKubeServiceBuilder
+{
+    private readonly IOcelotLogger _logger;
+    private readonly IKubeServiceCreator _serviceCreator;
+
+    public KubeServiceBuilder(IOcelotLoggerFactory factory, IKubeServiceCreator serviceCreator)
+    {
+        ArgumentNullException.ThrowIfNull(factory);
+        _logger = factory.CreateLogger<KubeServiceBuilder>();
+
+        ArgumentNullException.ThrowIfNull(serviceCreator);
+        _serviceCreator = serviceCreator;
+    }
+
+    public virtual IEnumerable<Service> BuildServices(KubeRegistryConfiguration configuration, EndpointsV1 endpoint)
+    {
+        ArgumentNullException.ThrowIfNull(configuration);
+        ArgumentNullException.ThrowIfNull(endpoint);
+
+        var services = endpoint.Subsets
+            .SelectMany(subset => _serviceCreator.Create(configuration, endpoint, subset))
+            .ToArray();
+
+        _logger.LogDebug(() => $"K8s '{Check(endpoint.Kind)}:{Check(endpoint.ApiVersion)}:{Check(endpoint.Metadata?.Name)}' endpoint: Total built {services.Length} services.");
+        return services;
+    }
+
+    private static string Check(string str) => string.IsNullOrEmpty(str) ? "?" : str;
+}

src/Ocelot.Provider.Kubernetes/KubeServiceCreator.cs

@@ -0,0 +1,59 @@
+using KubeClient.Models;
+using Ocelot.Logging;
+using Ocelot.Provider.Kubernetes.Interfaces;
+using Ocelot.Values;
+
+namespace Ocelot.Provider.Kubernetes;
+
+public class KubeServiceCreator : IKubeServiceCreator
+{
+    private readonly IOcelotLogger _logger;
+
+    public KubeServiceCreator(IOcelotLoggerFactory factory)
+    {
+        ArgumentNullException.ThrowIfNull(factory);
+        _logger = factory.CreateLogger<KubeServiceCreator>();
+    }
+
+    public virtual IEnumerable<Service> Create(KubeRegistryConfiguration configuration, EndpointsV1 endpoint, EndpointSubsetV1 subset)
+        => (configuration == null || endpoint == null || subset == null)
+            ? Array.Empty<Service>()
+            : subset.Addresses
+                .SelectMany(address => CreateInstance(configuration, endpoint, subset, address))
+                .ToArray();
+
+    public virtual IEnumerable<Service> CreateInstance(KubeRegistryConfiguration configuration, EndpointsV1 endpoint, EndpointSubsetV1 subset, EndpointAddressV1 address)
+    {
+        var instance = new Service(
+            GetServiceName(configuration, endpoint, subset, address),
+            GetServiceHostAndPort(configuration, endpoint, subset, address),
+            GetServiceId(configuration, endpoint, subset, address),
+            GetServiceVersion(configuration, endpoint, subset, address),
+            GetServiceTags(configuration, endpoint, subset, address)
+        );
+        return new Service[] { instance };
+    }
+
+    protected virtual string GetServiceName(KubeRegistryConfiguration configuration, EndpointsV1 endpoint, EndpointSubsetV1 subset, EndpointAddressV1 address)
+        => endpoint.Metadata?.Name;
+
+    protected virtual ServiceHostAndPort GetServiceHostAndPort(KubeRegistryConfiguration configuration, EndpointsV1 endpoint, EndpointSubsetV1 subset, EndpointAddressV1 address)
+    {
+        var ports = subset.Ports;
+        bool portNameToScheme(EndpointPortV1 p) => string.Equals(p.Name, configuration.Scheme, StringComparison.InvariantCultureIgnoreCase);
+        var portV1 = string.IsNullOrEmpty(configuration.Scheme) || !ports.Any(portNameToScheme)
+            ? ports.FirstOrDefault()
+            : ports.FirstOrDefault(portNameToScheme);
+        portV1 ??= new();
+        portV1.Name ??= configuration.Scheme ?? string.Empty;
+        _logger.LogDebug(() => $"K8s service with key '{configuration.KeyOfServiceInK8s}' and address {address.Ip}; Detected port is {portV1.Name}:{portV1.Port}. Total {ports.Count} ports of [{string.Join(',', ports.Select(p => p.Name))}].");
+        return new ServiceHostAndPort(address.Ip, portV1.Port, portV1.Name);
+    }
+
+    protected virtual string GetServiceId(KubeRegistryConfiguration configuration, EndpointsV1 endpoint, EndpointSubsetV1 subset, EndpointAddressV1 address)
+        => endpoint.Metadata?.Uid;
+    protected virtual string GetServiceVersion(KubeRegistryConfiguration configuration, EndpointsV1 endpoint, EndpointSubsetV1 subset, EndpointAddressV1 address)
+        => endpoint.ApiVersion;
+    protected virtual IEnumerable<string> GetServiceTags(KubeRegistryConfiguration configuration, EndpointsV1 endpoint, EndpointSubsetV1 subset, EndpointAddressV1 address)
+        => Enumerable.Empty<string>();
+}

src/Ocelot.Provider.Kubernetes/KubernetesProviderFactory.cs

@@ -1,6 +1,7 @@
 using Microsoft.Extensions.DependencyInjection;
 using Ocelot.Configuration;
-using Ocelot.Logging;
+using Ocelot.Logging;
+using Ocelot.Provider.Kubernetes.Interfaces;
 
 namespace Ocelot.Provider.Kubernetes
 {
@@ -17,14 +18,16 @@ private static IServiceDiscoveryProvider CreateProvider(IServiceProvider provide
         {
             var factory = provider.GetService<IOcelotLoggerFactory>();
             var kubeClient = provider.GetService<IKubeApiClient>();
+            var serviceBuilder = provider.GetService<IKubeServiceBuilder>();
 
             var configuration = new KubeRegistryConfiguration
             {
                 KeyOfServiceInK8s = route.ServiceName,
                 KubeNamespace = string.IsNullOrEmpty(route.ServiceNamespace) ? config.Namespace : route.ServiceNamespace,
+                Scheme = route.DownstreamScheme,
             };
 
-            var defaultK8sProvider = new Kube(configuration, factory, kubeClient);
+            var defaultK8sProvider = new Kube(configuration, factory, kubeClient, serviceBuilder);
 
             return PollKube.Equals(config.Type, StringComparison.OrdinalIgnoreCase)
                 ? new PollKube(config.PollingInterval, factory, defaultK8sProvider)

src/Ocelot.Provider.Kubernetes/OcelotBuilderExtensions.cs

@@ -1,16 +1,18 @@
 using Microsoft.Extensions.DependencyInjection;
 using Ocelot.DependencyInjection;
+using Ocelot.Provider.Kubernetes.Interfaces;
 
-namespace Ocelot.Provider.Kubernetes
+namespace Ocelot.Provider.Kubernetes;
+
+public static class OcelotBuilderExtensions
 {
-    public static class OcelotBuilderExtensions
+    public static IOcelotBuilder AddKubernetes(this IOcelotBuilder builder, bool usePodServiceAccount = true)
     {
-        public static IOcelotBuilder AddKubernetes(this IOcelotBuilder builder, bool usePodServiceAccount = true)
-        {
-            builder.Services
-                .AddSingleton(KubernetesProviderFactory.Get)
-                .AddKubeClient(usePodServiceAccount);
-            return builder;
-        }
+        builder.Services
+            .AddKubeClient(usePodServiceAccount)
+            .AddSingleton(KubernetesProviderFactory.Get)
+            .AddSingleton<IKubeServiceBuilder, KubeServiceBuilder>()
+            .AddSingleton<IKubeServiceCreator, KubeServiceCreator>();
+        return builder;
     }
 }

src/Ocelot/ServiceDiscovery/ServiceDiscoveryFinderDelegate.cs

@@ -1,7 +1,6 @@
 using Ocelot.Configuration;
 using Ocelot.ServiceDiscovery.Providers;
 
-namespace Ocelot.ServiceDiscovery
-{
-    public delegate IServiceDiscoveryProvider ServiceDiscoveryFinderDelegate(IServiceProvider provider, ServiceProviderConfiguration config, DownstreamRoute route);
-}
+namespace Ocelot.ServiceDiscovery;
+
+public delegate IServiceDiscoveryProvider ServiceDiscoveryFinderDelegate(IServiceProvider provider, ServiceProviderConfiguration config, DownstreamRoute route);

test/Ocelot.AcceptanceTests/Ocelot.AcceptanceTests.csproj

@@ -30,13 +30,14 @@
     </None>
   </ItemGroup>
   <ItemGroup>
-    <ProjectReference Include="..\..\src\Ocelot.Tracing.Butterfly\Ocelot.Tracing.Butterfly.csproj" />
-    <ProjectReference Include="..\..\src\Ocelot.Tracing.OpenTracing\Ocelot.Tracing.OpenTracing.csproj" />
     <ProjectReference Include="..\..\src\Ocelot\Ocelot.csproj" />
     <ProjectReference Include="..\..\src\Ocelot.Cache.CacheManager\Ocelot.Cache.CacheManager.csproj" />
     <ProjectReference Include="..\..\src\Ocelot.Provider.Consul\Ocelot.Provider.Consul.csproj" />
     <ProjectReference Include="..\..\src\Ocelot.Provider.Eureka\Ocelot.Provider.Eureka.csproj" />
+    <ProjectReference Include="..\..\src\Ocelot.Provider.Kubernetes\Ocelot.Provider.Kubernetes.csproj" />
     <ProjectReference Include="..\..\src\Ocelot.Provider.Polly\Ocelot.Provider.Polly.csproj" />
+    <ProjectReference Include="..\..\src\Ocelot.Tracing.Butterfly\Ocelot.Tracing.Butterfly.csproj" />
+    <ProjectReference Include="..\..\src\Ocelot.Tracing.OpenTracing\Ocelot.Tracing.OpenTracing.csproj" />
     <ProjectReference Include="..\Ocelot.ManualTest\Ocelot.ManualTest.csproj" />
     <ProjectReference Include="..\Ocelot.Testing\Ocelot.Testing.csproj" />
   </ItemGroup>