From 6ec4a80d4200bb0ff6db53e143b1e05ff0404c4b Mon Sep 17 00:00:00 2001 From: Hongliang Liu Date: Fri, 12 Jul 2024 18:04:04 +0800 Subject: [PATCH] Add document for BGPPolicy Signed-off-by: Hongliang Liu --- docs/bgp-policy.md | 221 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 221 insertions(+) create mode 100644 docs/bgp-policy.md diff --git a/docs/bgp-policy.md b/docs/bgp-policy.md new file mode 100644 index 00000000000..34665537450 --- /dev/null +++ b/docs/bgp-policy.md @@ -0,0 +1,221 @@ +# BGPPolicy + +## Table of Contents + + +- [What is BGPPolicy?](#what-is-bgppolicy) +- [Prerequisites](#prerequisites) +- [The BGPPolicy resource](#the-bgppolicy-resource) + - [NodeSelector](#nodeselector) + - [LocalASN](#localasn) + - [ListenPort](#listenport) + - [Advertisements](#advertisements) + - [BGPPeers](#bgppeers) +- [BGP router ID](#bgp-router-id) +- [Example Usage](#example-usage) + - [Combined Advertisements of Service, Pod, and Egress IPs](#combined-advertisements-of-service-pod-and-egress-ips) + - [Advertise Egress IPs to external BGP peers with more than one hop](#advertise-egress-ips-to-external-bgp-peers-with-more-than-one-hop) +- [Limitations](#limitations) + + +## What is BGPPolicy? + +`BGPPolicy` is a custom resource that allows users to run a BGP process on selected Kubernetes Nodes and advertise +Service IPs, Pod IPs, and Egress IPs to remote BGP peers, facilitating the integration of Kubernetes workloads with an +external BGP-enabled network. + +## Prerequisites + +BGPPolicy was introduced in Antrea v2.1 as an alpha feature. A feature gate, `BGPPolicy`, must be enabled on antrea-agent +in the `antrea-config` ConfigMap for the feature to work, like the following: + +```yaml +kind: ConfigMap +apiVersion: v1 +metadata: + name: antrea-config + namespace: kube-system +data: + antrea-agent.conf: | + featureGates: + BGPPolicy: true +``` + +## The BGPPolicy resource + +A BGPPolicy in Kubernetes is a Custom Resource Definition (CRD) object. + +The following manifest creates a BGPPolicy object. It will start a BGP process with ASN `64512`, listening on port `179`, +on Nodes labeled with `bgp=enabled`. The process will advertise ClusterIPs, LoadBalancerIPs, and ExternalIPs to a BGP +peer at IP address `192.168.77.200`, which has ASN `65001` and listens on port `179`: + +```yaml +apiVersion: crd.antrea.io/v1alpha1 +kind: BGPPolicy +metadata: + name: example-bgp-policy +spec: + nodeSelector: + matchLabels: + bgp: enabled + localASN: 64512 + listenPort: 179 + advertisements: + service: + ipTypes: [ClusterIP, ExternalIP, LoadBalancerIP] + bgpPeers: + - address: 192.168.77.200 + asn: 65001 + port: 179 +``` + +### NodeSelector + +The `nodeSelector` field selects which Kubernetes Nodes the BGPPolicy applies to based on the Node labels. The field is +mandatory. + +**Note**: If multiple BGPPolicy objects select the same Node, the one with the earliest creation time will be chosen +as the effective BGPPolicy. + +### LocalASN + +The `localASN` field defines the Autonomous System Number (ASN) that the local BGP process uses. The available private +ASN range is `64512-65535`. The field is mandatory. + +### ListenPort + +The `listenPort` field specifies the port on which the BGP process listens. The default value is 179. The valid port +range is `1-65535`. + +### Advertisements + +The `advertisements` field configures which IPs are advertised to BGP peers. + +- `pod`: Specifies how to advertise Pod IPs. The Node IPAM Pod CIDRs will be advertised by setting `pod:{}`. Note that + IPs allocated by Antrea Flexible IPAM are not yet supported.. +- `egress`: Specifies how to advertise Egress IPs. All Egress IPs will be advertised by setting `egress:{}`. A Node will + only advertise Egress IPs which are local (i.e., assigned to the Node). +- `service`: Specifies how to advertise Service IPs. The `ipTypes` field lists the types of Service IPs to be advertised, + which can include `ClusterIP`, `ExternalIP`, and `LoadBalancerIP`. + - All Nodes can advertise all ClusterIPs, respecting `internalTrafficPolicy`. If `internalTrafficPolicy` is set to + `Local`, a Node will only advertise ClusterIPs with at least one local Endpoint. + - All Nodes can advertise all ExternalIPs and LoadBalancerIPs, respecting `externalTrafficPolicy`. If + `externalTrafficPolicy` is set to `Local`, a Node will only advertise IPs with at least one local Endpoint. + +### BGPPeers + +The `bgpPeers` field lists the BGP peers to which the advertisements are sent. + +- `address`: The IP address of the BGP peer. +- `asn`: The Autonomous System Number of the BGP peer. +- `port`: The port number on which the BGP peer listens. The default value is 179. +- `multihopTTL`: The Time To Live (TTL) value used in BGP packets sent to the BGP peer, with a range of 1 to 255. + The default value is 1. +- `gracefulRestartTimeSeconds`: Specifies how long the BGP peer waits for the BGP session to re-establish after a + restart before deleting stale routes, with a range of 1 to 3600 seconds. The default value is 120 seconds. + +## BGP router ID + +The BGP router identifier (ID) is a 4-byte field that is usually represented as an IPv4 address. Antrea uses the following +steps to choose the BGP router ID: + +1. If the `node.antrea.io/bgp-router-id` annotation is present on the Node and its value is a valid IPv4 address string, + we will use the provided value. +2. Otherwise, for an IPv4-only or dual-stack Kubernetes cluster, the Node's IPv4 address (assigned to the transport + interface) is used. +3. Otherwise, for IPv6-only clusters, a 32-bit integer will be generated by hashing the Node name, then converted to the + string representation of an IPv4 address. + +After this selection process, the `node.antrea.io/bgp-router-id` annotation is added / updated as necessary to reflect +the selected BGP router ID. + +## BGP Authentication + +BGP authentication ensures that BGP sessions are established and maintained only with legitimate peers. Antrea supports +BGP authentication by leveraging a Kubernetes Secret, `antrea-bgp-passwords`, to store authentication passwords. + +By default, this Secret is not created, and BGP authentication is considered unconfigured for all BGP peers. If the +Secret is created like the following example, each entry should be a key that is a concatenated string of the BGP peer +IP address and ASN (e.g., `192.168.77.100-65000`, `2001:db8::1-65000` in the following example), with the value being +the password for that BGP peer. If a key for a BGP peer does not exist in the Secret, the peer is considered unconfigured +for that one. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: antrea-bgp-passwords + namespace: kube-system +stringData: + 192.168.77.100-65000: "password" + 2001:db8::1-65000: "password" +type: Opaque +``` + +## Example Usage + +### Combined Advertisements of Service, Pod, and Egress IPs + +In this example, we will advertise Service IPs of types ClusterIP and LoadBalancerIP, along with Pod CIDRs and Egress +IPs from the selected Nodes to multiple remote BGP peers. + +```yaml +apiVersion: crd.antrea.io/v1alpha1 +kind: BGPPolicy +metadata: + name: advertise-all-ips +spec: + nodeSelector: + matchLabels: + bgp: enabled + localASN: 64512 + listenPort: 179 + advertisements: + service: + ipTypes: [ClusterIP, LoadBalancerIP] + pod: {} + egress: {} + bgpPeers: + - address: 192.168.77.200 + asn: 65001 + port: 179 + - address: 192.168.77.201 + asn: 65001 + port: 179 +``` + +### Advertise Egress IPs to external BGP peers with more than one hop + +In this example, we configure the BGPPolicy to advertise Egress IPs from selected Nodes to a remote BGP peer located +multiple hops away from the cluster. It's crucial to set the `multihopTTL` to a value equal to or greater than the +number of hops, allowing BGP packets to traverse multiple hops to reach the peer. + +```yaml +apiVersion: crd.antrea.io/v1alpha1 +kind: BGPPolicy +metadata: + name: advertise-all-egress-ips +spec: + nodeSelector: + matchLabels: + bgp: enabled + localASN: 64512 + listenPort: 179 + advertisements: + egress: {} + bgpPeers: + - address: 192.168.78.201 + asn: 65001 + port: 179 + multihopTTL: 2 +``` + +## Limitations + +- The routes received from remote BGP peers will not be installed. Therefore, you must ensure that the path from Nodes + to the remote BGP network is properly configured and routable. This involves configuring your network infrastructure + to handle the routing of traffic between your Kubernetes cluster and the remote BGP network. +- Only Linux Nodes are supported. The feature has not been validated on Windows Nodes, though theoretically it can work + with Windows Nodes. +- Advanced BGP features such as BGP communities, route filtering, route reflection, confederations, and other BGP policy + mechanisms defined in BGP RFCs are not supported.