diff --git a/docs/bgp-policy.md b/docs/bgp-policy.md new file mode 100644 index 00000000000..7eb8638d6c0 --- /dev/null +++ b/docs/bgp-policy.md @@ -0,0 +1,190 @@ +# BGPPolicy + +## Table of Contents + + +- [What is BGPPolicy?](#what-is-bgppolicy) +- [Prerequisites](#prerequisites) +- [The BGPPolicy resource](#the-bgppolicy-resource) + - [NodeSelector](#nodeselector) + - [LocalASN](#localasn) + - [ListenPort](#listenport) + - [Advertisements](#advertisements) + - [BGPPeers](#bgppeers) +- [BGP router ID](#bgp-router-id) +- [Usage examples](#usage-examples) + - [Combined Advertisements of Service, Pod, and Egress IPs](#combined-advertisements-of-service-pod-and-egress-ips) + - [Advertise Egress IPs to external BGP peers with more than one hops](#advertise-egress-ips-to-external-bgp-peers-with-more-than-one-hops) +- [Limitations](#limitations) + + +## What is BGPPolicy? + +`BGPPolicy` is a custom resource that allows users to initialize BGP process on selected Kubernetes Nodes and advertise +Service IPs, Pod IPs, and Egress IPs to remote BGP peers, facilitating the integration of Kubernetes cluster with external +BGP-enabled network. + +## Prerequisites + +BGPPolicy was introduced in Antrea v2.1 as an alpha feature. A feature gate, `BGPPolicy` must be enabled on antrea-agent +in the `antrea-config` ConfigMap for the feature to work, like the following: + +```yaml +kind: ConfigMap +apiVersion: v1 +metadata: + name: antrea-config + namespace: kube-system +data: + antrea-agent.conf: | + featureGates: + BGPPolicy: true +``` + +## The BGPPolicy resource + +A BGPPolicy in Kubernetes is a Custom Resource Definition (CRD) object. Like all CRD objects, you can POST a BGPPolicy +definition to the API server to create a new instance. + +The following manifest creates a BGPPolicy object. It can start BGP process with ASN `64512`, listening on port `179`, +on Nodes labeled with `bgp=enabled`. The process will advertise ClusterIPs, LoadBalancerIPs, and ExternalIPs to a BGP +peer at IP address `192.168.77.200`, which has ASN `65001` and listens on port `179`: + +```yaml +apiVersion: crd.antrea.io/v1alpha1 +kind: BGPPolicy +metadata: + name: example-bgp-policy +spec: + nodeSelector: + matchLabels: + bgp: enabled + localASN: 64512 + listenPort: 179 + advertisements: + service: + ipTypes: + - ClusterIP + - ExternalIP + - LoadBalancerIP + bgpPeers: + - address: 192.168.77.200 + asn: 65001 + port: 179 +``` + +### NodeSelector + +The `nodeSelector` field selects which Kubernetes Nodes the BGPPolicy applies to based on the Node labels. The field is +mandatory. + +**Note**: If multiple BGPPolicy objects select the same Node, the effective BGPPolicy will be chosen randomly. + +### LocalASN + +The `localASN` field defines the Autonomous System Number (ASN) that the local BGP process uses. The available private +ASN range is `64512-65535`. The field is mandatory. + +### ListenPort + +The `listenPort` field specifies the port on which the BGP process listens. The default value is 179. The valid port +range is `1-65535`. + +### Advertisements + +The `advertisements` field configures which IPs are advertised to BGP peers. + +- `service`: Specifies how to advertise Service IPs. The `ipTypes` field lists the types of Service IPs to be advertised, + which can be `ClusterIP`, `ExternalIP`, and `LoadBalancerIP`. +- `pod`: Specifies how to advertise Pod IPs. The Node IPAM Pod CIDRs will be advertised by setting `pod:{}`. +- `egress`: Specifies how to advertise Egress IPs. All Egress IPs will be advertised by setting `egress:{}`. + +### BGPPeers + +The `bgpPeers` field lists the BGP peers to which the advertisements are sent. + +- `address`: The IP address of the BGP peer. +- `asn`: The Autonomous System Number of the BGP peer. +- `port`: The port number on which the BGP peer listens. The default value is 179. +- `multihopTTL`: The Time To Live (TTL) value used in BGP packets sent to the BGP peer, with a range of 1 to 255. + The default value is 1. +- `gracefulRestartTimeSeconds`: Specifies how long the BGP peer waits for the BGP session to re-establish after a + restart before deleting stale routes, with a range of 1 to 3600 seconds. The default value is 120 seconds. + +## BGP router ID + +Only an IPv4 address can be used as the BGP Identifier (BGP router ID). For IPv4-only or dual-stack Kubernetes clusters, +the Node's IPv4 address is used as the BGP router ID, ensuring uniqueness. For IPv6-only Kubernetes clusters without a +Node IPv4 address, the router ID must be specified in the Node annotation `node.antrea.io/bgp-router-id`. If the +annotation is not present, an IPv4 address will be generated by hashing the Node name and updated to the annotation. +If the annotation is present, its value will be used as the BGP router ID. + +## Usage examples + +### Combined Advertisements of Service, Pod, and Egress IPs + +In this example, we will advertise ClusterIP and LoadBalancerIP types of Service IPs, Pod CIDRs, and Egress IPs from the +selected Nodes to multiple remote BGP peers. + +```yaml +apiVersion: crd.antrea.io/v1alpha1 +kind: BGPPolicy +metadata: + name: advertise-all-ips +spec: + nodeSelector: + matchLabels: + bgp: enabled + localASN: 64512 + listenPort: 179 + advertisements: + service: + ipTypes: + - ClusterIP + - LoadBalancerIP + pod: {} + egress: {} + bgpPeers: + - address: 192.168.77.200 + asn: 65001 + port: 179 + - address: 192.168.77.201 + asn: 65001 + port: 179 +``` + +### Advertise Egress IPs to external BGP peers with more than one hops + +In this example, we configure the BGPPolicy to advertise Egress IPs from selected Nodes to a remote BGP peer located +multiple hops away from the cluster. It's crucial to set the `multihopTTL` to a value equal to or greater than the +number of hops, allowing BGP packets to traverse multiple hops to reach the peer. + +```yaml +apiVersion: crd.antrea.io/v1alpha1 +kind: BGPPolicy +metadata: + name: advertise-all-egress-ips +spec: + nodeSelector: + matchLabels: + bgp: enabled + localASN: 64512 + listenPort: 179 + advertisements: + egress: {} + bgpPeers: + - address: 192.168.78.201 + asn: 65001 + port: 179 + multihopTTL: 2 +``` + +## Limitations + +- The routes received from remote BGP peers will not be installed. Therefore, you must ensure that the path from Pods + to the remote BGP network is properly configured and routable. This involves configuring your network infrastructure + to handle the routing of traffic between your Kubernetes cluster and the remote BGP network. +- Only Linux Nodes are supported. The feature has not been validated on Windows Nodes, though theoretically it can work + with Windows Nodes. +- Advanced BGP features such as BGP communities, route filtering, route reflection, confederations, and other BGP policy + mechanisms defined in BGP RFCs are not supported.