Skip to content

Commit

Permalink
NPEP: Iron out Egress Support API Design
Browse files Browse the repository at this point in the history 
Signed-off-by: Surya Seetharaman <[email protected]>
  • Loading branch information
@tssurya
tssurya committed Oct 16, 2023
1 parent bb2cafa commit d6a0c93
Showing 1 changed file with 240 additions and 4 deletions.
244 changes: 240 additions & 4 deletions npep/npep-126-egress-traffic-control.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# NPEP-126: Add northbound traffic support in (B)ANP API

* Issue: [#126](https://github.com/kubernetes-sigs/network-policy-api/issues/126)
* Status: Provisional
* Status: Implementable

## TLDR

Expand Down Expand Up @@ -76,13 +76,249 @@ selected cluster workloads to k8s-apiservers for securing the server.

## API

(... details, can point to PR with changes)
Proof of Concept for the API design details can be found here: https://github.com/kubernetes-sigs/network-policy-api/pull/143

### Implementing egress traffic control towards cluster nodes

This NPEP proposes to add a new type of `AdminNetworkPolicyEgressPeer` called `Nodes`
to be able to explicitly select nodes (based on the node's labels) in the cluster.
This ensures that if the list of IPs on a node OR list of nodes change, the users
don't need to manually intervene to include those new IPs. The label selectors will
take care of this automatically.

```
// AdminNetworkPolicyEgressPeer defines a peer to allow traffic to.
// Exactly one of the selector pointers must be set for a given peer. If a
// consumer observes none of its fields are set, they must assume an unknown
// option has been specified and fail closed.
// +kubebuilder:validation:MaxProperties=1
// +kubebuilder:validation:MinProperties=1
type AdminNetworkPolicyEgressPeer struct {
<snipped>
// Nodes defines a way to select a set of nodes in
// in the cluster. This field follows standard label selector
// semantics; if present but empty, it selects all Nodes.
//
// Support: Core
//
// +optional
Nodes *metav1.LabelSelector `json:"nodes,omitempty"`
}
```

Note that `AdminNetworkPolicyPeer` will changed to
`AdminNetworkPolicyEgressPeer` and `AdminNetworkPolicyIngressPeer` since ingress and
egress peers have started to diverge at this point and it is easy to
maintain it with two sets of peer definitions.
This ensures nodes can be referred to only as "egress peers".

Example: Admin wants to deny egress traffic from tenants who don't have
`restricted`, `confidential` or `internal` level security clearance
to control-plane nodes at 443 and 6443 ports in the cluster

```
apiVersion: policy.networking.k8s.io/v1alpha1
kind: AdminNetworkPolicy
metadata:
name: node-as-egress-peers
spec:
priority: 55
subject:
namespaces:
matchExpressions:
- {key: security, operator: notIn, values: [restricted, confidential, internal]}
egress:
- name: "deny-all-egress-to-kapi-server"
action: "Deny"
to:
- nodes:
matchLabels:
node-role.kubernetes.io/control-plane: true
ports:
- portNumber:
protocol: TCP
port: 443
- portNumber:
protocol: TCP
port: 6443
```

### Implementing egress traffic control towards external destinations

This NPEP proposes to add a new type of `AdminNetworkPolicyEgressPeer` called `ExternalNetworks`
to be able to explicitly select external destinations (based on the `externalNetworkSet`'s
labels) in the cluster. This peer type will not be supported in `AdminNetworkPolicyIngressPeer`.

```
// AdminNetworkPolicyEgressPeer defines a peer to allow traffic to.
// Exactly one of the selector pointers must be set for a given peer. If a
// consumer observes none of its fields are set, they must assume an unknown
// option has been specified and fail closed.
// +kubebuilder:validation:MaxProperties=1
// +kubebuilder:validation:MinProperties=1
type AdminNetworkPolicyEgressPeer struct {
<snipped>
// ExternalNetworks defines a way to select ExternalNetworkSets
// that consist of network CIDRs that live outside the cluster as a peer.
// This field follows standard label selector semantics; if present
// but empty, it selects all ExternalNetworkSets defined in the cluster.
//
// Support: Core
//
// +optional
ExternalNetworks *metav1.LabelSelector `json:"externalNetworks,omitempty"`
}
```

An `externalNetworkSet` is a new object used to define a set of networks outside
the cluster. This is defined as a new object so that its easy for users to group
their external entities and then refer to that group from the different egress rules.
This ensures consistency without the users having to manually change the ANP/BANP
to accommodate the changing needs of the network cidrs. They can directly edit the
new CRD which guarantees all rules effecting that peer will take effect automatically.
In the future when we define `DeveloperNetworkPolicies`, this will come even more handy
for uniformity specially across `Pass` actions.

```
// ExternalNetworkSet is a cluster level resource that is used to define
// a set of networks outsides the cluster which can be referred to from
// the AdminNetworkPolicy && BaselineAdminNetworkPolicy APIs as an external peer
type ExternalNetworkSet struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata"`
// Specification of the desired behavior of ExternalNetworkSet.
Spec ExternalNetworkSetSpec `json:"spec"`
}
// ExternalNetworkSetSpec defines the desired state of ExternalNetworkSet.
type ExternalNetworkSetSpec struct {
// Networks is the list of NetworkCIDR (both v4 & v6) that can be used to define
// external destinations.
// A total of 100 CIDRs will be allowed in each NetworkSet instance.
// ANP & BANP APIs may use the .spec.in(e)gress.from(to).externalNetworks selector
// to select a set of external networks
// +optional
// +kubebuilder:validation:MaxItems=100
Networks []string `json:"networks,omitempty" validate:"omitempty,dive,cidr"`
}
```

Example: Let's define some sample sets of externalNetworks.

```
apiVersion: policy.networking.k8s.io/v1alpha1
kind: ExternalNetworkSet
metadata:
name: api-vm-network
labels:
network: intranet
spec:
networks:
- 10.0.0.10/32
- 10.0.0.11/32
- 10.0.0.12/32
---
apiVersion: policy.networking.k8s.io/v1alpha1
kind: ExternalNetworkSet
metadata:
name: storage-network
labels:
network: intranet
spec:
networks:
- 95.90.241.22/28
- 200.0.65.0/24
---
apiVersion: policy.networking.k8s.io/v1alpha1
kind: ExternalNetworkSet
metadata:
name: dns-network
labels:
network: dns
spec:
networks:
- 195.90.241.22/28
- 111.0.65.0/29
---
apiVersion: policy.networking.k8s.io/v1alpha1
kind: ExternalNetworkSet
metadata:
name: global-network
labels:
network: internet
spec:
networks:
- 0.0.0.0/0
```
```
$ oc get ens
NAME NETWORKS AGE
api-vm-network ["10.0.0.10/32","10.0.0.11/32","10.0.0.12/32"] 2s
dns-network ["195.90.241.22/28","111.0.65.0/29"] 2s
global-network ["0.0.0.0/0"] 2s
storage-network ["95.90.241.22/28","200.0.65.0/24"] 2s
```

Let's define ANP and BANP that refer to these external networks:
```
apiVersion: policy.networking.k8s.io/v1alpha1
kind: AdminNetworkPolicy
metadata:
name: cluster-egress-controls
spec:
priority: 70
subject:
namespaces: {}
egress:
- name: "allow-all-egress-to-intranet"
action: "Allow"
to:
- externalNetworks:
matchLabels:
network: intranet
- name: "allow-egress-to-external-dns"
action: "Allow"
to:
- externalNetworks:
matchLabels:
network: dns
ports:
- portNumber:
protocol: UDP
port: 53
- name: "pass-egress-to-internet"
action: "Pass"
to:
- externalNetworks:
matchLabels:
network: internet
---
apiVersion: policy.networking.k8s.io/v1alpha1
kind: BaselineAdminNetworkPolicy
metadata:
name: default
spec:
subject:
namespaces: {}
egress:
- name: "deny-egress-to-internet"
action: "Deny"
to:
- externalNetworks:
matchLabels:
network: internet
```
This allows admins to specify all cluster workloads can talk to
the company's internet CIDR's and the external DNS network. It
also allows them to put up default guardrails of not allowing
any other egress traffic towards internet in a BANP.

## Alternatives

(List other design alternatives and why we did not go in that
direction)
* Instead of defining a new object called `ExternalNetworkSet`,
we can go ahead with putting the CIDR ranges directly into the
main ANP/BANP objects.

## References

Expand Down

0 comments on commit d6a0c93

Please sign in to comment.