Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(Policy Assistant): support LoadBalancer and NodePort services in "generate" conformance tests #287

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

huntergregory
Copy link
Contributor

@huntergregory huntergregory commented Feb 12, 2025

Overview

Today in Policy Assistant and Cyclonus, the "generate" conformance tests only test k8s services with type=ClusterIP.

This PR adds 18 test cases in total for traffic to LoadBalancer and NodePort services (for both externalTrafficPolicy values).

Test Cases

For each service type/externalTrafficPolicy, testing 1) default deny ingress and 2) allowing ingress (via Pods and node CIDR if needed). Similar for egress.

LoadBalancer Test Cases

Tests send traffic from each Pod to each LoadBalancer service's external IP.

The source Pod's node is expected to "intercept" this traffic and redirect to the one backend Pod (which may be on another node).

In the case of externalTrafficPolicy =Local, the backend Pod must be on the source node. Therefore, tests ignore inter-node probes.

In the case of externalTrafficPolicy =Cluster, if the backend Pod is on another node, the CNI will SNAT traffic to the node IP. For some NetworkPolicy implementations, the source Pod info is lost in this case; to allow traffic requires a policy that allows traffic from the source node.

NodePort Test Cases

Tests send traffic from each Pod to each NodePort service's node port.

In the case of externalTrafficPolicy =Local, tests send traffic to the destination Pod's node.

In the case of externalTrafficPolicy =Cluster, tests send traffic to the source Pod's node (since this externalTrafficPolicy supports redirecting traffic to another node).

Verifying this change

New Test Cases

Verified traffic goes to correct services and that tests pass for two NetworkPolicy implementations.

There are 18 test cases. For Azure NPM, 2 must be ignored (see command below). For Cilium, 3 cases should be ignored since Cilium doesn't support ipBlock by default.

Old Test Cases

Verified that nothing seems to break. Only ClusterIP services are created.

Example runs for two NetworkPolicy implementations

Cilium:

policy-assistant generate --pod-creation-timeout-seconds 600 --server-protocol TCP,UDP --ignore-loopback --include special-services --exclude ip-block-no-except

Azure NPM:

policy-assistant generate --pod-creation-timeout-seconds 600 --server-protocol TCP,UDP --ignore-loopback --include special-services --exclude cni-brings-source-pod-info-to-other-node

CLI Outputs

All 18 test cases (none are ignored here):

test #1: LoadBalancer with externalTrafficPolicy=Cluster: deny all ingress
 - tags: deny-all, direction, external-traffic-policy-cluster, ingress, loadbalancer, rule, special-services
test #2: LoadBalancer with externalTrafficPolicy=Cluster: allow ingress from pods
 - tags: all-namespaces, cni-brings-source-pod-info-to-other-node, direction, external-traffic-policy-cluster, ingress, loadbalancer, peer-pods, special-services
test #3: LoadBalancer with externalTrafficPolicy=Cluster: allow ingress from pods and nodes
 - tags: all-namespaces, direction, external-traffic-policy-cluster, ingress, ip-block-no-except, loadbalancer, no-cni-source-pod-info-to-other-node, peer-ipblock, peer-pods, special-services
test #4: LoadBalancer with externalTrafficPolicy=Cluster: deny all egress
 - tags: deny-all, direction, egress, external-traffic-policy-cluster, loadbalancer, rule, special-services
test #5: LoadBalancer with externalTrafficPolicy=Cluster: allow egress to pods
 - tags: all-namespaces, direction, egress, external-traffic-policy-cluster, loadbalancer, peer-pods, special-services
test #6: NodePort with externalTrafficPolicy=Cluster: deny all ingress (to source pod's node)
 - tags: deny-all, direction, external-traffic-policy-cluster, ingress, nodeport, rule, special-services, to-source-pod-node
test #7: NodePort with externalTrafficPolicy=Cluster: allow ingress from pods (to source pod's node)
 - tags: all-namespaces, cni-brings-source-pod-info-to-other-node, direction, external-traffic-policy-cluster, ingress, nodeport, peer-pods, special-services, to-source-pod-node
test #8: NodePort with externalTrafficPolicy=Cluster: allow ingress from pods and nodes (to source pod's node)
 - tags: all-namespaces, direction, external-traffic-policy-cluster, ingress, ip-block-no-except, no-cni-source-pod-info-to-other-node, nodeport, peer-ipblock, peer-pods, special-services, to-source-pod-node
test #9: NodePort with externalTrafficPolicy=Cluster: deny all egress (to source pod's node)
 - tags: deny-all, direction, egress, external-traffic-policy-cluster, nodeport, rule, special-services, to-source-pod-node
test #10: NodePort with externalTrafficPolicy=Cluster: allow egress to pods (to source pod's node)
 - tags: all-namespaces, direction, egress, external-traffic-policy-cluster, nodeport, peer-pods, special-services, to-source-pod-node
test #11: LoadBalancer with externalTrafficPolicy=Local: deny all ingress
 - tags: deny-all, direction, external-traffic-policy-local, ingress, loadbalancer, rule, special-services
test #12: LoadBalancer with externalTrafficPolicy=Local: allow ingress from pods
 - tags: all-namespaces, direction, external-traffic-policy-local, ingress, loadbalancer, peer-pods, special-services
test #13: LoadBalancer with externalTrafficPolicy=Local: deny all egress
 - tags: deny-all, direction, egress, external-traffic-policy-local, loadbalancer, rule, special-services
test #14: LoadBalancer with externalTrafficPolicy=Local: allow egress to pods
 - tags: all-namespaces, direction, egress, external-traffic-policy-local, loadbalancer, peer-pods, special-services
test #15: NodePort with externalTrafficPolicy=Local: deny all ingress (to destination pod's node)
 - tags: deny-all, direction, external-traffic-policy-local, ingress, nodeport, rule, special-services, to-destination-pod-node
test #16: NodePort with externalTrafficPolicy=Local: allow ingress from pods (to destination pod's node)
 - tags: all-namespaces, direction, external-traffic-policy-local, ingress, nodeport, peer-pods, special-services, to-destination-pod-node
test #17: NodePort with externalTrafficPolicy=Local: deny all egress (to destination pod's node)
 - tags: deny-all, direction, egress, external-traffic-policy-local, nodeport, rule, special-services, to-destination-pod-node
test #18: NodePort with externalTrafficPolicy=Local: allow egress to pods and nodes (to destination pod's node)
 - tags: all-namespaces, direction, egress, external-traffic-policy-local, ip-block-no-except, nodeport, peer-ipblock, peer-pods, special-services, to-destination-pod-node

Result of Tests

Example run using the Azure NPM cluster:

$ policy-assistant generate --pod-creation-timeout-seconds 600 --server-protocol TCP,UDP --ignore-loopback --include special-services --exclude cni-brings-source-pod-info-to-other-node
...
finished policy #16
SummaryTable:
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|              TEST              | RESULT |   STEP/TRY    | WRONG | RIGHT | IGNORED |       TCP        | SCTP |       UDP        |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 1: LoadBalancer with           | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Cluster: |        |               |       |       |         |                  |      |                  |
| deny all ingress               |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 2: LoadBalancer with           | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Cluster: |        |               |       |       |         |                  |      |                  |
| allow ingress from pods and    |        |               |       |       |         |                  |      |                  |
| nodes                          |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 3: LoadBalancer with           | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Cluster: |        |               |       |       |         |                  |      |                  |
| deny all egress                |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 4: LoadBalancer with           | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Cluster: |        |               |       |       |         |                  |      |                  |
| allow egress to pods           |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 5: NodePort with               | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Cluster: |        |               |       |       |         |                  |      |                  |
| deny all ingress (to source    |        |               |       |       |         |                  |      |                  |
| pod's node)                    |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 6: NodePort with               | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Cluster: |        |               |       |       |         |                  |      |                  |
| allow ingress from pods and    |        |               |       |       |         |                  |      |                  |
| nodes (to source pod's node)   |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 7: NodePort with               | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Cluster: |        |               |       |       |         |                  |      |                  |
| deny all egress (to source     |        |               |       |       |         |                  |      |                  |
| pod's node)                    |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 8: NodePort with               | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Cluster: |        |               |       |       |         |                  |      |                  |
| allow egress to pods (to       |        |               |       |       |         |                  |      |                  |
| source pod's node)             |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 9: LoadBalancer with           | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Local:   |        |               |       |       |         |                  |      |                  |
| deny all ingress               |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    32 |      49 | 64 / 64 (100%)   | -    | 64 / 64 (100%)   |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 10: LoadBalancer with          | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Local:   |        |               |       |       |         |                  |      |                  |
| allow ingress from pods        |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    32 |      49 | 64 / 64 (100%)   | -    | 64 / 64 (100%)   |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 11: LoadBalancer with          | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Local:   |        |               |       |       |         |                  |      |                  |
| deny all egress                |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    32 |      49 | 64 / 64 (100%)   | -    | 64 / 64 (100%)   |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 12: LoadBalancer with          | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Local:   |        |               |       |       |         |                  |      |                  |
| allow egress to pods           |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    32 |      49 | 64 / 64 (100%)   | -    | 64 / 64 (100%)   |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 13: NodePort with              | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Local:   |        |               |       |       |         |                  |      |                  |
| deny all ingress (to           |        |               |       |       |         |                  |      |                  |
| destination pod's node)        |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 14: NodePort with              | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Local:   |        |               |       |       |         |                  |      |                  |
| allow ingress from pods (to    |        |               |       |       |         |                  |      |                  |
| destination pod's node)        |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 15: NodePort with              | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Local:   |        |               |       |       |         |                  |      |                  |
| deny all egress (to            |        |               |       |       |         |                  |      |                  |
| destination pod's node)        |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 16: NodePort with              | passed |               |       |       |         |                  |      |                  |
| externalTrafficPolicy=Local:   |        |               |       |       |         |                  |      |                  |
| allow egress to pods and nodes |        |               |       |       |         |                  |      |                  |
| (to destination pod's node)    |        |               |       |       |         |                  |      |                  |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
|                                |        | Step 1, try 1 |     0 |    72 |       9 | 144 / 144 (100%) | -    | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+

rule counts:
+----------+--------+--------+----------+
| FEATURE  | PASSED | FAILED | PASSED % |
+----------+--------+--------+----------+
| deny-all |      8 |      0 |      100 |
+----------+--------+--------+----------+

special-services counts:
+--------------------------------------+--------+--------+----------+
|               FEATURE                | PASSED | FAILED | PASSED % |
+--------------------------------------+--------+--------+----------+
| to-source-pod-node                   |      4 |      0 |      100 |
| nodeport                             |      8 |      0 |      100 |
| external-traffic-policy-local        |      8 |      0 |      100 |
| to-destination-pod-node              |      4 |      0 |      100 |
| external-traffic-policy-cluster      |      8 |      0 |      100 |
| loadbalancer                         |      8 |      0 |      100 |
| no-cni-source-pod-info-to-other-node |      2 |      0 |      100 |
+--------------------------------------+--------+--------+----------+

direction counts:
+---------+--------+--------+----------+
| FEATURE | PASSED | FAILED | PASSED % |
+---------+--------+--------+----------+
| ingress |      8 |      0 |      100 |
| egress  |      8 |      0 |      100 |
+---------+--------+--------+----------+

peer-ipblock counts:
+--------------------+--------+--------+----------+
|      FEATURE       | PASSED | FAILED | PASSED % |
+--------------------+--------+--------+----------+
| ip-block-no-except |      3 |      0 |      100 |
+--------------------+--------+--------+----------+

peer-pods counts:
+----------------+--------+--------+----------+
|    FEATURE     | PASSED | FAILED | PASSED % |
+----------------+--------+--------+----------+
| all-namespaces |      8 |      0 |      100 |
+----------------+--------+--------+----------+

Pass/Fail for probes on protocols:
+---------------+--------+--------+----------+
|   PROTOCOL    | PASSED | FAILED | PASSED % |
+---------------+--------+--------+----------+
| probe on TCP  |   1984 |      0 |      100 |
| probe on SCTP |      0 |      0 |        0 |
| probe on UDP  |   1984 |      0 |      100 |
+---------------+--------+--------+----------+

Feature results:
| Tag | Result |
| --- | --- |
| action | 16 / 16 = 100% ✅ |
|  - action: create policy | 16 / 16 = 100% ✅ |
| egress | 16 / 16 = 100% ✅ |
|  - 0 port/protocols | 4 / 4 = 100% ✅ |
|  - 0 rules | 12 / 12 = 100% ✅ |
|  - 1 peer | 3 / 3 = 100% ✅ |
|  - 1 rule | 4 / 4 = 100% ✅ |
|  - 2+ peers | 1 / 1 = 100% ✅ |
|  - IPBlock (no except) | 1 / 1 = 100% ✅ |
|  - peer namespace selector empty | 4 / 4 = 100% ✅ |
|  - peer pod selector nil | 4 / 4 = 100% ✅ |
| general | 16 / 16 = 100% ✅ |
|  - policy with egress | 4 / 4 = 100% ✅ |
|  - policy with ingress | 4 / 4 = 100% ✅ |
|  - target: empty pod selector | 16 / 16 = 100% ✅ |
|  - target: specific namespace | 16 / 16 = 100% ✅ |
| ingress | 16 / 16 = 100% ✅ |
|  - 0 port/protocols | 4 / 4 = 100% ✅ |
|  - 0 rules | 12 / 12 = 100% ✅ |
|  - 1 peer | 2 / 2 = 100% ✅ |
|  - 1 rule | 4 / 4 = 100% ✅ |
|  - 2+ peers | 2 / 2 = 100% ✅ |
|  - IPBlock (no except) | 2 / 2 = 100% ✅ |
|  - peer namespace selector empty | 4 / 4 = 100% ✅ |
|  - peer pod selector nil | 4 / 4 = 100% ✅ |

Tag results:
| Tag | Result |
| --- | --- |
| direction | 16 / 16 = 100% ✅ |
|  - egress | 8 / 8 = 100% ✅ |
|  - ingress | 8 / 8 = 100% ✅ |
| peer-ipblock | 3 / 3 = 100% ✅ |
|  - ip-block-no-except | 3 / 3 = 100% ✅ |
| peer-pods | 8 / 8 = 100% ✅ |
|  - all-namespaces | 8 / 8 = 100% ✅ |
| rule | 8 / 8 = 100% ✅ |
|  - deny-all | 8 / 8 = 100% ✅ |
| special-services | 16 / 16 = 100% ✅ |
|  - external-traffic-policy-cluster | 8 / 8 = 100% ✅ |
|  - external-traffic-policy-local | 8 / 8 = 100% ✅ |
|  - loadbalancer | 8 / 8 = 100% ✅ |
|  - no-cni-source-pod-info-to-other-node | 2 / 2 = 100% ✅ |
|  - nodeport | 8 / 8 = 100% ✅ |
|  - to-destination-pod-node | 4 / 4 = 100% ✅ |
|  - to-source-pod-node | 4 / 4 = 100% ✅ |

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 12, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: huntergregory

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Feb 12, 2025
Copy link

netlify bot commented Feb 12, 2025

Deploy Preview for kubernetes-sigs-network-policy-api ready!

Name Link
🔨 Latest commit 0d0e470
🔍 Latest deploy log https://app.netlify.com/sites/kubernetes-sigs-network-policy-api/deploys/67ad1b567752a8000855f743
😎 Deploy Preview https://deploy-preview-287--kubernetes-sigs-network-policy-api.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants