diff --git a/config.yaml b/config.yaml index 211e4051..2041a5c2 100644 --- a/config.yaml +++ b/config.yaml @@ -36,9 +36,10 @@ params: docs_search_index_name: index_name docs_search_api_key: api_key docs_versioning: true - docs_latest: v1.14.2 + docs_latest: v1.15.0 docs_versions: - main + - v1.15.0 - v1.14.2 - v1.14.1 - v1.14.0 diff --git a/content/docs/v1.15.0/ADOPTERS.md b/content/docs/v1.15.0/ADOPTERS.md new file mode 100644 index 00000000..5b692f64 --- /dev/null +++ b/content/docs/v1.15.0/ADOPTERS.md @@ -0,0 +1,105 @@ +# Antrea Adopters + + +{{< img alt="glasnostic.com" src="docs/assets/adopters/glasnostic-logo.png"height="50" >}}      + + +{{< img alt="https://www.transwarp.io" src="docs/assets/adopters/transwarp-logo.png"height="50" >}}      + + +{{< img alt="https://www.terasky.com" src="docs/assets/adopters/terasky-logo.png"height="50" >}}      + +## Success Stories + +Below is a list of adopters of Antrea that have publicly shared the details +of how they use it. + +**[Glasnostic](https://glasnostic.com)** + +Glasnostic makes modern cloud operations resilient. It does this by shaping how +systems interact, automatically and in real-time. As a result, DevOps and SRE +teams can deploy reliably, prevent failure and assure the customer experience. +We use Antrea's Open vSwitch support to tune how services interact in Kubernetes +clusters. We are @glasnostic on Twitter. + +**[Transwarp](https://www.transwarp.io)** + +Transwarp is committed to building enterprise-level big data infrastructure +software, providing enterprises with infrastructure software and supporting +around the whole data lifecycle to build a data world of the future. + +1. We use Antrea's AntreaClusterNetworkPolicy and AntreaNetworkPolicy to protect +big data software for every tenant of our kubernetes platform. +2. We use Antrea's Open vSwitch to support Pod-To-Pod network between flannel and +antrea clusters, and also between antrea clusters +3. We use Antrea's Open vSwitch to support Pod-To-Pod network between flannel and +antrea nodes in one cluster for upgrading. +4. We use Antrea's Egress feature to keep the original source ip to ensure +Internal Pods can get the real source IP of the request. + +You can contact us with + +**[TeraSky](https://terasky.com)** + +TeraSky is a Global Advanced Technology Solutions Provider. +Antrea is used in our internal Kubernetes clusters as well as by many of our customers. +Antrea helps us to apply a very strong and flexible security models in Kubernetes. +We are very heavily utilizing Antrea Cluster Network Policies, Antrea Network Policies, +and the Egress functionality. + +We are @TeraSkycom1 on Twitter. + +## Adding yourself as an Adopter + +It would be great to have your success story and logo on our list of +Antrea adopters! + +To add yourself, you can follow the steps outlined below, alternatively, +feel free to reach out via Slack or on Github to have our team +add your success story and logo. + +1. Prepare your addition and PR as described in the Antrea +[Contributor Guide](CONTRIBUTING.md). + +2. Add your name to the success stories, using **bold** format with a link to +your web site like this: `**[Example](https://example.com)**` + +3. Below your name, describe your organization or yourself and how you make +use of Antrea. Optionally, list the features of Antrea you are using. Please +keep the line width at 80 characters maximum, and avoid trailing spaces. + +4. If you are willing to share contact details, e.g. your Twitter handle, etc. +add a line where people can find you. + + Example: + + ```markdown + **[Example](https://example.com)** + Example.com is a company operating internationally, focusing on creating + documentation examples. We are using Antrea in our K8s clusters deployed + using Kubeadm. We making use of Antrea's Network Policy capabilities. + You can reach us on twitter @vmwopensource. + ``` + +5. (Optional) To add your logo, simply drop your logo in PNG or SVG format with +a maximum size of 50KB to the [adopters](docs/assets/adopters) directory. +Name the image file something that reflects your company (e.g., if your company +is called Acme, name the image acme-logo.png). Then add an inline html link +directly bellow the [Antrea Adopters section](#antrea-adopters). Use the +following format: + + ```html + + {{< img alt="example.com" src="docs/assets/adopters/example-logo.png" height="50" >}}      + ``` + +6. Send a PR with your addition as described in the Antrea +[Contributor Guide](CONTRIBUTING.md) + +## Adding a logo to Antrea.io + +We are working on adding an *Adopters* section on [antrea.io][1]. +Follow the steps above to add your organization to the list of Antrea Adopters. +We will follow up and publish it to the [antrea.io][1] website. + +[1]: https://antrea.io diff --git a/content/docs/v1.15.0/CHANGELOG.md b/content/docs/v1.15.0/CHANGELOG.md new file mode 100644 index 00000000..1a4acaea --- /dev/null +++ b/content/docs/v1.15.0/CHANGELOG.md @@ -0,0 +1 @@ +Changelogs have been moved to the [CHANGELOG](https://github.com/antrea-io/antrea/blob/v1.15.0/CHANGELOG) directory. diff --git a/content/docs/v1.15.0/CODE_OF_CONDUCT.md b/content/docs/v1.15.0/CODE_OF_CONDUCT.md new file mode 100644 index 00000000..94d03ef9 --- /dev/null +++ b/content/docs/v1.15.0/CODE_OF_CONDUCT.md @@ -0,0 +1,3 @@ +# Community Code of Conduct + +Project Antrea follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md). diff --git a/content/docs/v1.15.0/CONTRIBUTING.md b/content/docs/v1.15.0/CONTRIBUTING.md new file mode 100644 index 00000000..57f9eed8 --- /dev/null +++ b/content/docs/v1.15.0/CONTRIBUTING.md @@ -0,0 +1,410 @@ +# Developer Guide + +Thank you for taking the time out to contribute to project Antrea! + +This guide will walk you through the process of making your first commit and how +to effectively get it merged upstream. + + +- [Getting Started](#getting-started) + - [Accounts Setup](#accounts-setup) +- [Contribute](#contribute) + - [Git Client Hooks](#git-client-hooks) + - [GitHub Workflow](#github-workflow) + - [Getting reviewers](#getting-reviewers) + - [Getting your PR verified by CI](#getting-your-pr-verified-by-ci) + - [Cherry-picks to release branches](#cherry-picks-to-release-branches) + - [Conventions for Writing Documentation](#conventions-for-writing-documentation) + - [Inclusive Naming](#inclusive-naming) + - [Building and testing your change](#building-and-testing-your-change) + - [Reverting a commit](#reverting-a-commit) + - [Sign-off Your Work](#sign-off-your-work) +- [Issue and PR Management](#issue-and-pr-management) + - [Filing An Issue](#filing-an-issue) + - [Issue Triage](#issue-triage) + - [Issue and PR Kinds](#issue-and-pr-kinds) + + +## Getting Started + +To get started, let's ensure you have completed the following prerequisites for +contributing to project Antrea: + +1. Read and observe the [code of conduct](CODE_OF_CONDUCT.md). +2. Check out the [Architecture document](docs/design/architecture.md) for the Antrea + architecture and design. +3. Set up necessary [accounts](#accounts-setup). + +Now that you're setup, skip ahead to learn how to [contribute](#contribute). + +### Accounts Setup + +At minimum, you need the following accounts for effective participation: + +1. **Github**: Committing any change requires you to have a [github + account](https://github.com/join). +2. **Slack**: Join the [Kubernetes Slack](http://slack.k8s.io/) and look for our + [#antrea](https://kubernetes.slack.com/messages/CR2J23M0X) channel. +3. **Google Group**: Join our [mailing list](https://groups.google.com/forum/#!forum/projectantrea-dev). + +## Contribute + +There are multiple ways in which you can contribute, either by contributing +code in the form of new features or bug-fixes or non-code contributions like +helping with code reviews, triaging of bugs, documentation updates, filing +[new issues](#filing-an-issue) or writing blogs/manuals etc. + +In order to help you get your hands "dirty", there is a list of +[starter](https://github.com/antrea-io/antrea/labels/Good%20first%20issue) +issues from which you can choose. + +### Git Client Hooks + + There are a few recommended git client hooks which we advise you to use. You can find + them here: + [hack/git_client_side_hooks](hack/git_client_side_hooks). + You can run `make install-hooks` to copy them to your local `.git/hooks/` folder, and remove them via `make uninstall-hooks` + +### GitHub Workflow + +Developers work in their own forked copy of the repository and when ready, +submit pull requests to have their changes considered and merged into the +project's repository. + +1. Fork your own copy of the repository to your GitHub account by clicking on + `Fork` button on [Antrea's GitHub repository](https://github.com/antrea-io/antrea). +2. Clone the forked repository on your local setup. + + ```bash + git clone https://github.com/$user/antrea + ``` + + Add a remote upstream to track upstream Antrea repository. + + ```bash + git remote add upstream https://github.com/antrea-io/antrea + ``` + + Never push to upstream remote + + ```bash + git remote set-url --push upstream no_push + ``` + +3. Create a topic branch. + + ```bash + git checkout -b branchName + ``` + +4. Make changes and commit it locally. Make sure that your commit is + [signed](#sign-off-your-work). + + ```bash + git add + git commit -s + ``` + +5. Keeping branch in sync with upstream. + + ```bash + git checkout branchName + git fetch upstream + git rebase upstream/main + ``` + +6. Push local branch to your forked repository. + + ```bash + git push -f $remoteBranchName branchName + ``` + +7. Create a Pull request on GitHub. + Visit your fork at `https://github.com/antrea-io/antrea` and click + `Compare & Pull Request` button next to your `remoteBranchName` branch. + +### Getting reviewers + +Once you have opened a Pull Request (PR), reviewers will be assigned to your +PR and they may provide review comments which you need to address. +Commit changes made in response to review comments to the same branch on your +fork. Once a PR is ready to merge, squash any *fix review feedback, typo* +and *merged* sorts of commits. + +To make it easier for reviewers to review your PR, consider the following: + +1. Follow the golang [coding conventions](https://github.com/golang/go/wiki/CodeReviewComments) + and check out this [document](https://github.com/tnqn/code-review-comments#code-review-comments) + for common comments we made during reviews and suggestions for fixing them. +2. Format your code with `make golangci-fix`; if the [linters](https://github.com/antrea-io/antrea/blob/v1.15.0/ci/README.md) flag an issue that + cannot be fixed automatically, an error message will be displayed so you can address the issue. +3. Follow [git commit](https://chris.beams.io/posts/git-commit/) guidelines. +4. Follow [logging](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md) guidelines. +5. Please refer to [Conventions for Writing Documentation](#conventions-for-writing-documentation) for +spelling conventions when writing documentation or commenting code. + +If your PR fixes a bug or implements a new feature, add the appropriate test +cases to our [automated test suite](https://github.com/antrea-io/antrea/blob/v1.15.0/ci/README.md) to guarantee enough +coverage. A PR that makes significant code changes without contributing new test +cases will be flagged by reviewers and will not be accepted. + +### Getting your PR verified by CI + +It is a requirement to get your PR verified with CI checks before it gets merged. +Also, it helps to find possible bugs before the review work starts. Once you create +a PR, or you push new commits, CI checks at the bottom of a PR page will be refreshed. +Checks include Github Action ones and Jenkins ones. Github Action ones will be +triggered automatically when you push to the head branch of the PR but Jenkins ones +need to be triggered manually with comments. Please note that if you are a first-time +contributor, the Github workflows need approval from someone with write access to +the repo. It's a Github security mechanism. + +Here are the trigger phrases for individual checks: + +* `/test-e2e`: Linux IPv4 e2e tests +* `/test-conformance`: Linux IPv4 conformance tests +* `/test-networkpolicy`: Linux IPv4 networkpolicy tests +* `/test-all-features-conformance`: Linux IPv4 conformance tests with all features enabled +* `/test-windows-e2e`: Windows IPv4 e2e tests +* `/test-windows-conformance`: Windows IPv4 conformance tests +* `/test-windows-networkpolicy`: Windows IPv4 networkpolicy tests +* `/test-ipv6-e2e`: Linux dual stack e2e tests +* `/test-ipv6-conformance`: Linux dual stack conformance tests +* `/test-ipv6-networkpolicy`: Linux dual stack networkpolicy tests +* `/test-ipv6-only-e2e`: Linux IPv6 only e2e tests +* `/test-ipv6-only-conformance`: Linux IPv6 only conformance tests +* `/test-ipv6-only-networkpolicy`: Linux IPv6 only networkpolicy tests +* `/test-flexible-ipam-e2e`: Flexible IPAM e2e tests +* `/test-multicast-e2e`: Multicast e2e tests +* `/test-multicluster-e2e`: Multicluster e2e tests +* `/test-vm-e2e`: ExternalNode e2e tests +* `/test-whole-conformance`: All conformance tests on Linux +* `/test-hw-offload`: Hardware offloading e2e tests +* `/test-rancher-e2e`: Linux IPv4 e2e tests on Rancher clusters. +* `/test-rancher-conformance`: Linux IPv4 conformance tests on Rancher clusters. +* `/test-rancher-networkpolicy`: Linux IPv4 networkpolicy tests on Rancher clusters. + +Here are the trigger phrases for groups of checks: + +* `/test-all`: Linux IPv4 tests +* `/test-windows-all`: Windows IPv4 tests, including e2e tests with proxyAll enabled. It also includes all containerd runtime based Windows tests since 1.10.0. +* `/test-ipv6-all`: Linux dual stack tests +* `/test-ipv6-only-all`: Linux IPv6 only tests + +Besides, you can skip a check with `/skip-*`, e.g. `/skip-e2e`: skip Linux IPv4 +e2e tests. + +Skipping a check should be used only when the change doesn't influence the +specific function. For example: + +* doc change: skip all checks +* comment change: skip all checks +* test/e2e/* change: skip conformance and networkpolicy checks +* *_windows.go change: skip Linux checks + +Besides skipping specific checks you can also cancel all stale running or waiting capv jenkins jobs related to your PR with +`/stop-all-jobs`. + +For more information about the tests we run as part of CI, please refer to +[ci/README.md](https://github.com/antrea-io/antrea/blob/v1.15.0/ci/README.md). + +### Cherry-picks to release branches + +If your PR fixes a critical bug, it may need to be backported to older release +branches which are still maintained. If this is the case, one of the Antrea +maintainers will let you know once your PR is approved. Please refer to the +documentation on [cherry-picks](docs/contributors/cherry-picks.md) for more +information. + +### Conventions for Writing Documentation + +* Short name of `IP Security` should be `IPsec` as per [rfc 6071](https://datatracker.ietf.org/doc/html/rfc6071). +* Any Kubernetes object in log/comment should start with upper case, eg: Namespace, Pod, Service. + +### Inclusive Naming + +For symbol names and documentation, do not introduce new usage of harmful +language such as 'master / slave' (or 'slave' independent of 'master') and +'blacklist / whitelist'. For more information about what constitutes harmful +language and for a reference word replacement list, please refer to the +[Inclusive Naming Initiative](https://inclusivenaming.org/). + +We are committed to removing all harmful language from the project. If you +detect existing usage of harmful language in code or documentation, please +report the issue to us or open a Pull Request to address it directly. Thanks! + +### Building and testing your change + +To build the Antrea Docker image together with all Antrea bits, you can simply +do: + +1. Checkout your feature branch and `cd` into it. +2. Run `make` + +The second step will compile the Antrea code in a `golang` container, and build +an Ubuntu-based Docker image that includes all the generated binaries. [`Docker`](https://docs.docker.com/install) +must be installed on your local machine in advance. If you are a macOS user and +cannot use [Docker Desktop](https://www.docker.com/products/docker-desktop) to +contribute to Antrea for licensing reasons, check out this +[document](docs/contributors/docker-desktop-alternatives.md) for possible +alternatives. + +Alternatively, you can build the Antrea code in your local Go environment. The +Antrea project uses the [Go modules support](https://github.com/golang/go/wiki/Modules) which was introduced in Go 1.11. It +facilitates dependency tracking and no longer requires projects to live inside +the `$GOPATH`. + +To develop locally, you can follow these steps: + + 1. [Install Go 1.21](https://golang.org/doc/install) + 2. Checkout your feature branch and `cd` into it. + 3. To build all Go files and install them under `bin`, run `make bin` + 4. To run all Go unit tests, run `make test-unit` + 5. To build the Antrea Ubuntu Docker image separately with the binaries generated in step 2, run `make ubuntu` + +### Reverting a commit + +1. Create a branch in your forked repo + + ```bash + git checkout -b revertName + ``` + +2. Sync the branch with upstream + + ```bash + git fetch upstream + git rebase upstream/main + ``` + +3. Create a revert based on the SHA of the commit. The commit needs to be + [signed](#sign-off-your-work). + + ```bash + git revert -s SHA + ``` + +4. Push this new commit. + + ```bash + git push $remoteRevertName revertName + ``` + +5. Create a Pull Request on GitHub. + Visit your fork at `https://github.com/antrea-io/antrea` and click + `Compare & Pull Request` button next to your `remoteRevertName` branch. + +### Sign-off Your Work + +As a CNCF project, Antrea must enforce the [Developer Certificate of +Origin](https://developercertificate.org/) (DCO) on all Pull Requests. We +require that for all commits constituting the Pull Request, the commit message +contains the `Signed-off-by` line with an email address that matches the commit +author. By adding this line to their commit messages, contributors *sign-off* +that they adhere to the requirements of the DCO. + +Git provides the `-s` command-line option to append the required line +automatically to the commit message: + +```bash +git commit -s -m 'This is my commit message' +``` + +For an existing commit, you can also use this option with `--amend`: + +```bash +git commit -s --amend +``` + +If more than one person works on something it's possible for more than one +person to sign-off on it. For example: + +```bash +Signed-off-by: Some Developer somedev@example.com +Signed-off-by: Another Developer anotherdev@example.com +``` + +We use the [DCO Github App](https://github.com/apps/dco) to enforce that all +commits in a Pull Request include the required `Signed-off-by` line. If this is +not the case, the app will report a failed status for the Pull Request and it +will be blocked from being merged. + +Compared to our earlier CLA, DCO tends to make the experience simpler for new +contributors. If you are contributing as an employee, there is no need for your +employer to sign anything; the DCO assumes you are authorized to submit +contributions (it's your responsibility to check with your employer). + +## Issue and PR Management + +We use labels and workflows (some manual, some automated with GitHub Actions) to +help us manage triage, prioritize, and track issue progress. For a detailed +discussion, see [docs/issue-management.md](docs/contributors/issue-management.md). + +### Filing An Issue + +Help is always appreciated. If you find something that needs fixing, please file +an issue [here](https://github.com/antrea-io/antrea/issues). Please ensure +that the issue is self explanatory and has enough information for an assignee to +get started. + +Before picking up a task, go through the existing +[issues](https://github.com/antrea-io/antrea/issues) and make sure that your +change is not already being worked on. If it does not exist, please create a new +issue and discuss it with other members. + +For simple contributions to Antrea, please ensure that this minimum set of +labels are included on your issue: + +* **kind** -- common ones are `kind/feature`, `kind/support`, `kind/bug`, + `kind/documentation`, or `kind/design`. For an overview of the different types + of issues that can be submitted, see [Issue and PR + Kinds](#issue-and-pr-kinds). + The kind of issue will determine the issue workflow. +* **area** (optional) -- if you know the area the issue belongs in, you can assign it. + Otherwise, another community member will label the issue during triage. The + area label will identify the area of interest an issue or PR belongs in and + will ensure the appropriate reviewers shepherd the issue or PR through to its + closure. For an overview of areas, see the + [`docs/github-labels.md`](docs/contributors/github-labels.md). +* **size** (optional) -- if you have an idea of the size (lines of code, + complexity, effort) of the issue, you can label it using a size label. The + size can be updated during backlog grooming by contributors. This estimate is + used to guide the number of features selected for a milestone. + +All other labels will be assigned during issue triage. + +### Issue Triage + +Once an issue has been submitted, the CI (GitHub actions) or a human will +automatically review the submitted issue or PR to ensure that it has all relevant +information. If information is lacking or there is another problem with the +submitted issue, an appropriate `triage/` label will be applied. + +After an issue has been triaged, the maintainers can prioritize the issue with +an appropriate `priority/` label. + +Once an issue has been submitted, categorized, triaged, and prioritized it +is marked as `ready-to-work`. A ready-to-work issue should have labels +indicating assigned areas, prioritization, and should not have any remaining +triage labels. + +### Issue and PR Kinds + +Use a `kind` label to describe the kind of issue or PR you are submitting. Valid +kinds include: + +* [`kind/api-change`](docs/contributors/issue-management.md#api-change) -- for api changes +* [`kind/bug`](docs/contributors/issue-management.md#bug) -- for filing a bug +* [`kind/cleanup`](docs/contributors/issue-management.md#cleanup) -- for code cleanup and organization +* [`kind/deprecation`](docs/contributors/issue-management.md#deprecation) -- for deprecating a feature +* [`kind/design`](docs/contributors/issue-management.md#design) -- for proposing a design or architectural change +* [`kind/documentation`](docs/contributors/issue-management.md#documentation) -- for updating documentation +* [`kind/failing-test`](docs/contributors/issue-management.md#failing-test) -- for reporting a failed test (may + create with automation in future) +* [`kind/feature`](docs/contributors/issue-management.md#feature) -- for proposing a feature +* [`kind/support`](docs/contributors/issue-management.md#support) -- to request support. You may also get support by + using our [Slack](https://kubernetes.slack.com/archives/CR2J23M0X) channel for + interactive help. If you have not set up the appropriate accounts, please + follow the instructions in [accounts setup](#accounts-setup). + +For more details on how we manage issues, please read our [Issue Management doc](docs/contributors/issue-management.md). diff --git a/content/docs/v1.15.0/GOVERNANCE.md b/content/docs/v1.15.0/GOVERNANCE.md new file mode 100644 index 00000000..a58ca425 --- /dev/null +++ b/content/docs/v1.15.0/GOVERNANCE.md @@ -0,0 +1,85 @@ +# Antrea Governance + +This document defines the project governance for Antrea. + +## Overview + +**Antrea** is committed to building an open, inclusive, productive and +self-governing open source community focused on building a high-quality +[Kubernetes Network +Plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/). The +community is governed by this document which defines how all members should work +together to achieve this goal. + +## Code of Conduct + +The Antrea community abides by this [code of conduct](CODE_OF_CONDUCT.md). + +## Community Roles + +* **Users:** Members that engage with the Antrea community via any medium + (Slack, GitHub, mailing lists, etc.). +* **Contributors:** Do regular contributions to the Antrea project + (documentation, code reviews, responding to issues, participating in proposal + discussions, contributing code, etc.). +* **Maintainers**: Responsible for the overall health and direction of the + project. They are the final reviewers of PRs and responsible for Antrea + releases. + +### Contributors + +Anyone can contribute to the project (e.g. open a PR) as long as they follow the +guidelines in [CONTRIBUTING.md](CONTRIBUTING.md). + +Frequent contributors to the project can become members of the antrea-io Github +organization and receive write access to the repository. Write access is +required to trigger re-runs of workflows in [Github +Actions](https://docs.github.com/en/actions/managing-workflow-runs/re-running-a-workflow). Becoming +a member of the antrea-io Github organization does not come with additional +responsibilities for the contributor, but simplifies the contributing +process. To become a member, you may [open an +issue](https://github.com/antrea-io/antrea/issues/new?template=membership.md&title=REQUEST%3A%20New%20membership%20for%20%3Cyour-GH-handle%3E) +and your membership needs to be approved by two maintainers: approval is +indicated by leaving a `+1` comment. If a contributor is not active for a +duration of 12 months (no contribution of any kind), they may be removed from +the antrea-io Github organization. In case of privilege abuse (members receive +write access to the organization), any maintainer can decide to disable write +access temporarily for the member. Within the next 2 weeks, the maintainer must +either restore the member's privileges, or remove the member from the +organization. The latter requires approval from at least one other maintainer, +which must be obtained publicly either on Github or Slack. + +### Maintainers + +The list of current maintainers can be found in +[MAINTAINERS.md](MAINTAINERS.md). + +While anyone can review a PR and is encouraged to do so, only maintainers are +allowed to merge the PR. To maintain velocity, only one maintainer's approval is +required to merge a given PR. In case of a disagreement between maintainers, a +vote should be called (on Github or Slack) and a simple majority is required in +order for the PR to be merged. + +New maintainers must be nominated from contributors by an existing maintainer +and must be elected by a [supermajority](#supermajority) of the current +maintainers. Likewise, maintainers can be removed by a supermajority of the +maintainers or can resign by notifying the maintainers. + +### Supermajority + +A supermajority is defined as two-thirds of members in the group. + +## Code of Conduct + +The code of conduct is overseen by the Antrea project maintainers. Possible code +of conduct violations should be emailed to the project maintainers at +. + +If the possible violation is against one of the project maintainers that member +will be recused from voting on the issue. Such issues must be escalated to the +appropriate CNCF contact, and CNCF may choose to intervene. + +## Updating Governance + +All substantive changes in Governance require a supermajority vote of the +maintainers. diff --git a/content/docs/v1.15.0/MAINTAINERS.md b/content/docs/v1.15.0/MAINTAINERS.md new file mode 100644 index 00000000..d3ab761f --- /dev/null +++ b/content/docs/v1.15.0/MAINTAINERS.md @@ -0,0 +1,11 @@ +# Antrea Maintainers + +This is the current list of maintainers for the Antrea project. The maintainer +role is described in [GOVERNANCE.md](GOVERNANCE.md). + +| Maintainer | GitHub ID | Affiliation | +| ---------- | --------- | ----------- | +| Antonin Bas | antoninbas | VMware | +| Jianjun Shen | jianjuns | VMware | +| Quan Tian | tnqn | VMware | +| Salvatore Orlando | salv-orlando | VMware | diff --git a/content/docs/v1.15.0/README.md b/content/docs/v1.15.0/README.md new file mode 100644 index 00000000..1c466508 --- /dev/null +++ b/content/docs/v1.15.0/README.md @@ -0,0 +1,137 @@ +# Antrea + +![Antrea Logo](docs/assets/logo/antrea_logo.svg) + +![Build Status](https://github.com/antrea-io/antrea/workflows/Go/badge.svg?branch=main) +[![Go Report Card](https://goreportcard.com/badge/antrea.io/antrea)](https://goreportcard.com/report/antrea.io/antrea) +[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4173/badge)](https://bestpractices.coreinfrastructure.org/projects/4173) +[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) +![GitHub release](https://img.shields.io/github/v/release/antrea-io/antrea?display_name=tag&sort=semver) +[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea?ref=badge_shield) + +## Overview + +Antrea is a [Kubernetes](https://kubernetes.io) networking solution intended +to be Kubernetes native. It operates at Layer 3/4 to provide networking and +security services for a Kubernetes cluster, leveraging +[Open vSwitch](https://www.openvswitch.org/) as the networking data plane. + +

+{{< img src="docs/assets/antrea_overview.svg.png" width="500" alt="Antrea Overview" >}} +

+ +Open vSwitch is a widely adopted high-performance programmable virtual +switch; Antrea leverages it to implement Pod networking and security features. +For instance, Open vSwitch enables Antrea to implement Kubernetes +Network Policies in a very efficient manner. + +## Prerequisites + +Antrea has been tested with Kubernetes clusters running version 1.16 or later. + +* `NodeIPAMController` must be enabled in the Kubernetes cluster.\ + When deploying a cluster with kubeadm the `--pod-network-cidr ` + option must be specified. + Alternately, NodeIPAM feature of Antrea Controller should be enabled and + configured. +* Open vSwitch kernel module must be present on every Kubernetes node. + +## Getting Started + +Getting started with Antrea is very simple, and takes only a few minutes. +See how it's done in the [Getting started](docs/getting-started.md) document. + +## Contributing + +The Antrea community welcomes new contributors. We are waiting for your PRs! + +* Before contributing, please get familiar with our +[Code of Conduct](CODE_OF_CONDUCT.md). +* Check out the Antrea [Contributor Guide](CONTRIBUTING.md) for information +about setting up your development environment and our contribution workflow. +* Learn about Antrea's [Architecture and Design](docs/design/architecture.md). +Your feedback is more than welcome! +* Check out [Open Issues](https://github.com/antrea-io/antrea/issues). +* Join the Antrea [community](#community) and ask us any question you may have. + +### Community + +* Join the [Kubernetes Slack](http://slack.k8s.io/) and look for our +[#antrea](https://kubernetes.slack.com/messages/CR2J23M0X) channel. +* Check the [Antrea Team Calendar](https://calendar.google.com/calendar/embed?src=uuillgmcb1cu3rmv7r7jrhcrco%40group.calendar.google.com) + and join the developer and user communities! + + The [Antrea community meeting](https://broadcom.zoom.us/j/91668049513?pwd=WHpaYTE2eWhja0xUN21MRU1BWllYdz09), +every two weeks on Tuesday at 5AM GMT+1 (United Kingdom time). See Antrea team calendar for localized times. + - [Meeting minutes](https://github.com/antrea-io/antrea/wiki/Community-Meetings) + - [Meeting recordings](https://www.youtube.com/playlist?list=PLuzde2hYeDBdw0BuQCYbYqxzoJYY1hfwv) + + [Antrea live office hours](https://antrea.io/live) archives. +* Join our mailing lists to always stay up-to-date with Antrea development: + + [projectantrea-announce](https://groups.google.com/forum/#!forum/projectantrea-announce) +for important project announcements. + + [projectantrea](https://groups.google.com/forum/#!forum/projectantrea) +for updates about Antrea or provide feedback. + + [projectantrea-dev](https://groups.google.com/forum/#!forum/projectantrea-dev) +to participate in discussions on Antrea development. + +Also check out [@ProjectAntrea](https://twitter.com/ProjectAntrea) on Twitter! + +## Features + +* **Kubernetes-native**: Antrea follows best practices to extend the Kubernetes + APIs and provide familiar abstractions to users, while also leveraging + Kubernetes libraries in its own implementation. +* **Powered by Open vSwitch**: Antrea relies on Open vSwitch to implement all + networking functions, including Kubernetes Service load-balancing, and to + enable hardware offloading in order to support the most demanding workloads. +* **Run everywhere**: Run Antrea in private clouds, public clouds and on bare + metal, and select the appropriate traffic mode (with or without overlay) based + on your infrastructure and use case. +* **Comprehensive policy model**: Antrea provides a comprehensive network policy + model, which builds upon Kubernetes Network Policies with new features such as + policy tiering, rule priorities and cluster-level policies. Refer to the + [Antrea Network Policy documentation](docs/antrea-network-policy.md) for a + full list of features. +* **Windows Node support**: Thanks to the portability of Open vSwitch, Antrea + can use the same data plane implementation on both Linux and Windows + Kubernetes Nodes. +* **Multi-cluster networking**: Federate multiple Kubernetes clusters and + benefit from a unified data plane (including multi-cluster Services) and a + unified security posture. Refer to the [Antrea Multi-cluster documentation](docs/multicluster/user-guide.md) + to get started. +* **Troubleshooting and monitoring tools**: Antrea comes with CLI and UI tools + which provide visibility and diagnostics capabilities (packet tracing, policy + analysis, flow inspection). It exposes Prometheus metrics and supports + exporting network flow information to collectors and analyzers. +* **Network observability and analytics**: Antrea + [Theia](https://github.com/antrea-io/theia) + enable fine-grained visibility into the communication among Kubernetes + workloads. Theia provides visualization for Antrea network flows in Grafana + dashboards, and recommends Network Policies to secure the workloads. +* **Network Policies for virtual machines**: Antrea native policies can be + enforced on non-Kubernetes Nodes including VMs and baremetal servers. Project + [Nephe](https://github.com/antrea-io/nephe) implements security policies for + VMs across clouds, leveraging Antrea native policies. +* **Encryption**: Encryption of inter-Node Pod traffic with IPsec or WireGuard + tunnels. +* **Easy deployment**: Antrea is deployed by applying a single YAML manifest + file. + +To explore more Antrea features and their usage, check the [Getting started](docs/getting-started.md#features) +document and user guides in the [Antrea documentation folder](docs/). Refer to +the [Changelogs](https://github.com/antrea-io/antrea/blob/v1.15.0/CHANGELOG/README.md) for a detailed list of features +introduced for each version release. + +## Adopters + +For a list of Antrea Adopters, please refer to [ADOPTERS.md](ADOPTERS.md). + +## Roadmap + +We are adding features very quickly to Antrea. Check out the list of features we +are considering on our [Roadmap](ROADMAP.md) page. Feel free to throw your ideas +in! + +## License + +Antrea is licensed under the [Apache License, version 2.0](https://github.com/antrea-io/antrea/blob/v1.15.0/LICENSE) + +[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea?ref=badge_large) diff --git a/content/docs/v1.15.0/ROADMAP.md b/content/docs/v1.15.0/ROADMAP.md new file mode 100644 index 00000000..e831599c --- /dev/null +++ b/content/docs/v1.15.0/ROADMAP.md @@ -0,0 +1,125 @@ +# Antrea Roadmap + +This document lists the new features being considered for the future. The +intention is for Antrea contributors and users to know what features could come +in the near future, and to share feedback and ideas. Priorities for the project +may change over time and so this roadmap is likely to evolve. A feature that is +not listed now does not mean it will not be considered for Antrea. We definitely +welcome suggestions and ideas from everyone about the roadmap and Antrea +features. Reach us through Issues, Slack and / or Google Group! + +## Roadmap Items + +### Antrea v2 + +Antrea [version 2](https://github.com/antrea-io/antrea/issues/4832) is coming in +2024. We are graduating some popular features to Beta or GA, deprecating some +legacy APIs, dropping support for old K8s versions (< 1.19) to improve support +for newer ones, and more! This is a big milestone for the project, stay tuned! + +### K8s Node security + +So far Antrea has focused on K8s Pod networking and security, but we would like +to extend Antrea-native NetworkPolicies to cover protection of K8s Nodes +too. There is ongoing work for this, so expect this feature very soon! + +### Quality of life improvements for installation and upgrade + +We have a few things planned to improve basic usability: + +* provide separate container images for the Agent and Controller: this will + reduce image size and speed up deployment of new Antrea versions. +* support for installation and upgrade using the antctl CLI: this will provide + an alternative installation method and antctl will ensure that Antrea + components are upgraded in the right order to minimize workload disruption. +* CLI tools to facilitate migration from another CNI: we will take care of + provisioning the correct network resources for your existing workloads. + +### Core networking features + +We are currently working on supporting VLAN tagging for Egress traffic. In the +long term, we plan to add BGP support to the Antrea Agent, as it is a much +requested feature. + +### Windows support improvements + +Antrea [supports Windows K8s Nodes](docs/windows.md). However, a few features +including: Egress, NodePortLocal, IPsec encryption are not supported for Windows +yet. We will continue to add more features for Windows (starting with Egress) +and aim for feature parity with Linux. We encourage users to reach out if they +would like us to prioritize a specific feature. While the installation procedure +has improved significantly since we first added Windows support, we plan to keep +on streamlining the procedure (more automation) and on improving the user +documentation. + +### More robust FQDN support in Antrea NetworkPolicy + +Antrea provides a comprehensive network policy model, which builds upon K8s +Network Policies and provides many additional capabilities. One of them is the +ability to define policy rules using domain names (FQDNs). We think there is +some room to improve user experience with this feature, and we are working on +making it more stable. + +### Implementation of new upstream NetworkPolicy APIs + +[SIG Network](https://github.com/kubernetes/community/tree/master/sig-network) +is working on [new standard APIs](https://network-policy-api.sigs.k8s.io/) to +extend the base K8s NetworkPolicy resource. We are closely monitoring the +upstream work and implementing these APIs as their development matures. + +### Better network troubleshooting with packet capture + +Antrea comes with many tools for network diagnostics and observability. You may +already be familiar with Traceflow, which lets you trace a single packet through +the Antrea network. We plan on also providing users with the ability to capture +live traffic and export it in PCAP format. Think tcpdump, but for K8s and +through a dedicated Antrea API! + +### Multi-network support for Pods + +We recently added the SecondaryNetwork feature, which supports provisioning +additional networks for Pods, using the same constructs made popular by +[Multus](https://github.com/k8snetworkplumbingwg/multus-cni). However, at the +moment, options for network "types" are limited. We plan on supporting new use +cases (e.g., secondary network overlays, network acceleration with DPDK), as +well as on improving user experience for this feature (with some useful +documentation). + +### L7 security policy + +Support for L7 NetworkPolicies was added in version 1.10, providing the ability +to select traffic based on the application-layer context. However, the feature +currently only supports HTTP and TLS traffic, and we plan to extend support to +other protocols, such as DNS. + +### Multi-cluster networking + +Antrea can federate multiple K8s clusters, but this feature (introduced in +version 1.7) is still considered Alpha today. Most of the functionality is +already there (multi-cluster Services, cross-cluster connectivity, +and multi-cluster NetworkPolicies), but we think there is some room for +improvement when it comes to stability and usability. + +### NetworkPolicy scale and performance tests + +We are working on a framework to empower contributors and users to benchmark the +performance of Antrea at scale. + +### Investigate better integration with service meshes + +As service meshes start introducing alternatives to the sidecar approach, +we believe there is an opportunity to improve the synergy between the K8s +network plugin and the service mesh provider. In particular, we are looking at +how Antrea can integrate with the new Istio ambient data plane mode. Take a look +at [#5682](https://github.com/antrea-io/antrea/issues/5682) for more +information. + +### Investigate multiple replicas for the Controller + +While today the Antrea Controller can scale to 1000s of K8s Nodes and 100,000 +Pods, and failover to a new replica in case of failure can happen in under a +minute, we believe we should still investigate the possibility of deploying +multiple replicas for the Controller (Active-Active or Active-Standby), to +enable horizontal scaling and achieve high-availability with very quick +failover. Horizontal scaling could help reduce the memory footprint of each +Controller instance for very large K8s clusters. diff --git a/content/docs/v1.15.0/SECURITY.md b/content/docs/v1.15.0/SECURITY.md new file mode 100644 index 00000000..6456edbd --- /dev/null +++ b/content/docs/v1.15.0/SECURITY.md @@ -0,0 +1,81 @@ +# Security Procedures + +The Antrea community holds security in the highest regard. +The community adopted this security disclosure policy to ensure vulnerabilities are responsibly handled. + +## Reporting a Vulnerability + +If you believe you have identified a vulnerability, please work with the Antrea maintainers to fix it and disclose the issue responsibly. +All security issues, confirmed or suspected, should be reported privately. +Please avoid using github issues, and instead report the vulnerability to . + +A vulnerability report should be filed if any of the following applies: + +* You have discovered and confirmed a vulnerability in Antrea. +* You believe Antrea might be vulnerable to some published [CVE](https://cve.mitre.org/cve/). +* You have found a potential security flaw in Antrea but you're not yet sure whether there's a viable attack vector. +* You have confirmed or suspect any of Antrea's dependencies has a vulnerability. + +### Vulnerability report template + +Provide a descriptive subject and include the following information in the body: + +* Detailed steps to reproduce the vulnerability (scripts, screenshots, packet captures, manual procedures, etc.). +* Describe the effects of the vulnerability on the Kubernetes cluster, on the applications running on it, and on the underlying infrastructure, if applicable. +* How the vulnerability affects Antrea workflows. +* Potential attack vectors and an estimation of the attack surface, if applicable. +* Other software that was used to expose the vulnerability. + +## Responding to a vulnerability + +A coordinator is assigned to each reported security issue. The coordinator is a member from the Antrea maintainers team, and will drive the fix and disclosure process. +At the moment reports are received via email at . +The first steps performed by the coordinator are to confirm the validity of the report and send an embargo reminder to all parties involved. +Antrea maintainers and issue reporters will review the issue for confirmation of impact and determination of affected components. + +With reference to the scale reported below, reported vulnerabilities will be disclosed and treated as regular issues if their issue risk is low (level 4 or higher in the scale). +For these lower-risk issues the fix process will proceed with the usual github workflow. + +### Reference taxonomy for issue risk + +1. Vulnerability must be fixed in main and any other supported branch. +2. Vulnerability must be fixed in main only for next release. +3. Vulnerability in experimental features or troubleshooting code. +4. Vulnerability without practical attack vector (e.g.: needs GUID guessing). +5. Not a vulnerability per se, but an opportunity to strengthen security (in code, architecture, protocols, and/or processes). +6. Not a vulnerability or a strengthening opportunity. +7. Vulnerability only exist in some PR or non-release branch. + +## Developing a patch for a vulnerability + +This part of the process applies only to confirmed vulnerabilities. +The reporter and Antrea maintainers, plus anyone they deem necessary to develop and validate a fix will be included the discussion. + +**Please refrain from creating a PR for the fix!** + +A fix is proposed as a patch to the current main branch, formatted with: + +```bash +git format-patch --stdout HEAD~1 > path/to/local/file.patch +``` + +and then sent to . + +**Please don't push the patch to the Antrea fork on your github account!** + +Patch review will be performed via email. Reviewers will suggest modifications and/or improvements, and then pre-approve it for merging. +Pre-approval will ensure patches can be fast-tracked through public code review later at disclosure time. + +## Disclosing the vulnerability + +In preparation for this, at least a maintainer must be available to help pushing the fix at disclosure time. + +At the disclosure time, one of the maintainers (or the reporter) will open an issue on github and create a PR with the patch for the main branch and any other applicable branch. +Available maintainers will fast-track approvals and merge the patch. + +Regardless of the owner of the issue and the corresponding PR, the original reporter and the submitter of the fix will be properly credited. +As for the git history, the commit message and author of the pre-approved patch will be preserved in the final patch submitted into the Antrea repository. + +### Notes + +At the moment the Antrea project does not have a process to assign a CVE to a confirmed vulnerability. diff --git a/content/docs/v1.15.0/_index.md b/content/docs/v1.15.0/_index.md new file mode 100644 index 00000000..5b7b4cd0 --- /dev/null +++ b/content/docs/v1.15.0/_index.md @@ -0,0 +1,7 @@ +--- +cascade: + layout: docs + version: v1.15.0 +--- + +{{% include-md "README.md" %}} diff --git a/content/docs/v1.15.0/docs/admin-network-policy.md b/content/docs/v1.15.0/docs/admin-network-policy.md new file mode 100644 index 00000000..4fac588f --- /dev/null +++ b/content/docs/v1.15.0/docs/admin-network-policy.md @@ -0,0 +1,119 @@ +# AdminNetworkPolicy API Support in Antrea + +## Table of Contents + + +- [Introduction](#introduction) +- [Prerequisites](#prerequisites) +- [Usage](#usage) + - [Sample specs for AdminNetworkPolicy and BaselineAdminNetworkPolicy](#sample-specs-for-adminnetworkpolicy-and-baselineadminnetworkpolicy) + - [Relationship with Antrea-native Policies](#relationship-with-antrea-native-policies) + + +## Introduction + +Kubernetes provides the NetworkPolicy API as a simple way for developers to control traffic flows of their applications. +While NetworkPolicy is embraced throughout the community, it was designed for developers instead of cluster admins. +Therefore, traits such as the lack of explicit deny rules make securing workloads at the cluster level difficult. +The Network Policy API working group (subproject of Kubernetes SIG-Network) has then introduced the +[AdminNetworkPolicy APIs](https://network-policy-api.sigs.k8s.io/api-overview/) which aims to solve the cluster admin +policy usecases. + +Starting with v1.13, Antrea supports the `AdminNetworkPolicy` and `BaselineAdminNetworkPolicy` API types, except for +advanced Namespace selection mechanisms (namely `sameLabels` and `notSameLabels` rules) which are still in the +experimental phase and not required as part of conformance. + +## Prerequisites + +AdminNetworkPolicy was introduced in v1.13 as an alpha feature and is disabled by default. A feature gate, +`AdminNetworkPolicy`, must be enabled in antrea-controller.conf in the `antrea-config` ConfigMap when Antrea is deployed: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: antrea-config + namespace: kube-system +data: + antrea-controller.conf: | + featureGates: + AdminNetworkPolicy: true +``` + +Note that the `AdminNetworkPolicy` feature also requires the `AntreaPolicy` featureGate to be set to true, which is +enabled by default since Antrea v1.0. + +In addition, the AdminNetworkPolicy CRD types need to be installed in the K8s cluster. +Refer to [this document](https://network-policy-api.sigs.k8s.io/getting-started/) for more information. + +## Usage + +### Sample specs for AdminNetworkPolicy and BaselineAdminNetworkPolicy + +Please refer to the [examples page](https://network-policy-api.sigs.k8s.io/reference/examples/) of the network-policy-api +repo, which contains several user stories for the AdminNetworkPolicy APIs, as well as sample specs for each of the user +story. Shown below are sample specs of `AdminNetworkPolicy` and `BaselineAdminNetworkPolicy` for demonstration purposes: + +```yaml +apiVersion: policy.networking.k8s.io/v1alpha1 +kind: AdminNetworkPolicy +metadata: + name: cluster-wide-deny-example +spec: + priority: 10 + subject: + namespaces: + matchLabels: + kubernetes.io/metadata.name: sensitive-ns + ingress: + - action: Deny + from: + - namespaces: + namespaceSelector: {} + name: select-all-deny-all +``` + +```yaml +apiVersion: policy.networking.k8s.io/v1alpha1 +kind: BaselineAdminNetworkPolicy +metadata: + name: default +spec: + subject: + namespaces: {} + ingress: + - action: Deny # zero-trust cluster default security posture + from: + - namespaces: + namespaceSelector: {} +``` + +Note that for a single cluster, the `BaselineAdminNetworkPolicy` resource is supported as a singleton with the name of +`default`. + +### Relationship with Antrea-native Policies + +AdminNetworkPolicy API objects and Antrea-native policies can co-exist with each other in the same cluster. + +AdminNetworkPolicy and BaselineAdminNetworkPolicy API types provide K8s upstream supported, cluster admin facing +guardrails that are portable and CNI-agnostic. AntreaClusterNetworkPolicy and AntreaNetworkPolicy on the other hand, +are designed for similar use cases but provide a richer feature set, including FQDN policies, nodeSelectors and L7 rules. +See the [Antrea-native policy doc](antrea-network-policy.md) and [L7 policy doc](antrea-network-policy.md) for details. + +Both the AdminNetworkPolicy object and Antrea-native policy objects use a `priority` field to determine its precedence +compared to other policy objects. The following diagram describes the relative precedence between the AdminNetworkPolicy +API types and Antrea-native policy types: + +```text +Antrea-native Policies (tier != baseline) > +AdminNetworkPolicies > +K8s NetworkPolicies > +Antrea-native Policies (tier == baseline) > +BaselineAdminNetworkPolicy +``` + +In other words, any Antrea-native policies that are not created in the `baseline` tier will have higher precedence over, +and thus evaluated before, all AdminNetworkPolicies at any `priority`. Effectively, the AdminNetworkPolicy objects are +associated with a tier priority lower than Antrea-native policies, but higher than K8s NetworkPolicies. Similarly, +baseline-tier Antrea-native policies will have a higher precedence over the BaselineAdminNetworkPolicy object. +For more information on policy and rule precedence, refer to [this section](antrea-network-policy.md#notes-and-constraints). diff --git a/content/docs/v1.15.0/docs/aks-installation.md b/content/docs/v1.15.0/docs/aks-installation.md new file mode 100644 index 00000000..7cded58a --- /dev/null +++ b/content/docs/v1.15.0/docs/aks-installation.md @@ -0,0 +1,283 @@ +# Deploying Antrea on AKS and AKS Engine + +This document describes steps to deploy Antrea to an AKS cluster or an AKS +Engine cluster. + +## Deploy Antrea to an AKS cluster + +Antrea can be deployed to an AKS cluster either in `networkPolicyOnly` mode or +in `encap` mode. + +In `networkPolicyOnly` mode, Antrea enforces NetworkPolicies and implements +other services for the AKS cluster, while the Azure CNI takes care of Pod IPAM +and traffic routing across Nodes. For more information about `networkPolicyOnly` +mode, refer to [this design document](design/policy-only.md). + +In `encap` mode, Antrea is in charge of Pod IPAM and of all the networking +functions on the Nodes. Using `encap` mode provides access to additional Antrea +features, such as Multicast, as inter-Node Pod traffic is encapsulated, and is +not handled directly by the Azure Virtual Network. Note that the [caveats](eks-installation.md#deploying-antrea-in-encap-mode) +which apply when deploying Antrea in `encap` mode on EKS do *not* apply for AKS. + +We recommend `encap` mode, as it will give you access to the most Antrea +features. + +### AKS Prerequisites + +Install the Azure Cloud CLI. Refer to [Azure CLI installation guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) + +We recommend using the latest version available (use at least version 2.39.0). + +### Deploying Antrea in `networkPolicyOnly` mode + +#### Creating the cluster + +You can use any method to create an AKS cluster. The example given here is using the Azure Cloud CLI. + +1. Create an AKS cluster + + ```bash + export RESOURCE_GROUP_NAME=aks-antrea-cluster + export CLUSTER_NAME=aks-antrea-cluster + export LOCATION=westus + + az group create --name $RESOURCE_GROUP_NAME --location $LOCATION + az aks create \ + --resource-group $RESOURCE_GROUP_NAME \ + --name $CLUSTER_NAME \ + --node-count 2 \ + --network-plugin azure + ``` + + **Note** Do not specify network-policy option. + +2. Get AKS cluster credentials + + ```bash + az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME + ``` + +3. Access your cluster + + ```bash + kubectl get nodes + NAME STATUS ROLES AGE VERSION + aks-nodepool1-84330359-vmss000000 Ready agent 6m21s v1.16.10 + aks-nodepool1-84330359-vmss000001 Ready agent 6m25s v1.16.10 + ``` + +#### Deploying Antrea + +1. Prepare the cluster Nodes + + Deploy ``antrea-node-init`` DaemonSet to enable ``azure cni`` to operate in transparent mode. + + ```bash + kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/antrea-aks-node-init.yml + ``` + +2. Deploy Antrea + + To deploy a released version of Antrea, pick a deployment manifest from the +[list of releases](https://github.com/antrea-io/antrea/releases). +Note that AKS support was added in release 0.9.0, which means you cannot +pick a release older than 0.9.0. For any given release `` (e.g. `v0.9.0`), +you can deploy Antrea as follows: + + ```bash + kubectl apply -f https://github.com/antrea-io/antrea/releases/download//antrea-aks.yml + ``` + + To deploy the latest version of Antrea (built from the main branch), use the +checked-in [deployment yaml](https://github.com/antrea-io/antrea/blob/v1.15.0/build/yamls/antrea-aks.yml): + + ```bash + kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/antrea-aks.yml + ``` + + The command will deploy a single replica of Antrea controller to the AKS +cluster and deploy Antrea agent to every Node. After a successful deployment +you should be able to see these Pods running in your cluster: + + ```bash + $ kubectl get pods --namespace kube-system -l app=antrea + NAME READY STATUS RESTARTS AGE + antrea-agent-bpj72 2/2 Running 0 40s + antrea-agent-j2sjz 2/2 Running 0 40s + antrea-controller-6f7468cbff-5sk4t 1/1 Running 0 43s + antrea-node-init-6twqg 1/1 Running 0 2m + antrea-node-init-mqsqr 1/1 Running 0 2m + ``` + +3. Restart remaining Pods + + Once Antrea is up and running, restart all Pods in all Namespaces (kube-system, etc) so they can be managed by Antrea. + + ```bash + kubectl delete pods -n kube-system $(kubectl get pods -n kube-system -o custom-columns=NAME:.metadata.name,HOSTNETWORK:.spec.hostNetwork --no-headers=true | grep '' | awk '{ print $1 }') + pod "coredns-544d979687-96xm9" deleted + pod "coredns-544d979687-p7dfb" deleted + pod "coredns-autoscaler-78959b4578-849k8" deleted + pod "dashboard-metrics-scraper-5f44bbb8b5-5qkkx" deleted + pod "kube-proxy-6qxdw" deleted + pod "kube-proxy-h6d89" deleted + pod "kubernetes-dashboard-785654f667-7twsm" deleted + pod "metrics-server-85c57978c6-pwzcx" deleted + pod "tunnelfront-649ff5fb55-5lxg7" deleted + ``` + +### Deploying Antrea in `encap` mode + +AKS now officially supports [Bring your own Container Network Interface (BYOCNI)](https://learn.microsoft.com/en-us/azure/aks/use-byo-cni). +Thanks to this, you can deploy Antrea on AKS in `encap` mode, and you will not +lose access to any functionality. Check the AKS BYOCNI documentation for +prerequisites, in particular for AKS version requirements. + +#### Creating the cluster + +You can use any method to create an AKS cluster. The example given here is using the Azure Cloud CLI. + +1. Create an AKS cluster + + ```bash + export RESOURCE_GROUP_NAME=aks-antrea-cluster + export CLUSTER_NAME=aks-antrea-cluster + export LOCATION=westus + + az group create --name $RESOURCE_GROUP_NAME --location $LOCATION + az aks create \ + --resource-group $RESOURCE_GROUP_NAME \ + --name $CLUSTER_NAME \ + --node-count 2 \ + --network-plugin none + ``` + + Notice `--network-plugin none`, which tells AKS not to install any CNI plugin. + +2. Get AKS cluster credentials + + ```bash + az aks get-credentials --name $CLUSTER_NAME --resource-group $RESOURCE_GROUP_NAME + ``` + +3. Access your cluster + + ```bash + kubectl get nodes + NAME STATUS ROLES AGE VERSION + aks-nodepool1-40948307-vmss000000 NotReady agent 18m v1.27.7 + aks-nodepool1-40948307-vmss000001 NotReady agent 17m v1.27.7 + ``` + + The Nodes are supposed to report a `NotReady` Status, since no CNI plugin is + installed yet. + +#### Deploying Antrea + +You can use Helm to easily install Antrea (or any other supported installation +method). Just make sure that you configure Antrea NodeIPAM: + +```bash +# you may not need this: +helm repo add antrea https://charts.antrea.io +helm repo update + +cat <> values-aks.yml +nodeIPAM: + enable: true + clusterCIDRs: ["10.10.0.0/16"] +EOF + +helm install -n kube-system -f values-aks.yml antrea antrea/antrea +``` + +For more information about how to configure Antrea Node IPAM, please refer to +[Antrea Node IPAM guide](antrea-ipam.md#running-nodeipam-within-antrea-controller). + +After a while, make sure that all your Nodes report a `Ready` Status and that +all your Pods are running correctly. Some Pods, and in particular the +`metrics-server` Pods, may restart once after installing Antrea; this is not an +issue. + +After a successful installation, Pods should look like this: + +```bash +NAMESPACE NAME READY STATUS RESTARTS AGE +kube-system antrea-agent-bpskv 2/2 Running 0 7m34s +kube-system antrea-agent-pfqrn 2/2 Running 0 7m34s +kube-system antrea-controller-555b8c799d-wk8zz 1/1 Running 0 7m34s +kube-system cloud-node-manager-2nszz 1/1 Running 0 31m +kube-system cloud-node-manager-wj68q 1/1 Running 0 31m +kube-system coredns-789789675-2nwd7 1/1 Running 0 6m48s +kube-system coredns-789789675-lbkfn 1/1 Running 0 31m +kube-system coredns-autoscaler-649b947bbd-j5wqc 1/1 Running 0 31m +kube-system csi-azuredisk-node-4bnnl 3/3 Running 0 31m +kube-system csi-azuredisk-node-52nwd 3/3 Running 0 31m +kube-system csi-azurefile-node-2h66l 3/3 Running 0 31m +kube-system csi-azurefile-node-dhrf2 3/3 Running 0 31m +kube-system konnectivity-agent-5fc7989878-6nhwl 1/1 Running 0 31m +kube-system konnectivity-agent-5fc7989878-t2n6h 1/1 Running 0 30m +kube-system kube-proxy-96c9p 1/1 Running 0 31m +kube-system kube-proxy-x8g8s 1/1 Running 0 31m +kube-system metrics-server-5955767688-2hjvn 2/2 Running 0 3m45s +kube-system metrics-server-5955767688-vmcq7 2/2 Running 0 3m45s +``` + +## Deploy Antrea to an AKS Engine cluster + +Antrea is an integrated CNI of AKS Engine, and can be installed in +`networkPolicyOnly` mode or `encap` mode to an AKS Engine cluster as part of the +AKS Engine cluster deployment. To learn basics of AKS Engine cluster deployment, +please refer to [AKS Engine Quickstart Guide](https://github.com/Azure/aks-engine/blob/master/docs/tutorials/quickstart.md). + +### Deploying Antrea in `networkPolicyOnly` mode + +To configure Antrea to enforce NetworkPolicies for the AKS Engine cluster, +`"networkPolicy": "antrea"` needs to be set in `kubernetesConfig` of the AKS +Engine cluster definition (Azure CNI will be used as the `networkPlugin`): + +```json + "apiVersion": "vlabs", + "properties": { + "orchestratorProfile": { + "kubernetesConfig": { + "networkPolicy": "antrea" + } + } + } +``` + +You can use the deployment template +[`examples/networkpolicy/kubernetes-antrea.json`](https://github.com/Azure/aks-engine/blob/master/examples/networkpolicy/kubernetes-antrea.json) +to deploy an AKS Engine cluster with Antrea in `networkPolicyOnly` mode: + +```bash +$ aks-engine deploy --dns-prefix \ + --resource-group \ + --location westus2 \ + --api-model examples/networkpolicy/kubernetes-antrea.json \ + --auto-suffix +``` + +### Deploying Antrea in `encap` mode + +To deploy Antrea in `encap` mode for an AKS Engine cluster, both +`"networkPlugin": "antrea"` and `"networkPolicy": "antrea"` need to be set in +`kubernetesConfig` of the AKS Engine cluster definition: + +```json + "apiVersion": "vlabs", + "properties": { + "orchestratorProfile": { + "kubernetesConfig": { + "networkPlugin": "antrea", + "networkPolicy": "antrea" + } + } + } +``` + +You can add `"networkPlugin": "antrea"` to the deployment template +[`examples/networkpolicy/kubernetes-antrea.json`](https://github.com/Azure/aks-engine/blob/master/examples/networkpolicy/kubernetes-antrea.json), +and use the template to deploy an AKS Engine cluster with Antrea in `encap` +mode. diff --git a/content/docs/v1.15.0/docs/antctl.md b/content/docs/v1.15.0/docs/antctl.md new file mode 100644 index 00000000..96976844 --- /dev/null +++ b/content/docs/v1.15.0/docs/antctl.md @@ -0,0 +1,758 @@ +# Antctl + +antctl is the command-line tool for Antrea. At the moment, antctl supports +running in three different modes: + +* "controller mode": when run out-of-cluster or from within the Antrea + Controller Pod, antctl can connect to the Antrea Controller and query + information from it (e.g. the set of computed NetworkPolicies). +* "agent mode": when run from within an Antrea Agent Pod, antctl can connect to + the Antrea Agent and query information local to that Agent (e.g. the set of + computed NetworkPolicies received by that Agent from the Antrea Controller, as + opposed to the entire set of computed policies). +* "flowaggregator mode": when run from within a Flow Aggregator Pod, antctl can + connect to the Flow Aggregator and query information from it (e.g. flow records + related statistics). + +## Table of Contents + + +- [Installation](#installation) +- [Usage](#usage) + - [Showing or changing log verbosity level](#showing-or-changing-log-verbosity-level) + - [Showing feature gates status](#showing-feature-gates-status) + - [Collecting support information](#collecting-support-information) + - [controllerinfo and agentinfo commands](#controllerinfo-and-agentinfo-commands) + - [NetworkPolicy commands](#networkpolicy-commands) + - [Mapping endpoints to NetworkPolicies](#mapping-endpoints-to-networkpolicies) + - [Dumping Pod network interface information](#dumping-pod-network-interface-information) + - [Dumping OVS flows](#dumping-ovs-flows) + - [OVS packet tracing](#ovs-packet-tracing) + - [Traceflow](#traceflow) + - [Antctl Proxy](#antctl-proxy) + - [Flow Aggregator commands](#flow-aggregator-commands) + - [Dumping flow records](#dumping-flow-records) + - [Record metrics](#record-metrics) + - [Multi-cluster commands](#multi-cluster-commands) + - [Multicast commands](#multicast-commands) + - [Showing memberlist state](#showing-memberlist-state) + - [Upgrade existing objects of CRDs](#upgrade-existing-objects-of-crds) + + +## Installation + +The antctl binary is included in the Antrea Docker image +(`antrea/antrea-ubuntu`) which means that there is no need to install anything +to connect to the Antrea Agent. Simply exec into the antrea-agent container for +the appropriate antrea-agent Pod and run `antctl`: + +```bash +kubectl exec -it ANTREA-AGENT_POD_NAME -n kube-system -c antrea-agent -- bash +> antctl help +``` + +Starting with Antrea release v0.5.0, we publish the antctl binaries for +different OS / CPU Architecture combinations. Head to the [releases +page](https://github.com/antrea-io/antrea/releases) and download the +appropriate one for your machine. For example: + +On Mac & Linux: + +```bash +curl -Lo ./antctl "https://github.com/antrea-io/antrea/releases/download//antctl-$(uname)-x86_64" +chmod +x ./antctl +mv ./antctl /some-dir-in-your-PATH/antctl +antctl version +``` + +For Linux, we also publish binaries for Arm-based systems. + +On Windows, using PowerShell: + +```powershell +Invoke-WebRequest -Uri https://github.com/antrea-io/antrea/releases/download//antctl-windows-x86_64.exe -Outfile antctl.exe +Move-Item .\antctl.exe c:\some-dir-in-your-PATH\antctl.exe +antctl version +``` + +## Usage + +To see the list of available commands and options, run `antctl help`. The list +will be different based on whether you are connecting to the Antrea Controller +or Agent. + +When running out-of-cluster ("controller mode" only), antctl will look for your +kubeconfig file at `$HOME/.kube/config` by default. You can select a different +one by setting the `KUBECONFIG` environment variable or with `--kubeconfig` +(the latter taking precedence over the former). + +The following sub-sections introduce a few commands which are useful for +troubleshooting the Antrea system. + +### Showing or changing log verbosity level + +Starting from version 0.10.0, Antrea supports showing or changing the log +verbosity level of Antrea Controller or Antrea Agent using the `antctl log-level` +command. Starting from version 1.5, Antrea supports showing or changing the +log verbosity level of the Flow Aggregator using the `antctl log-level` command. +The command can only run locally inside the `antrea-controller`, `antrea-agent` +or `flow-aggregator` container. + +The following command prints the current log verbosity level: + +```bash +antctl log-level +``` + +This command updates the log verbosity level (the `LEVEL` argument must be an +integer): + +```bash +antctl log-level LEVEL +``` + +### Showing feature gates status + +The feature gates of Antrea Controller and Agent can be shown using the `antctl get featuregates` command. +The command can run locally inside the `antrea-controller` or `antrea-agent` container or out-of-cluster, +when it is running out-of-cluster or in Controller Pod, it will print both Controller and Agent's feature gates list. + +The following command prints the current feature gates: + +```bash +antctl get featuregates +``` + +### Collecting support information + +Starting with version 0.7.0, Antrea supports the `antctl supportbundle` command, +which can collect information from the cluster, the Antrea Controller and all +Antrea agents. This information is useful when trying to troubleshoot issues in +Kubernetes clusters using Antrea. In particular, when running the command +out-of-cluster, all the information can be collected under one single directory, +which you can upload and share when reporting issues on Github. Simply run the +command as follows: + +```bash +antctl supportbundle [-d TARGET_DIR] +``` + +If you omit to provide a directory, antctl will create one in the current +working directory, using the current timestamp as a suffix. The command also +provides additional flags to filter the results: run `antctl supportbundle +--help` for the full list. + +The collected support bundle will include the following (more information may be +included over time): + +* cluster information: description of the different K8s resources in the cluster + (Nodes, Deployments, etc.). +* Antrea Controller information: all the available logs (contents will vary + based on the verbosity selected when running the controller) and state stored + at the controller (e.g. computed NetworkPolicy objects). +* Antrea Agent information: all the available logs from the agent and the OVS + daemons, network configuration of the Node (e.g. routes, iptables rules, OVS + flows) and state stored at the agent (e.g. computed NetworkPolicy objects + received from the controller). + +**Be aware that the generated support bundle includes a lot of information, + including logs, so please review the contents of the directory before sharing + it on Github and ensure that you do not share anything sensitive.** + +The `antctl supportbundle` command can also be run inside a Controller or Agent +Pod, in which case only local information will be collected. + +Since v1.10.0, Antrea also supports collecting information by applying a +`SupportBundleCollection` CRD, you can refer to the [support bundle guide](./support-bundle-guide.md) +for more information. + +### controllerinfo and agentinfo commands + +`antctl` controller command `get controllerinfo` (or `get ci`) and agent command +`get agentinfo` (or `get ai`) print the runtime information of +`antrea-controller` and `antrea-agent` respectively. + +```bash +antctl get controllerinfo +antctl get agentinfo +``` + +### NetworkPolicy commands + +Both Antrea Controller and Agent support querying the NetworkPolicy objects in the Antrea +control plane API. The source of a control plane NetworkPolicy is the original policy resource +(K8s NetworkPolicy, Antrea-native Policy or AdminNetworkPolicy) from which the control plane +NetworkPolicy was derived. + +- `antctl` `get networkpolicy` (or `get netpol`) command can print all +NetworkPolicies, a specified NetworkPolicy, or NetworkPolicies in a specified +Namespace. +- `get appliedtogroup` (or `get atg`) command can print all NetworkPolicy +AppliedToGroups (AppliedToGroup includes the Pods to which a NetworkPolicy is +applied), or a specified AppliedToGroup. +- `get addressgroup` (or `get ag`) command can print all NetworkPolicy +AddressGroups (AddressGroup defines source or destination addresses of +NetworkPolicy rules), or a specified AddressGroup. + +Using the `json` or `yaml` antctl output format can print more information of +NetworkPolicy, AppliedToGroup, and AddressGroup, than using the default `table` +output format. The `NAME` of a control plane NetworkPolicy is the UID of its source +NetworkPolicy. + +```bash +antctl get networkpolicy [NAME] [-n NAMESPACE] [-o yaml] +antctl get appliedtogroup [NAME] [-o yaml] +antctl get addressgroup [NAME] [-o yaml] +``` + +NetworkPolicy, AppliedToGroup, and AddressGroup also support `sort-by=''` option, +which can be used to sort these resources on the basis of a particular field. Any +valid json path can be passed as flag value. If no value is passed it will use a +default field to sort results. For NetworkPolicy, the default field is the name of +the source NetworkPolicy. For AppliedToGroup and AddressGroup, the default field is +the object name (which is a generated UUID). + +```bash +antctl get networkpolicy --sort-by='.sourceRef.name' +antctl get appliedtogroup --sort-by='.metadata.name' +antctl get addressgroup --sort-by='.metadata.name' +``` + +NetworkPolicy also supports `sort-by=effectivePriority` option, which can be used to +view the effective order in which the NetworkPolicies are evaluated. Antrea-native +NetworkPolicy ordering is documented [here]( +antrea-network-policy.md#antrea-native-policy-ordering-based-on-priorities). + +```bash +antctl get networkpolicy --sort-by=effectivePriority +``` + +Antrea Agent supports some extra `antctl` commands. + +* Printing NetworkPolicies applied to a specific local Pod. + + ```bash + antctl get networkpolicy -p POD -n NAMESPACE + ``` + +* Printing NetworkPolicies with a specific source NetworkPolicy type. + + ```bash + antctl get networkpolicy -T (K8sNP|ACNP|ANNP|ANP) + ``` + +* Printing NetworkPolicies with a specific source NetworkPolicy name. + + ```bash + antctl get networkpolicy -S SOURCE_NAME [-n NAMESPACE] + ``` + +#### Mapping endpoints to NetworkPolicies + +`antctl` supports mapping a specific Pod to the NetworkPolicies which "select" +this Pod, either because they apply to the Pod directly or because one of their +policy rules selects the Pod. + +```bash +antctl query endpoint -p POD [-n NAMESPACE] +``` + +If no Namespace is provided with `-n`, the command will default to the "default" +Namespace. + +This command only works in "controller mode" and **as of now it can only be run +from inside the Antrea Controller Pod, and not from out-of-cluster**. + +### Dumping Pod network interface information + +`antctl` agent command `get podinterface` (or `get pi`) can dump network +interface information of all local Pods, or a specified local Pod, or local Pods +in the specified Namespace, or local Pods matching the specified Pod name. + +```bash +antctl get podinterface [NAME] [-n NAMESPACE] +``` + +### Dumping OVS flows + +Starting from version 0.6.0, Antrea Agent supports dumping Antrea OVS flows. The +`antctl` `get ovsflows` (or `get of`) command can dump all OVS flows, flows +added for a specified Pod, or flows added for Service load-balancing of a +specified Service, or flows added to realize a specified NetworkPolicy, or flows +in the specified OVS flow tables, or all or the specified OVS groups. + +```bash +antctl get ovsflows +antctl get ovsflows -p POD -n NAMESPACE +antctl get ovsflows -S SERVICE -n NAMESPACE +antctl get ovsflows -N NETWORKPOLICY -n NAMESPACE +antctl get ovsflows -T TABLE_A,TABLE_B +antctl get ovsflows -T TABLE_A,TABLE_B_NUM +antctl get ovsflows -G all +antctl get ovsflows -G GROUP_ID1,GROUP_ID2 +``` + +OVS flow tables can be specified using table names, or the table numbers. +`antctl get ovsflow --help` lists all Antrea flow tables. For more information +about Antrea OVS pipeline and flows, please refer to the [OVS pipeline doc](design/ovs-pipeline.md). + +Example outputs of dumping Pod and NetworkPolicy OVS flows: + +```bash +# Dump OVS flows of Pod "coredns-6955765f44-zcbwj" +$ antctl get of -p coredns-6955765f44-zcbwj -n kube-system +FLOW +table=classification, n_packets=513122, n_bytes=42615080, priority=190,in_port="coredns--d0c58e" actions=set_field:0x2/0xffff->reg0,resubmit(,10) +table=10, n_packets=513122, n_bytes=42615080, priority=200,ip,in_port="coredns--d0c58e",dl_src=52:bd:c6:e0:eb:c1,nw_src=172.100.1.7 actions=resubmit(,30) +table=10, n_packets=0, n_bytes=0, priority=200,arp,in_port="coredns--d0c58e",arp_spa=172.100.1.7,arp_sha=52:bd:c6:e0:eb:c1 actions=resubmit(,20) +table=80, n_packets=556468, n_bytes=166477824, priority=200,dl_dst=52:bd:c6:e0:eb:c1 actions=load:0x5->NXM_NX_REG1[],set_field:0x10000/0x10000->reg0,resubmit(,90) +table=70, n_packets=0, n_bytes=0, priority=200,ip,dl_dst=aa:bb:cc:dd:ee:ff,nw_dst=172.100.1.7 actions=set_field:62:39:b4:e8:05:76->eth_src,set_field:52:bd:c6:e0:eb:c1->eth_dst,dec_ttl,resubmit(,80) + +# Get NetworkPolicies applied to Pod "coredns-6955765f44-zcbwj" +$ antctl get netpol -p coredns-6955765f44-zcbwj -n kube-system +NAMESPACE NAME APPLIED-TO RULES +kube-system kube-dns 160ea6d7-0234-5d1d-8ea0-b703d0aa3b46 1 + +# Dump OVS flows of NetworkPolicy "kube-dns" +$ antctl get of -N kube-dns -n kube-system +FLOW +table=90, n_packets=0, n_bytes=0, priority=190,conj_id=1,ip actions=resubmit(,105) +table=90, n_packets=0, n_bytes=0, priority=200,ip actions=conjunction(1,1/3) +table=90, n_packets=0, n_bytes=0, priority=200,ip,reg1=0x5 actions=conjunction(2,2/3),conjunction(1,2/3) +table=90, n_packets=0, n_bytes=0, priority=200,udp,tp_dst=53 actions=conjunction(1,3/3) +table=90, n_packets=0, n_bytes=0, priority=200,tcp,tp_dst=53 actions=conjunction(1,3/3) +table=90, n_packets=0, n_bytes=0, priority=200,tcp,tp_dst=9153 actions=conjunction(1,3/3) +table=100, n_packets=0, n_bytes=0, priority=200,ip,reg1=0x5 actions=drop +``` + +### OVS packet tracing + +Starting from version 0.7.0, Antrea Agent supports tracing the OVS flows that a +specified packet traverses, leveraging the [OVS packet tracing tool](https://docs.openvswitch.org/en/latest/topics/tracing/). + +`antctl trace-packet` command starts a packet tracing operation. +`antctl help trace-packet` shows the usage of the command. This section lists a +few trace-packet command examples. + +```bash +# Trace an IP packet between two Pods +antctl trace-packet -S ns1/pod1 -D ns2/pod2 +# Trace a Service request from a local Pod +antctl trace-packet -S ns1/pod1 -D ns2/svc2 -f "tcp,tcp_dst=80" +# Trace the Service reply packet (assuming "ns2/pod2" is the Service backend Pod) +antctl trace-packet -D ns1/pod1 -S ns2/pod2 -f "tcp,tcp_src=80" +# Trace an IP packet from a Pod to gateway port +antctl trace-packet -S ns1/pod1 -D antrea-gw0 +# Trace a UDP packet from a Pod to an IP address +antctl trace-packet -S ns1/pod1 -D 10.1.2.3 -f udp,udp_dst=1234 +# Trace a UDP packet from an IP address to a Pod +antctl trace-packet -D ns1/pod1 -S 10.1.2.3 -f udp,udp_src=1234 +# Trace an ARP request from a local Pod +antctl trace-packet -p ns1/pod1 -f arp,arp_spa=10.1.2.3,arp_sha=00:11:22:33:44:55,arp_tpa=10.1.2.1,dl_dst=ff:ff:ff:ff:ff:ff +``` + +Example outputs of tracing a UDP (DNS request) packet from a remote Pod to a +local (coredns) Pod: + +```bash +$ antctl trace-packet -S default/web-client -D kube-system/coredns-6955765f44-zcbwj -f udp,udp_dst=53 +result: | + Flow: udp,in_port=1,vlan_tci=0x0000,dl_src=aa:bb:cc:dd:ee:ff,dl_dst=aa:bb:cc:dd:ee:ff,nw_src=172.100.2.11,nw_dst=172.100.1.7,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=53 + + bridge("br-int") + ---------------- + 0. in_port=1, priority 200, cookie 0x5e000000000000 + load:0->NXM_NX_REG0[0..15] + resubmit(,30) + 30. ip, priority 200, cookie 0x5e000000000000 + ct(table=31,zone=65520) + drop + -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 31. + -> Sets the packet to an untracked state, and clears all the conntrack fields. + + Final flow: unchanged + Megaflow: recirc_id=0,eth,udp,in_port=1,nw_frag=no,tp_src=0x0/0xfc00 + Datapath actions: ct(zone=65520),recirc(0x53) + + =============================================================================== + recirc(0x53) - resume conntrack with default ct_state=trk|new (use --ct-next to customize) + =============================================================================== + + Flow: recirc_id=0x53,ct_state=new|trk,ct_zone=65520,eth,udp,in_port=1,vlan_tci=0x0000,dl_src=aa:bb:cc:dd:ee:ff,dl_dst=aa:bb:cc:dd:ee:ff,nw_src=172.100.2.11,nw_dst=172.100.1.7,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=0,tp_dst=53 + + bridge("br-int") + ---------------- + thaw + Resuming from table 31 + 31. priority 0, cookie 0x5e000000000000 + resubmit(,40) + 40. priority 0, cookie 0x5e000000000000 + resubmit(,50) + 50. priority 0, cookie 0x5e000000000000 + resubmit(,60) + 60. priority 0, cookie 0x5e000000000000 + resubmit(,70) + 70. ip,dl_dst=aa:bb:cc:dd:ee:ff,nw_dst=172.100.1.7, priority 200, cookie 0x5e030000000000 + set_field:62:39:b4:e8:05:76->eth_src + set_field:52:bd:c6:e0:eb:c1->eth_dst + dec_ttl + resubmit(,80) + 80. dl_dst=52:bd:c6:e0:eb:c1, priority 200, cookie 0x5e030000000000 + set_field:0x5->reg1 + set_field:0x10000/0x10000->reg0 + resubmit(,90) + 90. conj_id=2,ip, priority 190, cookie 0x5e050000000000 + resubmit(,105) + 105. ct_state=+new+trk,ip, priority 190, cookie 0x5e000000000000 + ct(commit,table=110,zone=65520) + drop + -> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 110. + -> Sets the packet to an untracked state, and clears all the conntrack fields. + + Final flow: recirc_id=0x53,eth,udp,reg0=0x10000,reg1=0x5,in_port=1,vlan_tci=0x0000,dl_src=62:39:b4:e8:05:76,dl_dst=52:bd:c6:e0:eb:c1,nw_src=172.100.2.11,nw_dst=172.100.1.7,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=0,tp_dst=53 + Megaflow: recirc_id=0x53,ct_state=+new-est-inv+trk,ct_mark=0,eth,udp,in_port=1,dl_src=aa:bb:cc:dd:ee:ff,dl_dst=aa:bb:cc:dd:ee:ff,nw_src=192.0.0.0/2,nw_dst=172.100.1.7,nw_ttl=64,nw_frag=no,tp_dst=53 + Datapath actions: set(eth(src=62:39:b4:e8:05:76,dst=52:bd:c6:e0:eb:c1)),set(ipv4(ttl=63)),ct(commit,zone=65520),recirc(0x54) + + =============================================================================== + recirc(0x54) - resume conntrack with default ct_state=trk|new (use --ct-next to customize) + =============================================================================== + + Flow: recirc_id=0x54,ct_state=new|trk,ct_zone=65520,eth,udp,reg0=0x10000,reg1=0x5,in_port=1,vlan_tci=0x0000,dl_src=62:39:b4:e8:05:76,dl_dst=52:bd:c6:e0:eb:c1,nw_src=172.100.2.11,nw_dst=172.100.1.7,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=0,tp_dst=53 + + bridge("br-int") + ---------------- + thaw + Resuming from table 110 + 110. ip,reg0=0x10000/0x10000, priority 200, cookie 0x5e000000000000 + output:NXM_NX_REG1[] + -> output port is 5 + + Final flow: unchanged + Megaflow: recirc_id=0x54,eth,ip,in_port=1,nw_frag=no + Datapath actions: 3 +``` + +### Traceflow + +`antctl traceflow` (or `antctl tf`) command is used to start a Traceflow and +retrieve its result. After the result is collected, the Traceflow will be +deleted. Users can also create a Traceflow with `kubectl`, but `antctl traceflow` +offers a simpler way. For more information about Traceflow, refer to the +[Traceflow guide](traceflow-guide.md). + +To start a regular Traceflow, both `--source` (or `-S`) and `--destination` (or +`-D`) arguments must be specified, and the source must be a Pod. For example: + +```bash +$ antctl tf -S busybox0 -D busybox1 +name: busybox0-to-busybox1-fpllngzi +phase: Succeeded +source: default/busybox0 +destination: default/busybox1 +results: +- node: antrea-linux-testbed7-1 + timestamp: 1596435607 + observations: + - component: SpoofGuard + action: Forwarded + - component: Forwarding + componentInfo: Output + action: Delivered +``` + +To start a live-traffic Traceflow, add the `--live-traffic` (or `-L`) flag. Add +the `--dropped-only` flag to indicate only the packet dropped by a NetworkPolicy +should be captured in the live-traffic Traceflow. A live-traffic Traceflow +just requires one of `--source` and `--destination` arguments to be specified, +and at least one of them must be a Pod. + +The `--flow` (or `-f`) argument can be used to specify the Traceflow packet +headers with the [ovs-ofctl](http://www.openvswitch.org//support/dist-docs/ovs-ofctl.8.txt) +flow syntax. The supported flow fields include: IP family (`ipv6` to indicate an +IPv6 packet), IP protocol (`icmp`, `icmpv6`, `tcp`, `udp`), source and +destination ports (`tcp_src`, `tcp_dst`, `udp_src`, `udp_dst`), and TCP flags +(`tcp_flags`). + +By default, the command will wait for the Traceflow to succeed or fail, or +timeout. The default timeout is 10 seconds, but can be changed with the +`--timeout` (or `-t`) argument. Add the `--no-wait` flag to start a Traceflow +without waiting for its results. In this case, the command will not delete the +Traceflow resource. The `traceflow` command supports yaml and json output. + +More examples of `antctl traceflow`: + +```bash +# Start a Traceflow from pod1 to pod2, both Pods are in Namespace default +$ antctl traceflow -S pod1 -D pod2 +# Start a Traceflow from pod1 in Namepace ns1 to a destination IP +$ antctl traceflow -S ns1/pod1 -D 123.123.123.123 +# Start a Traceflow from pod1 to Service svc1 in Namespace ns1 +$ antctl traceflow -S pod1 -D ns1/svc1 -f tcp,tcp_dst=80 +# Start a Traceflow from pod1 to pod2, with a UDP packet to destination port 1234 +$ antctl traceflow -S pod1 -D pod2 -f udp,udp_dst=1234 +# Start a Traceflow for live TCP traffic from pod1 to svc1, with 1 minute timeout +$ antctl traceflow -S pod1 -D svc1 -f tcp --live-traffic -t 1m +# Start a Traceflow to capture the first dropped TCP packet to pod1 on port 80, within 10 minutes +$ antctl traceflow -D pod1 -f tcp,tcp_dst=80 --live-traffic --dropped-only -t 10m +``` + +### Antctl Proxy + +antctl can run as a reverse proxy for the Antrea API (Controller or arbitrary +Agent). Usage is very similar to `kubectl proxy` and the implementation is +essentially the same. + +To run a reverse proxy for the Antrea Controller API, use: + +```bash +antctl proxy --controller +```` + +To run a reverse proxy for the Antrea Agent API for the antrea-agent Pod running +on Node , use: + +```bash +antctl proxy --agent-node +``` + +You can then access the API at `127.0.0.1:8001`. To implement this +functionality, antctl retrieves the Node IP address and API server port for the +Antrea Controller or for the specified Agent from the K8s API, and it proxies +all the requests received on `127.0.0.1:8001` directly to that IP / port. One +thing to keep in mind is that the TLS connection between the proxy and the +Antrea Agent or Controller will not be secure (no certificate verification), and +the proxy should be used for debugging only. + +To see the full list of supported options, run `antctl proxy --help`. + +This feature is useful if one wants to use the Go +[pprof](https://golang.org/pkg/net/http/pprof/) tool to collect runtime +profiling data about the Antrea components. Please refer to this +[document](troubleshooting.md#profiling-antrea-components) for more information. + +### Flow Aggregator commands + +antctl supports dumping the flow records handled by the Flow Aggregator, and +printing metrics about flow record processing. These commands are only available +when you exec into the Flow Aggregator Pod. + +#### Dumping flow records + +antctl supports dumping flow records stored in the Flow Aggregator. The +`antctl get flowrecords` command can dump all matching flow records. It supports +the 5-tuple flow key or a subset of the 5-tuple as a filter. A 5-tuple flow key +contains Source IP, Destination IP, Source Port, Destination Port and Transport +Protocol. If the filter is empty, all flow records will be dumped. + +The command provides a compact display of the flow records in the default table +output format, which contains the flow key, source pod name, destination pod name, +source pod namespace, destination pod namespace and destination service name for +each flow record. Using the `json` or `yaml` antctl output format will include +output flow record information in a structured format, and will include more +information about each flow record. `antctl get flowrecords --help` shows the +usage of the command. This section lists a few dumping flow records command +examples. + +```bash +# Get the list of all flow records +antctl get flowrecords +# Get the list of flow records with a complete filter and output in json format +antctl get flowrecords --srcip 10.0.0.1 --dstip 10.0.0.2 --proto 6 --srcport 1234 --dstport 5678 -o json +# Get the list of flow records with a partial filter, e.g. source address and source port +antctl get flowrecords --srcip 10.0.0.1 --srcport 1234 +``` + +Example outputs of dumping flow records: + +```bash +$ antctl get flowrecords --srcip 10.10.1.4 --dstip 10.10.0.2 +SRC_IP DST_IP SPORT DPORT PROTO SRC_POD DST_POD SRC_NS DST_NS SERVICE +10.10.1.4 10.10.0.2 38581 53 17 flow-aggregator-67dc8ddfc8-zx8sg coredns-78fcd69978-7vc6k flow-aggregator kube-system kube-system/kube-dns:dns +10.10.1.4 10.10.0.2 56505 53 17 flow-aggregator-67dc8ddfc8-zx8sg coredns-78fcd69978-7vc6k flow-aggregator kube-system kube-system/kube-dns:dns + +$ antctl get flowrecords --srcip 10.10.0.1 --srcport 50497 -o json +[ + { + "destinationClusterIPv4": "0.0.0.0", + "destinationIPv4Address": "10.10.1.2", + "destinationNodeName": "k8s-node-worker-1", + "destinationPodName": "coredns-78fcd69978-x2twv", + "destinationPodNamespace": "kube-system", + "destinationServicePort": 0, + "destinationServicePortName": "", + "destinationTransportPort": 53, + "egressNetworkPolicyName": "", + "egressNetworkPolicyNamespace": "", + "egressNetworkPolicyRuleAction": 0, + "egressNetworkPolicyRuleName": "", + "egressNetworkPolicyType": 0, + "flowEndReason": 3, + "flowEndSeconds": 1635546893, + "flowStartSeconds": 1635546867, + "flowType": 2, + "ingressNetworkPolicyName": "", + "ingressNetworkPolicyNamespace": "", + "ingressNetworkPolicyRuleAction": 0, + "ingressNetworkPolicyRuleName": "", + "ingressNetworkPolicyType": 0, + "octetDeltaCount": 99, + "octetDeltaCountFromDestinationNode": 99, + "octetDeltaCountFromSourceNode": 0, + "octetTotalCount": 99, + "octetTotalCountFromDestinationNode": 99, + "octetTotalCountFromSourceNode": 0, + "packetDeltaCount": 1, + "packetDeltaCountFromDestinationNode": 1, + "packetDeltaCountFromSourceNode": 0, + "packetTotalCount": 1, + "packetTotalCountFromDestinationNode": 1, + "packetTotalCountFromSourceNode": 0, + "protocolIdentifier": 17, + "reverseOctetDeltaCount": 192, + "reverseOctetDeltaCountFromDestinationNode": 192, + "reverseOctetDeltaCountFromSourceNode": 0, + "reverseOctetTotalCount": 192, + "reverseOctetTotalCountFromDestinationNode": 192, + "reverseOctetTotalCountFromSourceNode": 0, + "reversePacketDeltaCount": 1, + "reversePacketDeltaCountFromDestinationNode": 1, + "reversePacketDeltaCountFromSourceNode": 0, + "reversePacketTotalCount": 1, + "reversePacketTotalCountFromDestinationNode": 1, + "reversePacketTotalCountFromSourceNode": 0, + "sourceIPv4Address": "10.10.0.1", + "sourceNodeName": "", + "sourcePodName": "", + "sourcePodNamespace": "", + "sourceTransportPort": 50497, + "tcpState": "" + } +] +``` + +#### Record metrics + +Flow Aggregator supports printing record metrics. The `antctl get recordmetrics` +command can print all metrics related to the Flow Aggregator. The metrics include +the following: + +* number of records received by the collector process in the Flow Aggregator +* number of records exported by the Flow Aggregator +* number of active flows that are being tracked +* number of exporters connected to the Flow Aggregator + +Example outputs of record metrics: + +```bash +RECORDS-EXPORTED RECORDS-RECEIVED FLOWS EXPORTERS-CONNECTED +46 118 7 2 +``` + +### Multi-cluster commands + +For information about Antrea Multi-cluster commands, please refer to the +[antctl Multi-cluster commands](./multicluster/antctl.md). + +### Multicast commands + +The `antctl get podmulticaststats [POD_NAME] [-n NAMESPACE]` command prints inbound +and outbound multicast statistics for each Pod. Note that IGMP packets are not counted. + +Example output of podmulticaststats: + +```bash +$ antctl get podmulticaststats + +NAMESPACE NAME INBOUND OUTBOUND +testmulticast-vw7gx5b9 test3-receiver-2 30 0 +testmulticast-vw7gx5b9 test3-sender-1 0 10 +``` + +### Showing memberlist state + +`antctl` agent command `get memberlist` (or `get ml`) prints the state of memberlist +cluster of Antrea Agent. + +```bash +$ antctl get memberlist + +NODE IP STATUS +worker1 172.18.0.4 Alive +worker2 172.18.0.3 Alive +worker3 172.18.0.2 Dead +``` + +### Upgrade existing objects of CRDs + +antctl supports upgrading existing objects of Antrea CRDs to the storage version. +The related sub-commands should be run out-of-cluster. Please ensure that the +kubeconfig file used by antctl has the necessary permissions. The required permissions +are listed in the following sample ClusterRole. + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: antctl +rules: + - apiGroups: + - apiextensions.k8s.io + resources: + - customresourcedefinitions + verbs: + - get + - list + - apiGroups: + - apiextensions.k8s.io + resources: + - customresourcedefinitions/status + verbs: + - update + - apiGroups: + - crd.antrea.io + resources: + - "*" + verbs: + - get + - list + - update +``` + +This command performs a dry-run to upgrade all existing objects of Antrea CRDs to +the storage version: + +```bash +antctl upgrade api-storage --dry-run +``` + +This command upgrades all existing objects of Antrea CRDs to the storage version: + +```bash +antctl upgrade api-storage +``` + +This command upgrades existing AntreaAgentInfo objects to the storage version: + +```bash +antctl upgrade api-storage --crds=antreaagentinfos.crd.antrea.io +``` + +This command upgrades existing Egress and Group objects to the storage version: + +```bash +antctl upgrade api-storage --crds=egresses.crd.antrea.io,groups.crd.antrea.io +``` + +If you encounter any errors related to permissions while running the commands, double-check +the permissions of the kubeconfig used by antctl. Ensure that the ClusterRole has the +required permissions. The following sample errors are caused by insufficient permissions: + +```bash +Error: failed to get CRD list: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "user" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope + +Error: externalippools.crd.antrea.io is forbidden: User "user" cannot list resource "externalippools" in API group "crd.antrea.io" at the cluster scope + +Error: error upgrading object prod-external-ip-pool of CRD "externalippools.crd.antrea.io": externalippools.crd.antrea.io "prod-external-ip-pool" is forbidden: User "user" cannot update resource "externalippools" in API group "crd.antrea.io" at the cluster scope + +Error: error updating CRD "externalippools.crd.antrea.io" status.storedVersion: customresourcedefinitions.apiextensions.k8s.io "externalippools.crd.antrea.io" is forbidden: User "user" cannot update resource "customresourcedefinitions/status" in API group "apiextensions.k8s.io" at the cluster scope +``` diff --git a/content/docs/v1.15.0/docs/antrea-agent-simulator.md b/content/docs/v1.15.0/docs/antrea-agent-simulator.md new file mode 100644 index 00000000..d2d265ea --- /dev/null +++ b/content/docs/v1.15.0/docs/antrea-agent-simulator.md @@ -0,0 +1,53 @@ +# Run Antrea agent simulator + +This document describes how to run the Antrea agent simulator. The simulator is +useful for Antrea scalability testing, without having to create a very large +cluster. + +## Build the images + +```bash +make build-scale-simulator +``` + +## Create the yaml file + +This demo uses 1 simulator, this command will create a yaml file +build/yamls/antrea-scale.yml + +```bash +make manifest-scale +``` + +The above yaml will create one simulated Node/Pod, to change the number of +instances, you can modify `spec.replicas` of the StatefulSet +`antrea-agent-simulator` in the yaml, or scale it via +`kubectl scale statefulset/antrea-agent-simulator -n kube-system --replicas=` +after deploying it. + +## Taint the simulator node + +To prevent Pods from being scheduled on the simulated Node(s), you can use the +following taint. + +```bash +kubectl taint -l 'antrea/instance=simulator' node mocknode=true:NoExecute +``` + +## Create secret for kubemark + +```bash +kubectl create secret generic kubeconfig --type=Opaque --namespace=kube-system --from-file=admin.conf= +``` + +## Apply the yaml file + +```bash +kubectl apply -f build/yamls/antrea-scale.yml +``` + +check the simulated Node: + + ```bash +kubectl get nodes -l 'antrea/instance=simulator' + ``` diff --git a/content/docs/v1.15.0/docs/antrea-ipam.md b/content/docs/v1.15.0/docs/antrea-ipam.md new file mode 100644 index 00000000..7bb6cbac --- /dev/null +++ b/content/docs/v1.15.0/docs/antrea-ipam.md @@ -0,0 +1,482 @@ +# Antrea IPAM Capabilities + + +* [Antrea IPAM Capabilities](#antrea-ipam-capabilities) + * [Running NodeIPAM within Antrea Controller](#running-nodeipam-within-antrea-controller) + * [Configuration](#configuration) + * [Antrea Flexible IPAM](#antrea-flexible-ipam) + * [Usage](#usage) + * [Enable AntreaIPAM feature gate and bridging mode](#enable-antreaipam-feature-gate-and-bridging-mode) + * [Create IPPool CR](#create-ippool-cr) + * [IPPool Annotations on Namespace](#ippool-annotations-on-namespace) + * [IPPool Annotations on Pod (available since Antrea 1.5)](#ippool-annotations-on-pod-available-since-antrea-15) + * [Persistent IP for StatefulSet Pod (available since Antrea 1.5)](#persistent-ip-for-statefulset-pod-available-since-antrea-15) + * [Data path behaviors](#data-path-behaviors) + * [Requirements for this Feature](#requirements-for-this-feature) + * [Flexible IPAM design](#flexible-ipam-design) + * [On IPPool CR create/update event](#on-ippool-cr-createupdate-event) + * [On StatefulSet create event](#on-statefulset-create-event) + * [On StatefulSet delete event](#on-statefulset-delete-event) + * [On Pod create](#on-pod-create) + * [On Pod delete](#on-pod-delete) + * [IPAM for Secondary Network](#ipam-for-secondary-network) + * [Prerequisites](#prerequisites) + * [CNI IPAM configuration](#cni-ipam-configuration) + * [Configuration with `NetworkAttachmentDefinition` CRD](#configuration-with-networkattachmentdefinition-crd) + * [`IPPool` CRD](#ippool-crd) + * [Secondary Network creation with Multus](#secondary-network-creation-with-multus) + + +## Running NodeIPAM within Antrea Controller + +NodeIPAM is a Kubernetes component, which manages IP address pool allocation per +each Node, when the Node initializes. + +On single stack deployments, NodeIPAM allocates a single IPv4 or IPv6 CIDR per +Node, while in dual stack deployments, NodeIPAM allocates two CIDRs per each +Node: one for each IP family. + +NodeIPAM is configured with a CIDR per each family, which it slices into smaller +per-Node CIDRs. When a Node is initialized, the CIDRs are set to the podCIDRs +attribute of the Node spec. + +Antrea NodeIPAM controller can be executed in scenarios where the +NodeIPAMController is disabled in kube-controller-manager. + +Note that running Antrea NodeIPAM while NodeIPAMController runs within +kube-controller-manager would cause conflicts and result in an unstable +behavior. + +### Configuration + +Antrea Controller NodeIPAM configuration items are grouped under `nodeIPAM` +dictionary key. + +NodeIPAM dictionary contains the following items: + +- `enableNodeIPAM`: Enable the integrated NodeIPAM controller within the Antrea +controller. Default is false. + +- `clusterCIDRs`: CIDR ranges for Pods in cluster. String array containing single +CIDR range, or multiple ranges. The CIDRs could be either IPv4 or IPv6. At most +one CIDR may be specified for each IP family. Example values: +`[172.100.0.0/16]`, `[172.100.0.0/20, fd00:172:100::/60]`. + +- `serviceCIDR`: CIDR range for IPv4 Services in cluster. It is not necessary to +specify it when there is no overlap with clusterCIDRs. + +- `serviceCIDRv6`: CIDR range for IPv6 Services in cluster. It is not necessary to + specify it when there is no overlap with clusterCIDRs. + +- `nodeCIDRMaskSizeIPv4`: Mask size for IPv4 Node CIDR in IPv4 or dual-stack +cluster. Valid range is 16 to 30. Default is 24. + +- `nodeCIDRMaskSizeIPv6`: Mask size for IPv6 Node CIDR in IPv6 or dual-stack +cluster. Valid range is 64 to 126. Default is 64. + +Below is a sample of needed changes in the Antrea deployment YAML: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: antrea-config + namespace: kube-system +data: + antrea-controller.conf: | + nodeIPAM: + enableNodeIPAM: true + clusterCIDRs: [172.100.0.0/16] +``` + +When running Antrea NodeIPAM in a particular version or scenario, you may need to +be aware of the following: + +* Prior to v1.12, a feature gate, `NodeIPAM` must also be enabled for + `antrea-controller`. +* Prior to v1.13, running Antrea NodeIPAM without kube-proxy is not supported. + Starting with v1.13, the `kubeAPIServerOverride` option in the `antrea-controller` + configuration must be set to the address of Kubernetes apiserver when kube-proxy + is not deployed. + +## Antrea Flexible IPAM + +Antrea supports flexible control over Pod IP addressing since version 1.4. Pod +IP addresses can be allocated from an `IPPool`. When a Pod's IP is allocated +from an IPPool, the traffic from the Pod to Pods on another Node or from the Pod to +external network will be sent to the underlay network through the Node's transport +network interface, and will be forwarded/routed by the underlay network. We also +call this forwarding mode `bridging mode`. + +`IPPool` CRD defines a desired set of IP ranges and VLANs. An `IPPool` can be annotated +to Namespace, Pod and PodTemplate of StatefulSet/Deployment. Then Antrea will +manage IP address assignment for corresponding Pods according to `IPPool` spec. +Note that the IP pool annotation cannot be updated or deleted without recreating +the resource. An `IPPool` can be extended, but cannot be shrunk if already +assigned to a resource. The IP ranges of IPPools must not overlap, otherwise it +would lead to undefined behavior. + +Regular `Subnet per Node` IPAM will continue to be used for resources without the +IPPool annotation, or when the `AntreaIPAM` feature is disabled. + +### Usage + +#### Enable AntreaIPAM feature gate and bridging mode + +To enable flexible IPAM, you need to enable the `AntreaIPAM` feature gate for +both `antrea-controller` and `antrea-agent`, and set the `enableBridgingMode` +configuration parameter of `antrea-agent` to `true`. + +When Antrea is installed from YAML, the needed changes in the Antrea +ConfigMap `antrea-config` YAML are as below: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: antrea-config + namespace: kube-system +data: + antrea-controller.conf: | + featureGates: + AntreaIPAM: true + antrea-agent.conf: | + featureGates: + AntreaIPAM: true + enableBridgingMode: true + trafficEncapMode: "noEncap" + noSNAT: true +``` + +Alternatively, you can use the following helm install/upgrade command to configure +the above options: + + ```bash + helm upgrade --install antrea antrea/antrea --namespace kube-system --set +enableBridgingMode=true,featureGates.AntreaIPAM=true,trafficEncapMode=noEncap,noSNAT=true + ``` + +#### Create IPPool CR + +The following example YAML manifest creates an IPPool CR. + +```yaml +apiVersion: "crd.antrea.io/v1alpha2" +kind: IPPool +metadata: + name: pool1 +spec: + ipVersion: 4 + ipRanges: + - start: "10.2.0.12" + end: "10.2.0.20" + gateway: "10.2.0.1" + prefixLength: 24 + vlan: 2 # Default is 0 (untagged). Valid value is 0~4095. +``` + +#### IPPool Annotations on Namespace + +The following example YAML manifest creates a Namespace to allocate Pod IPs from the IP pool. + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: namespace1 + annotations: + ipam.antrea.io/ippools: 'pool1' +``` + +#### IPPool Annotations on Pod (available since Antrea 1.5) + +Since Antrea v1.5.0, Pod IPPool annotation is supported and has a higher +priority than the Namespace IPPool annotation. This annotation can be added to +`PodTemplate` of a controller resource such as StatefulSet and Deployment. + +Pod IP annotation is supported for a single Pod to specify a fixed IP for the Pod. + +Examples of annotations on a Pod or PodTemplate: + +```yaml +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: statefulset1 +spec: + replicas: 1 # Do not increase replicas if there is pod-ips annotation in PodTemplate + template: + metadata: + annotations: + ipam.antrea.io/ippools: 'sts-ip-pool1' # This annotation will be set automatically on all Pods managed by this resource + ipam.antrea.io/pod-ips: '' +``` + +```yaml +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: statefulset1 +spec: + replicas: 4 + template: + metadata: + annotations: + ipam.antrea.io/ippools: 'sts-ip-pool1' # This annotation will be set automatically on all Pods managed by this resource + # Do not add pod-ips annotation to PodTemplate if there is more than 1 replica +``` + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: pod1 + annotations: + ipam.antrea.io/ippools: 'pod-ip-pool1' +``` + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: pod1 + annotations: + ipam.antrea.io/ippools: 'pod-ip-pool1' + ipam.antrea.io/pod-ips: '' +``` + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: pod1 + annotations: + ipam.antrea.io/pod-ips: '' +``` + +#### Persistent IP for StatefulSet Pod (available since Antrea 1.5) + +A StatefulSet Pod's IP will be kept after Pod restarts, when the IP is allocated from the +annotated IPPool. + +### Data path behaviors + +When `AntreaIPAM` is enabled, `antrea-agent` will connect the Node's network interface +to the OVS bridge at startup, and it will detach the interface from the OVS bridge and +restore its configurations at exit. Node may lose network connection when `antrea-agent` +or OVS daemons are stopped unexpectedly, which can be recovered by rebooting the Node. +`AntreaIPAM` Pods' traffic will not be routed by local Node's network stack. + +Traffic from `AntreaIPAM` Pods without VLAN, regular `Subnet per Node` IPAM Pods, and K8s +Nodes is recognized as VLAN 0 (untagged). + +Traffic to a local Pod in the Pod's VLAN will be sent to the Pod's OVS port directly, +after the destination MAC is rewritten to the Pod's MAC address. This includes +`AntreaIPAM` Pods and regular `Subnet per Node` IPAM Pods, even when they are not in the +same subnet. Traffic to a Pod in different VLAN will be sent to the underlay network, +where the underlay router will route the traffic to the destination VLAN. + +### Requirements for this Feature + +As of now, this feature is supported on Linux Nodes, with IPv4, `system` OVS datapath +type, `noEncap`, `noSNAT` traffic mode, and `AntreaProxy` feature enabled. Configuration +with `ProxyAll` feature enabled is not verified. + +The IPs in the `IPPools` without VLAN must be in the same underlay subnet as the Node +IP, because inter-Node traffic of AntreaIPAM Pods is forwarded by the Node network. +`IPPools` with VLAN must not overlap with other network subnets, and the underlay network +router should provide the network connectivity for these VLANs. Only a single IP pool can +be included in the Namespace annotation. In the future, annotation of up to two pools for +IPv4 and IPv6 respectively will be supported. + +### Flexible IPAM design + +When the `AntreaIPAM` feature gate is enabled, `antrea-controller` will watch IPPool CRs and +StatefulSets from `kube-apiserver`. + +#### On IPPool CR create/update event + +`antrea-controller` will update IPPool counters, and periodically clean up stale IP addresses. + +#### On StatefulSet create event + +`antrea-controller` will check the Antrea IPAM annotations on the StatefullSet, and preallocate +IPs from the specified IPPool for the StatefullSet Pods + +#### On StatefulSet delete event + +`antrea-controller` will clean up IP allocations for this StatefulSet. + +#### On Pod create + +`antrea-agent` will receive a CNI add request, and it will then check the Antrea IPAM annotations +and allocate an IP for the Pod, which can be a pre-allocated IP StatefulSet IP, a user-specified +IP, or the next available IP in the specified IPPool. + +#### On Pod delete + +`antrea-agent` will receive a CNI del request and release the IP allocation from the IPPool. +If the IP is a pre-allocated StatefulSet IP, it will stay in the pre-allocated status thus the Pod +will get same IP after recreated. + +## IPAM for Secondary Network + +With the AntreaIPAM feature, Antrea can allocate IPs for Pod secondary networks. At the +moment, AntreaIPAM supports secondary networks managed by [Multus](https://github.com/k8snetworkplumbingwg/multus-cni), +we will add support for [secondary networks managed by Antrea](feature-gates.md#secondarynetwork) +in the future. + +### Prerequisites + +The IPAM capability for secondary network was added in Antrea version 1.7. It +requires the `AntreaIPAM` feature gate to be enabled on both `antrea-controller` +and `antrea-agent`, as `AntreaIPAM` is still an alpha feature at this moment and +is not enabled by default. + +### CNI IPAM configuration + +To configure Antrea IPAM, `antrea` should be specified as the IPAM plugin in the +the CNI IPAM configuration, and at least one Antrea IPPool should be specified +in the `ippools` field. IPs will be allocated from the specified IPPool(s) for +the secondary network. + +```json +{ + "cniVersion": "0.3.0", + "name": "ipv4-net-1", + "type": "macvlan", + "master": "eth0", + "mode": "bridge", + "ipam": { + "type": "antrea", + "ippools": [ "ipv4-pool-1" ] + } +} +``` + +Multiple IPPools can be specified to allocate multiple IPs from each IPPool for +the secondary network. For example, you can specify one IPPool to allocate an +IPv4 address and another IPPool to allocate an IPv6 address in the dual-stack +case. + +```json +{ + "cniVersion": "0.3.0", + "name": "dual-stack-net-1", + "type": "macvlan", + "master": "eth0", + "mode": "bridge", + "ipam": { + "type": "antrea", + "ippools": [ "ipv4-pool-1", "ipv6-pool-1" ] + } +} +``` + +Additionally, Antrea IPAM also supports the same configuration of static IP +addresses, static routes, and DNS settings, as what is supported by the +[static IPAM plugin](https://www.cni.dev/plugins/current/ipam/static). The +following example requests an IP from an IPPool and also specifies two +additional static IP addresses. It also includes static routes and DNS settings. + +```json +{ + "cniVersion": "0.3.0", + "name": "pool-and-static-net-1", + "type": "bridge", + "bridge": "br0", + "ipam": { + "type": "antrea", + "ippools": [ "ipv4-pool-1" ], + "addresses": [ + { + "address": "10.10.0.1/24", + "gateway": "10.10.0.254" + }, + { + "address": "3ffe:ffff:0:01ff::1/64", + "gateway": "3ffe:ffff:0::1" + } + ], + "routes": [ + { "dst": "0.0.0.0/0" }, + { "dst": "192.168.0.0/16", "gw": "10.10.5.1" }, + { "dst": "3ffe:ffff:0:01ff::1/64" } + ], + "dns": { + "nameservers" : ["8.8.8.8"], + "domain": "example.com", + "search": [ "example.com" ] + } + } +} +``` + +The CNI IPAM configuration can include only static addresses without IPPools, if +only static IP addresses are needed. + +### Configuration with `NetworkAttachmentDefinition` CRD + +CNI and IPAM configuration of a secondary network is typically defined with the +`NetworkAttachmentDefinition` CRD. For example: + +```yaml +apiVersion: "k8s.cni.cncf.io/v1" +kind: NetworkAttachmentDefinition +metadata: + name: ipv4-net-1 +spec: + { + "cniVersion": "0.3.0", + "type": "macvlan", + "master": "eth0", + "mode": "bridge", + "ipam": { + "type": "antrea", + "ippools": [ "ipv4-pool-1" ] + } + } +``` + +## `IPPool` CRD + +Antrea IP pools are defined with the `IPPool` CRD. The following two examples +define an IPv4 and an IPv6 IP pool respectively. + +```yaml +apiVersion: "crd.antrea.io/v1alpha2" +kind: IPPool +metadata: + name: ipv4-pool-1 +spec: + ipVersion: 4 + ipRanges: + - cidr: "10.10.1.0/26" + gateway: "10.10.1.1" + prefixLength: 24 +``` + +```yaml +apiVersion: "crd.antrea.io/v1alpha2" +kind: IPPool +metadata: + name: ipv6-pool-1 +spec: + ipVersion: 6 + ipRanges: + - start: "3ffe:ffff:1:01ff::0100" + end: "3ffe:ffff:1:01ff::0200" + gateway: "3ffe:ffff:1:01ff::1" + prefixLength: 64 +``` + +VLAN ID in the IP range subnet definition of `IPPool` CRD is not supported for +secondary network IPAM. + +### Secondary Network creation with Multus + +To leverage Antrea for secondary network IPAM, Antrea must be used as the CNI +for the Pods' primary network, while the secondary networks are implemented by +other CNIs which are managed by Multus. The [Antrea + Multus guide](cookbooks/multus) +talks about how to use Antrea with Multus, including the option of using Antrea +IPAM for secondary networks. diff --git a/content/docs/v1.15.0/docs/antrea-l7-network-policy.md b/content/docs/v1.15.0/docs/antrea-l7-network-policy.md new file mode 100644 index 00000000..3b1aaa2c --- /dev/null +++ b/content/docs/v1.15.0/docs/antrea-l7-network-policy.md @@ -0,0 +1,367 @@ +# Antrea Layer 7 NetworkPolicy + +## Table of Contents + + +- [Introduction](#introduction) +- [Prerequisites](#prerequisites) +- [Usage](#usage) + - [HTTP](#http) + - [More examples](#more-examples) + - [TLS](#tls) + - [More examples](#more-examples-1) + - [Logs](#logs) +- [Limitations](#limitations) + + +## Introduction + +NetworkPolicy was initially used to restrict network access at layer 3 (Network) and 4 (Transport) in the OSI model, +based on IP address, transport protocol, and port. Securing applications at IP and port level provides limited security +capabilities, as the service an application provides is either entirely exposed to a client or not accessible by that +client at all. Starting with v1.10, Antrea introduces support for layer 7 NetworkPolicy, an application-aware policy +which provides fine-grained control over the network traffic beyond IP, transport protocol, and port. It enables users +to protect their applications by specifying how they are allowed to communicate with others, taking into account +application context. For example, you can enforce policies to: + +- Grant access of privileged URLs to specific clients while make other URLs publicly accessible. +- Prevent applications from accessing unauthorized domains. +- Block network traffic using an unauthorized application protocol regardless of port used. + +This guide demonstrates how to configure layer 7 NetworkPolicy. + +## Prerequisites + +Layer 7 NetworkPolicy was introduced in v1.10 as an alpha feature and is disabled by default. A feature gate, +`L7NetworkPolicy`, must be enabled in antrea-controller.conf and antrea-agent.conf in the `antrea-config` ConfigMap. +Additionally, due to the constraint of the application detection engine, TX checksum offloading must be disabled via the +`disableTXChecksumOffload` option in antrea-agent.conf for the feature to work. An example configuration is as below: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: antrea-config + namespace: kube-system +data: + antrea-agent.conf: | + disableTXChecksumOffload: true + featureGates: + L7NetworkPolicy: true + antrea-controller.conf: | + featureGates: + L7NetworkPolicy: true +``` + +Alternatively, you can use the following helm installation command to configure the above options: + +```bash +helm install antrea antrea/antrea --namespace kube-system --set featureGates.L7NetworkPolicy=true,disableTXChecksumOffload=true +``` + +## Usage + +There isn't a separate resource type for layer 7 NetworkPolicy. It is one kind of Antrea-native policies, which has the +`l7Protocols` field specified in the rules. Like layer 3 and layer 4 policies, the `l7Protocols` field can be specified +for ingress and egress rules in Antrea ClusterNetworkPolicy and Antrea NetworkPolicy. It can be used with the `from` or +`to` field to select the network peer, and the `ports` to select the transport protocol and/or port for which the layer +7 rule applies to. The `action` of a layer 7 rule can only be `Allow`. + +**Note**: Any traffic matching the layer 3/4 criteria (specified by `from`, `to`, and `port`) of a layer 7 rule will be +forwarded to an application-aware engine for protocol detection and rule enforcement, and the traffic will be allowed if +the layer 7 criteria is also matched, otherwise it will be dropped. Therefore, any rules after a layer 7 rule will not +be enforced for the traffic that match the layer 7 rule's layer 3/4 criteria. + +As of now, the only supported layer 7 protocol is HTTP. Support for more protocols may be added in the future and we +welcome feature requests for protocols that you are interested in. + +### HTTP + +An example layer 7 NetworkPolicy for the HTTP protocol is like below: + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: NetworkPolicy +metadata: + name: ingress-allow-http-request-to-api-v2 +spec: + priority: 5 + tier: application + appliedTo: + - podSelector: + matchLabels: + app: web + ingress: + - name: allow-http # Allow inbound HTTP GET requests to "/api/v2" from Pods with label "app=client". + action: Allow # All other traffic from these Pods will be automatically dropped, and subsequent rules will not be considered. + from: + - podSelector: + matchLabels: + app: client + l7Protocols: + - http: + path: "/api/v2/*" + host: "foo.bar.com" + method: "GET" + - name: drop-other # Drop all other inbound traffic (i.e., from Pods without label "app=client" or from external clients). + action: Drop +``` + +**path**: The `path` field represents the URI path to match. Both exact matches and wildcards are supported, e.g. +`/api/v2/*`, `*/v2/*`, `/index.html`. If not set, the rule matches all URI paths. + +**host**: The `host` field represents the hostname present in the URI or the HTTP Host header to match. It does not +contain the port associated with the host. Both exact matches and wildcards are supported, e.g. `*.foo.com`, `*.foo.*`, +`foo.bar.com`. If not set, the rule matches all hostnames. + +**method**: The `method` field represents the HTTP method to match. It could be GET, POST, PUT, HEAD, DELETE, TRACE, +OPTIONS, CONNECT and PATCH. If not set, the rule matches all methods. + +#### More examples + +The following NetworkPolicy grants access of privileged URLs to specific clients while make other URLs publicly +accessible: + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: NetworkPolicy +metadata: + name: allow-privileged-url-to-admin-role +spec: + priority: 5 + tier: securityops + appliedTo: + - podSelector: + matchLabels: + app: web + ingress: + - name: for-admin # Allow inbound HTTP GET requests to "/admin" and "/public" from Pods with label "role=admin". + action: Allow + from: + - podSelector: + matchLabels: + role: admin + l7Protocols: + - http: + path: "/admin/*" + - http: + path: "/public/*" + - name: for-public # Allow inbound HTTP GET requests to "/public" from Pods with label "app=client". + action: Allow # All other inbound traffic will be automatically dropped. + l7Protocols: + - http: + path: "/public/*" +``` + +The following NetworkPolicy prevents applications from accessing unauthorized domains: + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: allow-web-access-to-internal-domain +spec: + priority: 5 + tier: securityops + appliedTo: + - podSelector: + matchLabels: + egress-restriction: internal-domain-only + egress: + - name: allow-dns # Allow outbound DNS requests. + action: Allow + ports: + - protocol: TCP + port: 53 + - protocol: UDP + port: 53 + - name: allow-http-only # Allow outbound HTTP requests towards "*.bar.com". + action: Allow # As the rule's "to" and "ports" are empty, which means it selects traffic to any network + l7Protocols: # peer's any port using any transport protocol, all outbound HTTP requests towards other + - http: # domains and non-HTTP requests will be automatically dropped, and subsequent rules will + host: "*.bar.com" # not be considered. +``` + +The following NetworkPolicy blocks network traffic using an unauthorized application protocol regardless of the port used. + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: NetworkPolicy +metadata: + name: allow-http-only +spec: + priority: 5 + tier: application + appliedTo: + - podSelector: + matchLabels: + app: web + ingress: + - name: http-only # Allow inbound HTTP requests only. + action: Allow # As the rule's "from" and "ports" are empty, which means it selects traffic from any network + l7Protocols: # peer to any port of the Pods this policy applies to, all inbound non-HTTP requests will be + - http: {} # automatically dropped, and subsequent rules will not be considered. +``` + +### TLS + +An example layer 7 NetworkPolicy for the TLS protocol is like below: + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: NetworkPolicy +metadata: + name: ingress-allow-tls-handshake +spec: + priority: 5 + tier: application + appliedTo: + - podSelector: + matchLabels: + app: web + ingress: + - name: allow-tls # Allow inbound TLS/SSL handshake packets to server name "foo.bar.com" from Pods with label "app=client". + action: Allow # All other traffic from these Pods will be automatically dropped, and subsequent rules will not be considered. + from: + - podSelector: + matchLabels: + app: client + l7Protocols: + - tls: + sni: "foo.bar.com" + - name: drop-other # Drop all other inbound traffic (i.e., from Pods without label "app=client" or from external clients). + action: Drop +``` + +**sni**: The `sni` field matches the TLS/SSL Server Name Indication (SNI) field in the TLS/SSL handshake process. Both +exact matches and wildcards are supported, e.g. `*.foo.com`, `*.foo.*`, `foo.bar.com`. If not set, the rule matches all names. + +#### More examples + +The following NetworkPolicy prevents applications from accessing unauthorized SSL/TLS server names: + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: allow-tls-handshake-to-internal +spec: + priority: 5 + tier: securityops + appliedTo: + - podSelector: + matchLabels: + egress-restriction: internal-tls-only + egress: + - name: allow-dns # Allow outbound DNS requests. + action: Allow + ports: + - protocol: TCP + port: 53 + - protocol: UDP + port: 53 + - name: allow-tls-only # Allow outbound SSL/TLS handshake packets towards "*.bar.com". + action: Allow # As the rule's "to" and "ports" are empty, which means it selects traffic to any network + l7Protocols: # peer's any port of any transport protocol, all outbound SSL/TLS handshake packets towards + - tls: # other server names and non-SSL/non-TLS handshake packets will be automatically dropped, + sni: "*.bar.com" # and subsequent rules will not be considered. +``` + +The following NetworkPolicy blocks network traffic using an unauthorized application protocol regardless of the port used. + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: NetworkPolicy +metadata: + name: allow-tls-only +spec: + priority: 5 + tier: application + appliedTo: + - podSelector: + matchLabels: + app: web + ingress: + - name: tls-only # Allow inbound SSL/TLS handshake packets only. + action: Allow # As the rule's "from" and "ports" are empty, which means it selects traffic from any network + l7Protocols: # peer to any port of the Pods this policy applies to, all inbound non-SSL/non-TLS handshake + - tls: {} # packets will be automatically dropped, and subsequent rules will not be considered. +``` + +### Logs + +Layer 7 traffic that matches the NetworkPolicy will be logged in an event +triggered log file (`/var/log/antrea/networkpolicy/l7engine/eve-YEAR-MONTH-DAY.json`). +The event type for this log is `alert`. If `enableLogging` is set for the rule, +packets that match the rule will also be logged in addition to the event with +event type `packet`. Below is an example of the two event types. + +Deny ingress from client (10.10.1.5) to web (10.10.1.4/admin) + +```json +{ + "timestamp": "2023-03-09T20:00:28.210821+0000", + "flow_id": 627175734391745, + "in_iface": "antrea-l7-tap0", + "event_type": "alert", + "vlan": [ + 1 + ], + "src_ip": "10.10.1.5", + "src_port": 43352, + "dest_ip": "10.10.1.4", + "dest_port": 80, + "proto": "TCP", + "alert": { + "action": "blocked", + "gid": 1, + "signature_id": 1, + "rev": 0, + "signature": "Reject by AntreaClusterNetworkPolicy:test-l7-ingress", + "category": "", + "severity": 3, + "tenant_id": 1 + }, + "http": { + "hostname": "10.10.1.4", + "url": "/admin", + "http_user_agent": "curl/7.74.0", + "http_method": "GET", + "protocol": "HTTP/1.1", + "length": 0 + }, + "app_proto": "http", + "flow": { + "pkts_toserver": 3, + "pkts_toclient": 1, + "bytes_toserver": 284, + "bytes_toclient": 74, + "start": "2023-03-09T20:00:28.209857+0000" + } +} +``` + +```json +{ + "timestamp": "2023-03-09T20:00:28.225016+0000", + "flow_id": 627175734391745, + "in_iface": "antrea-l7-tap0", + "event_type": "packet", + "vlan": [ + 1 + ], + "src_ip": "10.10.1.4", + "src_port": 80, + "dest_ip": "10.10.1.5", + "dest_port": 43352, + "proto": "TCP", + "packet": "/lhtPRglzmQvxnJoCABFAAAoUGYAAEAGFE4KCgEECgoBBQBQqVhIGzbi/odenlAUAfsR7QAA", + "packet_info": { + "linktype": 1 + } +} +``` + +## Limitations + +This feature is currently only supported for Nodes running Linux. diff --git a/content/docs/v1.15.0/docs/antrea-network-policy.md b/content/docs/v1.15.0/docs/antrea-network-policy.md new file mode 100644 index 00000000..5fdd9a2d --- /dev/null +++ b/content/docs/v1.15.0/docs/antrea-network-policy.md @@ -0,0 +1,1841 @@ +# Antrea Network Policy CRDs + +## Table of Contents + + +- [Summary](#summary) +- [Tier](#tier) + - [Tier CRDs](#tier-crds) + - [Static tiers](#static-tiers) + - [kubectl commands for Tier](#kubectl-commands-for-tier) +- [Antrea ClusterNetworkPolicy](#antrea-clusternetworkpolicy) + - [The Antrea ClusterNetworkPolicy resource](#the-antrea-clusternetworkpolicy-resource) + - [ACNP with stand-alone selectors](#acnp-with-stand-alone-selectors) + - [ACNP with ClusterGroup reference](#acnp-with-clustergroup-reference) + - [ACNP for complete Pod isolation in selected Namespaces](#acnp-for-complete-pod-isolation-in-selected-namespaces) + - [ACNP for strict Namespace isolation](#acnp-for-strict-namespace-isolation) + - [ACNP for default zero-trust cluster security posture](#acnp-for-default-zero-trust-cluster-security-posture) + - [ACNP for toServices rule](#acnp-for-toservices-rule) + - [ACNP for ICMP traffic](#acnp-for-icmp-traffic) + - [ACNP for IGMP traffic](#acnp-for-igmp-traffic) + - [ACNP for multicast egress traffic](#acnp-for-multicast-egress-traffic) + - [ACNP for HTTP traffic](#acnp-for-http-traffic) + - [ACNP for Kubernetes Node traffic](#acnp-for-kubernetes-node-traffic) + - [ACNP with log settings](#acnp-with-log-settings) + - [Behavior of to and from selectors](#behavior-of-to-and-from-selectors) + - [Key differences from K8s NetworkPolicy](#key-differences-from-k8s-networkpolicy) + - [kubectl commands for Antrea ClusterNetworkPolicy](#kubectl-commands-for-antrea-clusternetworkpolicy) +- [Antrea NetworkPolicy](#antrea-networkpolicy) + - [The Antrea NetworkPolicy resource](#the-antrea-networkpolicy-resource) + - [Key differences from Antrea ClusterNetworkPolicy](#key-differences-from-antrea-clusternetworkpolicy) + - [Antrea NetworkPolicy with Group reference](#antrea-networkpolicy-with-group-reference) + - [kubectl commands for Antrea NetworkPolicy](#kubectl-commands-for-antrea-networkpolicy) +- [Antrea-native Policy ordering based on priorities](#antrea-native-policy-ordering-based-on-priorities) + - [Ordering based on Tier priority](#ordering-based-on-tier-priority) + - [Ordering based on policy priority](#ordering-based-on-policy-priority) + - [Rule enforcement based on priorities](#rule-enforcement-based-on-priorities) +- [Advanced peer selection mechanisms of Antrea-native Policies](#advanced-peer-selection-mechanisms-of-antrea-native-policies) + - [Selecting Namespace by Name](#selecting-namespace-by-name) + - [K8s clusters with version 1.21 and above](#k8s-clusters-with-version-121-and-above) + - [K8s clusters with version 1.20 and below](#k8s-clusters-with-version-120-and-below) + - [Selecting Pods in the same Namespace with Self](#selecting-pods-in-the-same-namespace-with-self) + - [FQDN based filtering](#fqdn-based-filtering) + - [Node Selector](#node-selector) + - [toServices egress rules](#toservices-egress-rules) + - [ServiceAccount based selection](#serviceaccount-based-selection) + - [Apply to NodePort Service](#apply-to-nodeport-service) +- [ClusterGroup](#clustergroup) + - [ClusterGroup CRD](#clustergroup-crd) + - [kubectl commands for ClusterGroup](#kubectl-commands-for-clustergroup) +- [Group](#group) + - [Group CRD](#group-crd) + - [Restrictions and Key differences from ClusterGroup](#restrictions-and-key-differences-from-clustergroup) + - [kubectl commands for Group](#kubectl-commands-for-group) +- [RBAC](#rbac) +- [Notes and constraints](#notes-and-constraints) + + +## Summary + +Antrea supports standard K8s NetworkPolicies to secure ingress/egress traffic for +Pods. These NetworkPolicies are written from an application developer's perspective, +hence they lack the ability to gain a finer-grained control over the security +policies that a cluster administrator would require. This document describes a +few new CRDs supported by Antrea to provide the administrator with more control +over security within the cluster, and which are meant to co-exist with and +complement the K8s NetworkPolicy. + +Starting with Antrea v1.0, Antrea-native policies are enabled by default, which +means that no additional configuration is required in order to use the +Antrea-native policy CRDs. + +## Tier + +Antrea supports grouping Antrea-native policy CRDs together in a tiered fashion +to provide a hierarchy of security policies. This is achieved by setting the +`tier` field when defining an Antrea-native policy CRD (e.g. an Antrea +ClusterNetworkPolicy object) to the appropriate Tier name. Each Tier has a +priority associated with it, which determines its relative order among other Tiers. + +**Note**: K8s NetworkPolicies will be enforced once all policies in all Tiers (except +for the baseline Tier) have been enforced. For more information, refer to the following +[Static Tiers section](#static-tiers) + +### Tier CRDs + +Creating Tiers as CRDs allows users the flexibility to create and delete +Tiers as per their preference i.e. not be bound to 5 static tiering options +as was the case initially. + +An example Tier might look like this: + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: Tier +metadata: + name: mytier +spec: + priority: 10 + description: "my custom tier" +``` + +Tiers have the following characteristics: + +- Policies can associate themselves with an existing Tier by setting the `tier` + field in an Antrea NetworkPolicy CRD spec to the Tier's name. +- A Tier must exist before an Antrea-native policy can reference it. +- Policies associated with higher ordered (low `priority` value) Tiers are + enforced first. +- No two Tiers can be created with the same priority. +- Updating the Tier's `priority` field is unsupported. +- Deleting Tier with existing references from policies is not allowed. + +### Static tiers + +On startup, antrea-controller will create 5 static, read-only Tier CRD resources +corresponding to the static tiers for default consumption, as well as a "baseline" +Tier CRD object, that will be enforced after developer-created K8s NetworkPolicies. +The details for these Tiers are shown below: + +```text + Emergency -> Tier name "emergency" with priority "50" + SecurityOps -> Tier name "securityops" with priority "100" + NetworkOps -> Tier name "networkops" with priority "150" + Platform -> Tier name "platform" with priority "200" + Application -> Tier name "application" with priority "250" + Baseline -> Tier name "baseline" with priority "253" +``` + +Any Antrea-native policy CRD referencing a static tier in its spec will now internally +reference the corresponding Tier resource, thus maintaining the order of enforcement. + +The static Tier CRD Resources are created as follows in the relative order of +precedence compared to K8s NetworkPolicies: + +```text + Emergency > SecurityOps > NetworkOps > Platform > Application > K8s NetworkPolicy > Baseline +``` + +Thus, all Antrea-native Policy resources associated with the "emergency" Tier will be +enforced before any Antrea-native Policy resource associated with any other +Tiers, until a match occurs, in which case the policy rule's `action` will be +applied. **Any Antrea-native Policy resource without a `tier` name set in its spec +will be associated with the "application" Tier.** Policies associated with the first +5 static, read-only Tiers, as well as with all the custom Tiers created with a priority +value lower than 250 (priority values greater than or equal to 250 are not allowed +for custom Tiers), will be enforced before K8s NetworkPolicies. + +Policies created in the "baseline" Tier, on the other hand, will have lower precedence +than developer-created K8s NetworkPolicies, which comes in handy when administrators +want to enforce baseline policies like "default-deny inter-namespace traffic" for some +specific Namespace, while still allowing individual developers to lift the restriction +if needed using K8s NetworkPolicies. + +Note that baseline policies cannot counteract the isolated Pod behavior provided by +K8s NetworkPolicies. To read more about this Pod isolation behavior, refer to [this +document](https://kubernetes.io/docs/concepts/services-networking/network-policies/#the-two-sorts-of-pod-isolation). +If a Pod becomes isolated because a K8s NetworkPolicy is applied to it, and the policy +does not explicitly allow communications with another Pod, this behavior cannot be changed +by creating an Antrea-native policy with an "allow" action in the "baseline" Tier. +For this reason, it generally does not make sense to create policies in the "baseline" +Tier with the "allow" action. + +### *kubectl* commands for Tier + +The following `kubectl` commands can be used to retrieve Tier resources: + +```bash + # Use long name + kubectl get tiers + + # Use long name with API Group + kubectl get tiers.crd.antrea.io + + # Use short name + kubectl get tr + + # Use short name with API Group + kubectl get tr.crd.antrea.io + + # Sort output by Tier priority + kubectl get tiers --sort-by=.spec.priority +``` + +All the above commands produce output similar to what is shown below: + +```text + NAME PRIORITY AGE + emergency 50 27h + securityops 100 27h + networkops 150 27h + platform 200 27h + application 250 27h +``` + +## Antrea ClusterNetworkPolicy + +Antrea ClusterNetworkPolicy (ACNP), one of the two Antrea-native policy CRDs +introduced, is a specification of how workloads within a cluster communicate +with each other and other external endpoints. The ClusterNetworkPolicy is +supposed to aid cluster admins to configure the security policy for the +cluster, unlike K8s NetworkPolicy, which is aimed towards developers to secure +their apps and affects Pods within the Namespace in which the K8s NetworkPolicy +is created. Rules belonging to ClusterNetworkPolicies are enforced before any +rule belonging to a K8s NetworkPolicy. + +### The Antrea ClusterNetworkPolicy resource + +Example ClusterNetworkPolicies might look like these: + +#### ACNP with stand-alone selectors + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: acnp-with-stand-alone-selectors +spec: + priority: 5 + tier: securityops + appliedTo: + - podSelector: + matchLabels: + role: db + - namespaceSelector: + matchLabels: + env: prod + ingress: + - action: Allow + from: + - podSelector: + matchLabels: + role: frontend + - podSelector: + matchLabels: + role: nondb + namespaceSelector: + matchLabels: + role: db + ports: + - protocol: TCP + port: 8080 + endPort: 9000 + - protocol: TCP + port: 6379 + name: AllowFromFrontend + egress: + - action: Drop + to: + - ipBlock: + cidr: 10.0.10.0/24 + ports: + - protocol: TCP + port: 5978 + name: DropToThirdParty +``` + +#### ACNP with ClusterGroup reference + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: acnp-with-cluster-groups +spec: + priority: 8 + tier: securityops + appliedTo: + - group: "test-cg-with-db-selector" # defined separately with a ClusterGroup resource + ingress: + - action: Allow + from: + - group: "test-cg-with-frontend-selector" # defined separately with a ClusterGroup resource + ports: + - protocol: TCP + port: 8080 + endPort: 9000 + - protocol: TCP + port: 6379 + name: AllowFromFrontend + egress: + - action: Drop + to: + - group: "test-cg-with-ip-block" # defined separately with a ClusterGroup resource + ports: + - protocol: TCP + port: 5978 + name: DropToThirdParty +``` + +#### ACNP for complete Pod isolation in selected Namespaces + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: isolate-all-pods-in-namespace +spec: + priority: 1 + tier: securityops + appliedTo: + - namespaceSelector: + matchLabels: + app: no-network-access-required + ingress: + - action: Drop # For all Pods in those Namespaces, drop and log all ingress traffic from anywhere + name: drop-all-ingress + egress: + - action: Drop # For all Pods in those Namespaces, drop and log all egress traffic towards anywhere + name: drop-all-egress +``` + +#### ACNP for strict Namespace isolation + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: strict-ns-isolation +spec: + priority: 5 + tier: securityops + appliedTo: + - namespaceSelector: # Selects all non-system Namespaces in the cluster + matchExpressions: + - {key: kubernetes.io/metadata.name, operator: NotIn, values: [kube-system]} + ingress: + - action: Pass + from: + - namespaces: + match: Self # Skip ACNP evaluation for traffic from Pods in the same Namespace + name: PassFromSameNS + - action: Drop + from: + - namespaceSelector: {} # Drop from Pods from all other Namespaces + name: DropFromAllOtherNS + egress: + - action: Pass + to: + - namespaces: + match: Self # Skip ACNP evaluation for traffic to Pods in the same Namespace + name: PassToSameNS + - action: Drop + to: + - namespaceSelector: {} # Drop to Pods from all other Namespaces + name: DropToAllOtherNS +``` + +#### ACNP for default zero-trust cluster security posture + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: default-cluster-deny +spec: + priority: 1 + tier: baseline + appliedTo: + - namespaceSelector: {} # Selects all Namespaces in the cluster + ingress: + - action: Drop + egress: + - action: Drop +``` + +#### ACNP for toServices rule + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: acnp-drop-to-services +spec: + priority: 5 + tier: securityops + appliedTo: + - podSelector: + matchLabels: + role: client + namespaceSelector: + matchLabels: + env: prod + egress: + - action: Drop + toServices: + - name: svcName + namespace: svcNamespace + name: DropToServices +``` + +#### ACNP for ICMP traffic + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: acnp-reject-ping-request +spec: + priority: 5 + tier: securityops + appliedTo: + - podSelector: + matchLabels: + role: server + namespaceSelector: + matchLabels: + env: prod + egress: + - action: Reject + protocols: + - icmp: + icmpType: 8 + icmpCode: 0 + name: DropPingRequest +``` + +#### ACNP for IGMP traffic + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: acnp-with-igmp-drop +spec: + priority: 5 + tier: securityops + appliedTo: + - podSelector: + matchLabels: + app: mcjoin6 + ingress: + - action: Drop + protocols: + - igmp: + igmpType: 0x11 + groupAddress: 224.0.0.1 + name: dropIGMPQuery + egress: + - action: Drop + protocols: + - igmp: + igmpType: 0x16 + groupAddress: 225.1.2.3 + name: dropIGMPReport +``` + +#### ACNP for multicast egress traffic + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: acnp-with-multicast-traffic-drop +spec: + priority: 5 + tier: securityops + appliedTo: + - podSelector: + matchLabels: + app: mcjoin6 + egress: + - action: Drop + to: + - ipBlock: + cidr: 225.1.2.3/32 + name: dropMcastUDPTraffic +``` + +#### ACNP for HTTP traffic + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: NetworkPolicy +metadata: + name: ingress-allow-http-request-to-api-v2 +spec: + priority: 5 + tier: application + appliedTo: + - podSelector: + matchLabels: + app: web + ingress: + - name: allow-http # Allow inbound HTTP GET requests to "/api/v2" from Pods with app=client label. + action: Allow # All other traffic from these Pods will be automatically dropped, and subsequent rules will not be considered. + from: + - podSelector: + matchLabels: + app: client + l7Protocols: + - http: + path: "/api/v2/*" + host: "foo.bar.com" + method: "GET" + - name: drop-other # Drop all other inbound traffic (i.e., from Pods without the app=client label or from external clients). + action: Drop +``` + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: allow-web-access-to-internal-domain +spec: + priority: 5 + tier: securityops + appliedTo: + - podSelector: + matchLabels: + egress-restriction: internal-domain-only + egress: + - name: allow-dns # Allow outbound DNS requests. + action: Allow + ports: + - protocol: TCP + port: 53 + - protocol: UDP + port: 53 + - name: allow-http-only # Allow outbound HTTP requests towards foo.bar.com. + action: Allow # As the rule's "to" and "ports" are empty, which means it selects traffic to any network + l7Protocols: # peer's any port using any transport protocol, all outbound HTTP requests towards other + - http: # domains and non-HTTP requests will be automatically dropped, and subsequent rules will + host: "*.bar.com" # not be considered. +``` + +Please refer to [Antrea Layer 7 NetworkPolicy](antrea-l7-network-policy.md) for extra information. + +#### ACNP for Kubernetes Node traffic + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: acnp-node-egress-traffic-drop +spec: + priority: 5 + tier: securityops + appliedTo: + - nodeSelector: + matchLabels: + kubernetes.io/os: linux + egress: + - action: Drop + to: + - ipBlock: + cidr: 192.168.1.0/24 + ports: + - protocol: TCP + port: 80 + name: dropHTTPTrafficToCIDR +``` + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: acnp-node-ingress-traffic-drop +spec: + priority: 5 + tier: securityops + appliedTo: + - nodeSelector: + matchLabels: + kubernetes.io/os: linux + ingress: + - action: Drop + from: + - ipBlock: + cidr: 192.168.1.0/24 + ports: + - protocol: TCP + port: 22 + name: dropSSHTrafficFromCIDR +``` + +Please refer to [Antrea Node NetworkPolicy](antrea-node-network-policy.md) for more information. + +#### ACNP with log settings + +```yaml +apiVersion: crd.antrea.io/v1beta1 +kind: ClusterNetworkPolicy +metadata: + name: acnp-with-log-setting +spec: + priority: 5 + tier: securityops + appliedTo: + - podSelector: + matchLabels: + role: db + - namespaceSelector: + matchLabels: + env: prod + ingress: + - action: Allow + from: + - podSelector: + matchLabels: + role: frontend + namespaceSelector: + matchLabels: + role: db + name: AllowFromFrontend + enableLogging: true + logLabel: "frontend-allowed" +``` + +**spec**: The ClusterNetworkPolicy `spec` has all the information needed to +define a cluster-wide security policy. + +**appliedTo**: The `appliedTo` field at the policy level specifies the +grouping criteria of Pods to which the policy applies to. Pods can be +selected cluster-wide using `podSelector`. If set with a `namespaceSelector`, +all Pods from Namespaces selected by the namespaceSelector will be selected. +Specific Pods from specific Namespaces can be selected by providing both a +`podSelector` and a `namespaceSelector` in the same `appliedTo` entry. +The `appliedTo` field can also reference a ClusterGroup resource by setting +the ClusterGroup's name in `group` field in place of the stand-alone selectors. +The `appliedTo` field can also reference a Service by setting the Service's name +and Namespace in `service` field in place of the stand-alone selectors. Only a +NodePort Service can be referred by this field. More details can be found in the +[ApplyToNodePortService](#apply-to-nodeport-service) section. +IPBlock cannot be set in the `appliedTo` field. +An IPBlock ClusterGroup referenced in an `appliedTo` field will be ignored, +and the policy will have no effect. +This `appliedTo` field must not be set, if `appliedTo` per +rule is used. + +In the [first example](#acnp-with-stand-alone-selectors), the policy applies to Pods, which either match the labels +"role=db" in all the Namespaces, or are from Namespaces which match the +labels "env=prod". +The [second example](#acnp-with-clustergroup-reference) policy applies to all network endpoints selected by the +"test-cg-with-db-selector" ClusterGroup. +The [third example](#acnp-for-complete-pod-isolation-in-selected-namespaces) policy applies to all Pods in the +Namespaces that matches label "app=no-network-access-required". +`appliedTo' also supports ServiceAccount based selection. This allows users using ServiceAccount to select Pods. +More details can be found in the [ServiceAccountSelector](#serviceaccount-based-selection) section. + +**priority**: The `priority` field determines the relative priority of the +policy among all ClusterNetworkPolicies in the given cluster. This field is +mandatory. A lower priority value indicates higher precedence. Priority values +can range from 1.0 to 10000.0. +**Note**: Policies with the same priorities will be enforced +indeterministically. Users should therefore take care to use priorities to +ensure the behavior they expect. + +**tier**: The `tier` field associates an ACNP to an existing Tier. The `tier` +field can be set with the name of the Tier CRD to which this policy must be +associated with. If not set, the ACNP is associated with the lowest priority +default tier i.e. the "application" Tier. + +**action**: Each ingress or egress rule of a ClusterNetworkPolicy must have the +`action` field set. As of now, the available actions are ["Allow", "Drop", "Reject", "Pass"]. +When the rule action is "Allow" or "Drop", Antrea will allow or drop traffic which +matches both `from/to`, `ports` and `protocols` sections of that rule, given that traffic does not +match a higher precedence rule in the cluster (ACNP rules created in higher order +Tiers or policy instances in the same Tier with lower priority number). If a "Reject" +rule is matched, the client initiating the traffic will receive `ICMP host administratively +prohibited` code for ICMP, UDP and SCTP request, or an explicit reject response for +TCP request, instead of timeout. A "Pass" rule, on the other hand, skips this packet +for further ACNP rule evaluations (all ACNP rules that has lower priority than the +current "Pass" rule will be skipped, except for the Baseline Tier rules), and delegates +the decision to developer created namespaced NetworkPolicies. If no NetworkPolicy +matches this traffic, then the Baseline Tier rules will still be matched against. +Note that the "Pass" action does not make sense when configured in Baseline Tier +ACNP rules, and such configurations will be rejected by the admission controller. +Note: "Pass" and "Reject" actions are not supported for rules applied to multicast +traffic. + +**ingress**: Each ClusterNetworkPolicy may consist of zero or more ordered set of +ingress rules. Under `ports`, the optional field `endPort` can only be set when a +numerical `port` is set to represent a range of ports from `port` to `endPort` inclusive. +`protocols` defines additional protocols that are not supported by `ports`. +Currently only ICMP protocol and IGMP protocol are under `protocols`. For `ICMP` +protocol, `icmpType` and `icmpCode` could be used to specify the ICMP traffic that +this rule matches. And for `IGMP` protocol, `igmpType` and `groupAddress` can be +used to specify the IGMP traffic that this rule matches. Currently, only IGMP +query is supported in ingress rules. Other IGMP types and multicast data traffic +are not supported for ingress rules. Valid `igmpType` is: + +message type | value +-- | -- +Membership Query | 0x11 + +The group address in IGMP query packets can only be 224.0.0.1. As for Group-Specific +IGMP query, which encodes the target group in the IGMP message, it is not supported +yet because OVS can not recognize the address. Protocol `IGMP` can not be used with +`ICMP` or properties like `from`, `to`, `ports` and `toServices`. + +Also, each rule has an optional `name` field, which should be unique within +the policy describing the intention of this rule. If `name` is not provided for +a rule, it will be auto-generated by Antrea. The auto-generated name will be +of format `[ingress/egress]-[action]-[uid]`, e.g. ingress-allow-2f0ed6e, +where [uid] is the first 7 bits of hash value of the rule based on sha1 algorithm. +If a policy contains duplicate rules, or if a rule name is same as the auto-generated +name of some other rules in the same policy, it will cause a conflict, +and the policy will be rejected. +A ClusterGroup name can be set in the `group` field of an ingress `from` section in place +of stand-alone selectors to allow traffic from workloads/ipBlocks set in the ClusterGroup. + +The [first example](#acnp-with-stand-alone-selectors) policy contains a single rule, which allows matched traffic on a +single port, from one of two sources: the first specified by a `podSelector` +and the second specified by a combination of a `podSelector` and a +`namespaceSelector`. +The [second example](#acnp-with-clustergroup-reference) policy contains a single rule, which allows matched traffic on +multiple TCP ports (8000 through 9000 included, plus 6379) from all network endpoints +selected by the "test-cg-with-frontend-selector" ClusterGroup. +The [third example](#acnp-for-complete-pod-isolation-in-selected-namespaces) policy contains a single rule, +which drops all ingress traffic towards any Pod in Namespaces that have label `app` set to +`no-network-access-required`. Note that an empty `From` in the ingress rule means that +this rule matches all ingress sources. +Ingress `From` section also supports ServiceAccount based selection. This allows users to use ServiceAccount +to select Pods. More details can be found in the [ServiceAccountSelector](#serviceaccount-based-selection) section. +**Note**: The order in which the ingress rules are specified matters, i.e., rules will +be enforced in the order in which they are written. + +**egress**: Each ClusterNetworkPolicy may consist of zero or more ordered set +of egress rules. Each rule, depending on the `action` field of the rule, allows +or drops traffic which matches all `from`, `ports` sections. +Under `ports`, the optional field `endPort` can only be set when a numerical `port` +is set to represent a range of ports from `port` to `endPort` inclusive. +`protocols` defines additional protocols that are not supported by `ports`. Currently, only +ICMP protocol and IGMP protocol are under `protocols`. For `ICMP` protocol, `icmpType` +and `icmpCode` could be used to specify the ICMP traffic that this rule matches. And +for `IGMP` protocol, `igmpType` and `groupAddress` can be used to specify the IGMP +traffic that this rule matches. If `igmpType` is not set, all reports will be matched. +If `groupAddress` is empty, then all multicast group addresses will be matched here. +Only IGMP reports are supported in egress rules. Protocol `IGMP` can not be used with +`ICMP` or properties like `from`, `to`, `ports` and `toServices`. Valid `igmpType` are: + +message type | value +-- | -- +IGMPv1 Membership Report | 0x12 +IGMPv2 Membership Report | 0x16 +IGMPv3 Membership Report | 0x22 + +Also, each rule has an optional `name` field, which should be unique within +the policy describing the intention of this rule. If `name` is not provided for +a rule, it will be auto-generated by Antrea. The rule name auto-generation process +is the same as ingress rules. +A ClusterGroup name can be set in the `group` field of a egress `to` section in place +of stand-alone selectors to allow traffic to workloads/ipBlocks set in the ClusterGroup. +`toServices` field contains a list of combinations of Service Namespace and Service Name +to match traffic to this Service. + +More details can be found in the [toServices](#toservices-egress-rules) section. +The [first example](#acnp-with-stand-alone-selectors) policy contains a single rule, which drops matched traffic on a +single port, to the 10.0.10.0/24 subnet specified by the `ipBlock` field. +The [second example](#acnp-with-clustergroup-reference) policy contains a single rule, which drops matched traffic on +TCP port 5978 to all network endpoints selected by the "test-cg-with-ip-block" +ClusterGroup. +The [third example](#acnp-for-complete-pod-isolation-in-selected-namespaces) policy contains a single rule, +which drops all egress traffic initiated by any Pod in Namespaces that have `app` set to +`no-network-access-required`. +The [sixth example](#acnp-for-toservices-rule) policy contains a single rule, +which drops traffic from "role: client" labeled Pods from "env: prod" labeled Namespaces to Service svcNamespace/svcName +via ClusterIP. +Note that an empty `to` + an empty `toServices` in the egress rule means that +this rule matches all egress destinations. +Egress `To` section also supports FQDN based filtering. This can be applied to exact FQDNs or +wildcard expressions. More details can be found in the [FQDN](#fqdn-based-filtering) section. +Egress `To` section also supports ServiceAccount based selection. This allows users to use ServiceAccount +to select Pods. More details can be found in the [ServiceAccountSelector](#serviceaccount-based-selection) section. +**Note**: The order in which the egress rules are specified matters, i.e., rules will +be enforced in the order in which they are written. + +**enableLogging** and **logLabel**: Antrea-native policy ingress or egress rules +can be audited by setting its logging fields. When the `enableLogging` field is set +to `true`, the first packet of any connection that matches this rule will be +logged to a file (`/var/log/antrea/networkpolicy/np.log`) on the Node on which the +rule is enforced. The log files can then be used for further analysis. If `logLabel` +is provided, the label will be added in the log. For example, in the +[ACNP with log settings](#acnp-with-log-settings), traffic that hits the +"AllowFromFrontend" rule will be logged with log label "frontend-allowed". + +For drop and reject rules, deduplication is applied to reduce duplicated +log messages, and the duplication buffer length is set to 1 second. When a rule +does not have a name, an identifiable name will be generated for the rule and +added to the log. For rules in layer 7 NetworkPolicy, packets are logged with +action `Redirect` prior to analysis by the layer 7 engine, and the layer 7 engine +can log more information in its own logs. + +The rules are logged in the following format: + +```text +