diff --git a/.nojekyll b/.nojekyll new file mode 100644 index 000000000..e69de29bb diff --git a/0.26/.buildinfo b/0.26/.buildinfo new file mode 100644 index 000000000..e33ab616c --- /dev/null +++ b/0.26/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: 527d0961f8224af12e66dd0bbe3563fe +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/0.26/DEVEL.html b/0.26/DEVEL.html new file mode 100644 index 000000000..806d0ae62 --- /dev/null +++ b/0.26/DEVEL.html @@ -0,0 +1,456 @@ + + + + + + + Instructions for Device Plugin Development and Maintenance — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+
    +
    • +
  • Instructions for Device Plugin Development and Maintenance
    • +
  • + View page source +
    • +
+
+
+
+
+ +
+

Instructions for Device Plugin Development and Maintenance

+

Table of Contents

+ +
+

Day-to-day Development How to’s

+
+

Get the Source Code

+

With git installed on the system, just clone the repository:

+
$ export INTEL_DEVICE_PLUGINS_SRC=/path/to/intel-device-plugins-for-kubernetes
+$ git clone https://github.com/intel/intel-device-plugins-for-kubernetes ${INTEL_DEVICE_PLUGINS_SRC}
+
+
+
+
+

Build and Run Plugin Binaries

+

With go development environment installed on the system, build the plugin:

+
$ cd ${INTEL_DEVICE_PLUGINS_SRC}
+$ make <plugin-build-target>
+
+
+

Note: All the available plugin build targets is roughly the output of ls ${INTEL_DEVICE_PLUGINS_SRC}/cmd.

+

To test the plugin binary on the development system, run as administrator:

+
$ sudo -E ${INTEL_DEVICE_PLUGINS_SRC}/cmd/<plugin-build-target>/<plugin-build-target>
+
+
+
+
+

Build Container Images

+

The dockerfiles are generated on the fly from .in suffixed files and .docker include-snippets which are stitched together with +cpp preprocessor. You need to install cpp for that, e.g. in ubuntu it is found from build-essential (sudo apt install build-essential). +Don’t edit the generated dockerfiles. Edit the inputs.

+

The simplest way to build all the docker images, is:

+
$ make images
+
+
+

But it is very slow. You can drastically speed it up by first running once:

+
$ make vendor
+
+
+

Which brings the libraries into the builder container without downloading them again and again for each plugin.

+

But it is still slow. You can further speed it up by first running once:

+
$ make licenses
+
+
+

Which pre-creates the go-licenses for all plugins, instead of re-creating them for each built plugin, every time.

+

But it is still rather slow to build all the images, and unnecessary, if you iterate on just one. Instead, build just the one you are iterating on, example:

+
$ make <image-build-target>
+
+
+

Note: All the available image build targets is roughly the output of ls ${INTEL_DEVICE_PLUGINS_SRC}/build/docker/*.Dockerfile.

+

If you iterate on only one plugin and if you know what its target cmd is (see folder cmd/), you can opt to pre-create just its licenses, example:

+
$ make licenses/<plugin-build-target>
+
+
+

The container image target names in the Makefile are derived from the .Dockerfile.in suffixed filenames under folder build/docker/templates/.

+

Recap:

+
$ make vendor
+$ make licenses (or just make licenses/<plugin-build-target>)
+$ make <image-build-target>
+
+
+

Repeat the last step only, unless you change library dependencies. If you pull in new sources, start again from make vendor.

+

Note: The image build tool can be changed from the default docker by setting the BUILDER argument +to the Makefile: make <image-build-target> BUILDER=<builder>. Supported values are docker, buildah, and podman.

+
+
+

Build Against a Newer Version of Kubernetes

+

First, you need to update module dependencies. The easiest way is to use +scripts/upgrade_k8s.sh copied from a k/k issue:

+

Just run it inside the repo’s root, e.g.

+
$ ${INTEL_DEVICE_PLUGINS_SRC}/scripts/upgrade_k8s.sh <k8s version>
+
+
+

Finally, run:

+
$ make generate
+$ make test
+
+
+

and fix all new compilation issues.

+
+
+

Work with Intel Device Plugins Operator Modifications

+

There are few useful steps when working with changes to Device Plugins CRDs and controllers:

+
    +

  1. Install controller-gen: GO111MODULE=on go get -u sigs.k8s.io/controller-tools/cmd/controller-gen@<release ver>, e.g, v0.4.1

    2. +

  2. Generate CRD and Webhook artifacts: make generate

    3. +

  3. Test local changes using envtest: make envtest

    4. +

  4. Build a custom operator image: make intel-deviceplugin-operator

    5. +

  5. (Un)deploy operator: kubectl [apply|delete] -k deployments/operator/default

    6. +
+
+
+

Publish a New Version of the Intel Device Plugins Operator to operatorhub.io

+

Check if the fields mentioned below in the base CSV manifest file have the correct values. If not, fix them manually (operator-sdk does not support updating these fields in any other way).

+
    +

  • spec.version

    • +

  • spec.replaces

    • +

  • metadata.annotations.containerImage

    • +

  • metadata.annotations.createdAT

    • +
+

Fork the Community Operators repo and clone it:

+
$ git clone https://github.com/<GitHub Username>/community-operators
+
+
+

Generate bundle and build bundle image:

+
$ make bundle TAG=0.X.Y CHANNELS=alpha DEFAULT_CHANNEL=alpha
+$ make bundle-build
+
+
+

Push the image to a registry:

+
    +

  • If pushing to the Docker hub, specify docker.io/ in front of the image name for running bundle.

    • +

  • If pushing to the local registry, put the option --use-http for running bundle.

    • +
+

Verify the operator deployment works OK via OLM in your development cluster:

+
$ operator-sdk olm install
+$ kubectl create namespace testoperator
+$ operator-sdk run bundle <Registry>:<Tag> -n testoperator
+# do verification checks
+...
+# do clean up
+$ operator-sdk cleanup intel-device-plugins-operator --namespace testoperator
+$ kubectl delete namespace testoperator
+$ operator-sdk olm uninstall
+
+
+

Commit files:

+
$ cd community-operators
+$ git add operators/intel-device-plugins-operator/0.X.Y
+$ git commit -am 'operators intel-device-plugins-operator (0.X.Y)' -s
+
+
+

Submit a PR to Community Operators repo.

+

Check operator page +https://operatorhub.io/operator/intel-device-plugins-operator +after PR is merged.

+
+
+

Run E2E Tests

+

Currently the E2E tests require having a Kubernetes cluster already configured +on the nodes with the hardware required by the device plugins. Also all the +container images with the executables under test must be available in the +cluster. If these two conditions are satisfied, run the tests with:

+
$ go test -v ./test/e2e/...
+
+
+

In case you want to run only certain tests, e.g., QAT ones, run:

+
$ go test -v ./test/e2e/... -args -ginkgo.focus "QAT"
+
+
+

If you need to specify paths to your custom kubeconfig containing +embedded authentication info then add the -kubeconfig argument:

+
$ go test -v ./test/e2e/... -args -kubeconfig /path/to/kubeconfig
+
+
+

The full list of available options can be obtained with:

+
$ go test ./test/e2e/... -args -help
+
+
+

It is also possible to run the tests which don’t depend on hardware +without a pre-configured Kubernetes cluster. Just make sure you have +Kind installed on your host and run:

+
$ make test-with-kind
+
+
+
+
+

Run Controller Tests with a Local Control Plane

+

The controller-runtime library provides a package for integration testing by +starting a local control plane. The package is called +envtest. The +operator uses this package for its integration testing.

+

For setting up the environment for testing, setup-envtest can be used:

+
$ go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
+$ setup-envtest use <K8S_VERSION>
+$ KUBEBUILDER_ASSETS=$(setup-envtest use -i -p path <K8S_VERSION>) make envtest
+
+
+
+
+
+

How to Develop Simple Device Plugins

+

To create a simple device plugin without the hassle of developing your own gRPC +server, you can use a package included in this repository called +github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin.

+

All you have to do is instantiate a deviceplugin.Manager and call +its Run() method:

+
func main() {
+    ...
+
+    manager := dpapi.NewManager(namespace, plugin)
+    manager.Run()
+}
+
+
+

The manager’s constructor accepts two parameters:

+
    +

  1. namespace which is a string like “color.example.com”. All your devices +will be exposed under this name space, e.g. “color.example.com/yellow”. +Please note that one device plugin can register many such “colors”. +The manager will instantiate multiple gRPC servers for every registered “color”.

    2. +

  2. plugin which is a reference to an object implementing one mandatory +interface deviceplugin.Scanner.

    3. +
+

deviceplugin.Scanner defines one method Scan() which is called only once +for every device plugin by deviceplugin.Manager in a goroutine and operates +in an infinite loop. A Scan() implementation scans the host for devices and +sends all found devices to a deviceplugin.Notifier instance. The +deviceplugin.Notifier is implemented and provided by the deviceplugin +package itself. The found devices are organized in an instance of +deviceplugin.DeviceTree object. The object is filled in with its +AddDevice() method:

+
func (dp *devicePlugin) Scan(notifier deviceplugin.Notifier) error {
+    for {
+        devTree := deviceplugin.NewDeviceTree()
+        ...
+        devTree.AddDevice("yellow", devID, deviceplugin.DeviceInfo{
+            State: health,
+            Nodes: []pluginapi.DeviceSpec{
+                {
+                    HostPath:      devPath,
+                    ContainerPath: devPath,
+                    Permissions:   "rw",
+                },
+            },
+        })
+        ...
+        notifier.Notify(devTree)
+    }
+}
+
+
+

Optionally, your device plugin may also implement the +deviceplugin.PostAllocator interface. If implemented, its method +PostAllocate() modifies pluginapi.AllocateResponse responses just +before they are sent to kubelet. To see an example, refer to the FPGA +plugin which implements this interface to annotate its responses.

+

In case you want to implement the whole allocation functionality in your +device plugin, you can implement the optional deviceplugin.Allocator +interface. In this case PostAllocate() is not called. But if you decide in your +implementation of deviceplugin.Allocator that you need to resort to the default +implementation of the allocation functionality then return an error of the type +deviceplugin.UseDefaultMethodError.

+
+

Logging

+

The framework uses klog as its logging +framework. It is encouraged for plugins to also use klog to maintain uniformity +in the logs and command line options.

+

The framework initialises klog, so further calls to klog.InitFlags() by +plugins should not be necessary. This does add a number of log configuration +options to your plugin, which can be viewed with the -h command line option of your +plugin.

+

The framework tries to adhere to the Kubernetes +Logging Conventions. +The advise is to use the V() levels for Info() calls, as calling Info() +with no set level will make configuration and filtering of logging via the command +line more difficult.

+

The default is to not log Info() calls. This can be changed using the plugin command +line -v parameter. The additional annotations prepended to log lines by ‘klog’ can be disabled +with the -skip_headers option.

+
+
+

Error Conventions

+

The framework has a convention for producing and logging errors. Ideally plugins will also adhere +to the convention.

+

Errors generated within the framework and plugins are instantiated with the New() and +Errorf() functions of the errors package:

+
    return errors.New("error message")
+
+
+

Errors generated from outside the plugins and framework are augmented with their stack dump with code such as

+
    return errors.WithStack(err)
+
+
+

or

+
    return errors.Wrap(err, "some additional error message")
+
+
+

These errors are then logged using a default struct value format like:

+
    klog.Errorf("Example of an internal error death: %+v", err)
+
+
+

at the line where it’s certain that the error cannot be passed out farther nor handled gracefully. +Otherwise, they can be logged as simple values:

+
    klog.Warningf("Example of a warning due to an external error: %v", err)
+
+
+
+
+
+

Checklist for New Device Plugins

+

For new device plugins contributed to this repository, below is a +checklist to get the plugin on par feature and quality wise with +others:

+
    +

  1. Plugin binary available in cmd/, its corresponding Dockerfile in build/docker/ and deployment Kustomization/YAMLs in deployments/.

    2. +

  2. Plugin binary Go unit tests implemented and passing with >80% coverage: make test WHAT=./cmd/<plugin>.

    3. +

  3. Plugin binary linter checks passing: make lint.

    4. +

  4. Plugin e2e tests implemented in test/e2e/ and passing: go test -v ./test/e2e/... -args -ginkgo.focus "<plugin>".

    5. +

  5. Plugin CRD API added to pkg/apis/deviceplugin/v1 and CRDs generated: make generate.

    6. +

  6. Plugin CRD validation tests implemented in test/envtest/ and passing: make envtest.

    7. +

  7. Plugin CRD controller implemented in pkg/controllers/ and added to the manager in cmd/operator/main.go.

    8. +

  8. Plugin documentation written cmd/<plugin>/README.md and optionally end to end demos created in demo.

    9. +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/README.html b/0.26/README.html new file mode 100644 index 000000000..ccbbb673c --- /dev/null +++ b/0.26/README.html @@ -0,0 +1,519 @@ + + + + + + + Overview — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Overview

+

Build Status +Go Report Card +GoDoc

+

This repository contains a framework for developing plugins for the Kubernetes +device plugins framework, +along with a number of device plugin implementations utilizing that framework.

+

The v0.26 release +is the latest feature release with its documentation available here.

+

Table of Contents

+ +
+

Prerequisites

+

Prerequisites for building and running these device plugins include:

+ +
+
+

Plugins

+

The below sections detail existing plugins developed using the framework.

+
+

GPU Device Plugin

+

The GPU device plugin provides access to +discrete and integrated Intel GPU device files.

+

The demo subdirectory contains both a GPU plugin demo video +and an OpenCL sample deployment (intelgpu-job.yaml).

+
+
+

FPGA Device Plugin

+

The FPGA device plugin supports FPGA passthrough for +the following hardware:

+
    +

  • Intel® Arria® 10 devices

    • +

  • Intel® Stratix® 10 devices

    • +
+

The FPGA plugin comes as three parts.

+ +

Refer to each individual sub-components documentation for more details. +Brief overviews of the sub-components are below.

+

The demo subdirectory contains a +video showing deployment +and use of the FPGA plugin. Sources relating to the demo can be found in the +opae-nlb-demo subdirectory.

+
+

Device Plugin

+

The FPGA device plugin is responsible for +discovering and reporting FPGA devices to kubelet.

+
+
+

Admission Controller

+

The FPGA admission controller webhook +is responsible for performing mapping from user-friendly function IDs to the +Interface ID and Bitstream ID that are required for FPGA programming. It also +implements access control by namespacing FPGA configuration information.

+
+
+

CRI-O Prestart Hook

+

The FPGA prestart CRI-O hook performs discovery +of the requested FPGA function bitstream and programs FPGA devices based on the +environment variables in the workload description.

+
+
+
+

QAT Device Plugin

+

The QAT plugin supports device plugin for Intel QAT adapters, and includes +code showing deployment via DPDK.

+

The demo subdirectory includes details of both a +QAT DPDK demo +and a QAT OpenSSL demo. +Source for the OpenSSL demo can be found in the relevant subdirectory.

+

Details for integrating the QAT device plugin into Kata Containers +can be found in the +Kata Containers documentation repository.

+
+
+

VPU Device Plugin

+

The VPU device plugin supports Intel VCAC-A card +(https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/media-analytics-vcac-a-accelerator-card-by-celestica-datasheet.pdf) +the card has:

+
    +

  • 1 Intel Core i3-7100U processor

    • +

  • 12 MyriadX VPUs

    • +

  • 8GB DDR4 memory

    • +
+

The demo subdirectory includes details of a OpenVINO deployment and use of the +VPU plugin. Sources can be found in openvino-demo.

+
+
+

SGX Device Plugin

+

The SGX device plugin allows workloads to use +Intel® Software Guard Extensions (Intel® SGX) on +platforms with SGX Flexible Launch Control enabled, e.g.,:

+
    +

  • 3rd Generation Intel® Xeon® Scalable processor family, code-named “Ice Lake”

    • +

  • Intel® Xeon® E3 processor

    • +

  • Intel® NUC Kit NUC7CJYH

    • +
+

The Intel SGX plugin comes in three parts.

+ +

The demo subdirectory contains a video showing the deployment +and use of the Intel SGX device plugin. Sources relating to the demo can be found in the +sgx-sdk-demo and sgx-aesmd-demo subdirectories.

+

Brief overviews of the Intel SGX sub-components are given below.

+

+
+

device plugin

+

The SGX device plugin is responsible for discovering +and reporting Intel SGX device nodes to kubelet.

+

Containers requesting Intel SGX resources in the cluster should not use the +device plugins resources directly.

+
+
+

Intel SGX Admission Webhook

+

The Intel SGX admission webhook is responsible for performing Pod mutations based on +the sgx.intel.com/quote-provider pod annotation set by the user. The purpose +of the webhook is to hide the details of setting the necessary device resources +and volume mounts for using Intel SGX remote attestation in the cluster. Furthermore, +the Intel SGX admission webhook is responsible for writing a pod/sandbox +sgx.intel.com/epc annotation that is used by Kata Containers to dynamically +adjust its virtualized Intel SGX encrypted page cache (EPC) bank(s) size.

+

The Intel SGX admission webhook is available as part of +Intel Device Plugin Operator or +as a standalone SGX Admission webhook image.

+
+
+

Intel SGX EPC memory registration

+

The Intel SGX EPC memory available on each node is registered as a Kubernetes extended resource using +node-feature-discovery (NFD). A custom NFD source hook is installed as part of +SGX device plugin +operator deployment and NFD is configured to register the Intel SGX EPC memory +extended resource reported by the hook.

+

Containers requesting Intel SGX EPC resources in the cluster use +sgx.intel.com/epc resource which is of +type memory.

+
+
+
+

DSA Device Plugin

+

The DSA device plugin supports acceleration using +the Intel Data Streaming accelerator(DSA).

+
+
+

DLB Device Plugin

+

The DLB device plugin supports Intel Dynamic Load +Balancer accelerator(DLB).

+
+
+

IAA Device Plugin

+

The IAA device plugin supports acceleration using +the Intel Analytics accelerator(IAA).

+
+
+
+

Device Plugins Operator

+

To simplify the deployment of the device plugins, a unified device plugins +operator is implemented.

+

Currently the operator has support for the DSA, DLB, FPGA, GPU, IAA, QAT, and +Intel SGX device plugins. Each device plugin has its own custom resource +definition (CRD) and the corresponding controller that watches CRUD operations +to those custom resources.

+

The Device plugins operator README gives the installation and usage details for the community operator available on operatorhub.io.

+

The Device plugins Operator for OCP gives the installation and usage details for the operator available on Red Hat OpenShift Container Platform.

+
+ +
+

Demos

+

The demo subdirectory contains a number of demonstrations for +a variety of the available plugins.

+
+
+

Workload Authors

+

For workloads to get accesss to devices managed by the plugins, the +Pod spec must specify the hardware resources needed:

+
spec:
+  containers:
+    - name: demo-container
+      image: <registry>/<image>:<version>
+      resources:
+        limits:
+          <device namespace>/<resource>: X
+
+
+

The summary of resources available via plugins in this repository is given in the list below.

+

Device Namespace : Registered Resource(s)

+ +
+
+

Developers

+

For information on how to develop a new plugin using the framework or work on development task in +this repository, see the Developers Guide.

+
+
+

Supported Kubernetes Versions

+

Releases are made under the github releases area. Supported releases and +matching Kubernetes versions are listed below:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
BranchKubernetes branch/versionStatus
release-0.26Kubernetes 1.26 branch v1.26.xsupported
release-0.25Kubernetes 1.25 branch v1.25.xsupported
release-0.24Kubernetes 1.24 branch v1.24.xsupported
release-0.23Kubernetes 1.23 branch v1.23.xunsupported
release-0.22Kubernetes 1.22 branch v1.22.xunsupported
release-0.21Kubernetes 1.21 branch v1.21.xunsupported
release-0.20Kubernetes 1.20 branch v1.20.xunsupported
release-0.19Kubernetes 1.19 branch v1.19.xunsupported
release-0.18Kubernetes 1.18 branch v1.18.xunsupported
release-0.17Kubernetes 1.17 branch v1.17.xunsupported
release-0.15Kubernetes 1.15 branch v1.15.xunsupported
release-0.11Kubernetes 1.11 branch v1.11.xunsupported
+
+

Pre-built plugin images

+

Pre-built images of the plugins are available on the Docker hub. These images +are automatically built and uploaded to the hub from the latest main branch of +this repository.

+

Release tagged images of the components are also available on the Docker hub, +tagged with their release version numbers in the format x.y.z, corresponding to +the branches and releases in this repository.

+

Note: the default deployment files and operators are configured with +imagePullPolicy +IfNotPresent and can be changed with scripts/set-image-pull-policy.sh.

+
+
+

License

+

All of the source code required to build intel-device-plugins-for-kubernetes +is available under Open Source licenses. The source code files identify external Go +modules used. Binaries are distributed as container images on +DockerHub*. Those images contain license texts and source code under /licenses.

+
+

Helm Charts

+

Device Plugins Helm Charts are located in Intel Helm Charts repository Intel Helm Charts. This is another way of distributing Kubernetes resources of the device plugins framework.

+

To add repo:

+
helm repo add intel https://intel.github.io/helm-charts
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/SECURITY.html b/0.26/SECURITY.html new file mode 100644 index 000000000..eaf00b536 --- /dev/null +++ b/0.26/SECURITY.html @@ -0,0 +1,118 @@ + + + + + + + <no title> — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +

Reporting a Potential Security Vulnerability: If you have discovered +potential security vulnerability in this project, please send an e-mail to +secure@intel.com. Encrypt sensitive information using our +PGP public key.

+

Please provide as much information as possible, including:

+
    +

  • The projects and versions affected

    • +

  • Detailed description of the vulnerability

    • +

  • Information on known exploits

    • +
+

A member of the Intel Product Security Team will review your e-mail and +contact you to collaborate on resolving the issue. For more information on +how Intel works to resolve security issues, see Vulnerability Handling Guidelines.

+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/_images/FPGA-af.png b/0.26/_images/FPGA-af.png new file mode 100644 index 000000000..64934fdfa Binary files /dev/null and b/0.26/_images/FPGA-af.png differ diff --git a/0.26/_images/FPGA-region.png b/0.26/_images/FPGA-region.png new file mode 100644 index 000000000..b57d9a1ed Binary files /dev/null and b/0.26/_images/FPGA-region.png differ diff --git a/0.26/_images/SGX-BIOS.PNG b/0.26/_images/SGX-BIOS.PNG new file mode 100644 index 000000000..f8510ecec Binary files /dev/null and b/0.26/_images/SGX-BIOS.PNG differ diff --git a/0.26/_images/verify-operator.PNG b/0.26/_images/verify-operator.PNG new file mode 100644 index 000000000..55adad1b5 Binary files /dev/null and b/0.26/_images/verify-operator.PNG differ diff --git a/0.26/_sources/DEVEL.md.txt b/0.26/_sources/DEVEL.md.txt new file mode 100644 index 000000000..900dc6337 --- /dev/null +++ b/0.26/_sources/DEVEL.md.txt @@ -0,0 +1,369 @@ +# Instructions for Device Plugin Development and Maintenance + +Table of Contents + +* [Day-to-day Development How to's](#day-to-day-development) + * [Get the Source Code](#get-the-source-code) + * [Build and Run Plugin Binaries](#build-and-run-plugin-binaries) + * [Build Container Images](#build-container-images) + * [Build Against a Newer Version of Kubernetes](#build-against-a-newer-version-of-kubernetes) + * [Work with Intel Device Plugins Operator Modifications](#work-with-intel-device-plugins-operator-modifications) + * [Publish a New Version of the Intel Device Plugins Operator to operatorhub.io](#publish-a-new-version-of-the-intel-device-plugins-operator-to-operatorhubio) + * [Run E2E Tests](#run-e2e-tests) + * [Run Controller Tests with a Local Control Plane](#run-controller-tests-with-a-local-control-plane) +* [How to Develop Simple Device Plugins](#how-to-develop-simple-device-plugins) + * [Logging](#logging) + * [Error Conventions](#error-conventions) +* [Checklist for New Device Plugins](#checklist-for-new-device-plugins) + +## Day-to-day Development How to's +### Get the Source Code + +With `git` installed on the system, just clone the repository: + +```bash +$ export INTEL_DEVICE_PLUGINS_SRC=/path/to/intel-device-plugins-for-kubernetes +$ git clone https://github.com/intel/intel-device-plugins-for-kubernetes ${INTEL_DEVICE_PLUGINS_SRC} +``` + +### Build and Run Plugin Binaries + +With `go` development environment installed on the system, build the plugin: + +```bash +$ cd ${INTEL_DEVICE_PLUGINS_SRC} +$ make +``` + +**Note:** All the available plugin build targets is roughly the output of `ls ${INTEL_DEVICE_PLUGINS_SRC}/cmd`. + +To test the plugin binary on the development system, run as administrator: + +```bash +$ sudo -E ${INTEL_DEVICE_PLUGINS_SRC}/cmd// +``` + +### Build Container Images + +The dockerfiles are generated on the fly from `.in` suffixed files and `.docker` include-snippets which are stitched together with +cpp preprocessor. You need to install cpp for that, e.g. in ubuntu it is found from build-essential (sudo apt install build-essential). +Don't edit the generated dockerfiles. Edit the inputs. + +The simplest way to build all the docker images, is: +``` +$ make images +``` + +But it is very slow. You can drastically speed it up by first running once: +``` +$ make vendor +``` + +Which brings the libraries into the builder container without downloading them again and again for each plugin. + +But it is still slow. You can further speed it up by first running once: +``` +$ make licenses +``` + +Which pre-creates the go-licenses for all plugins, instead of re-creating them for each built plugin, every time. + +But it is still rather slow to build all the images, and unnecessary, if you iterate on just one. Instead, build just the one you are iterating on, example: + +``` +$ make +``` + +**Note:** All the available image build targets is roughly the output of `ls ${INTEL_DEVICE_PLUGINS_SRC}/build/docker/*.Dockerfile`. + +If you iterate on only one plugin and if you know what its target cmd is (see folder `cmd/`), you can opt to pre-create just its licenses, example: +``` +$ make licenses/ +``` + +The container image target names in the Makefile are derived from the `.Dockerfile.in` suffixed filenames under folder `build/docker/templates/`. + +Recap: +``` +$ make vendor +$ make licenses (or just make licenses/) +$ make +``` + +Repeat the last step only, unless you change library dependencies. If you pull in new sources, start again from `make vendor`. + +**Note:** The image build tool can be changed from the default `docker` by setting the `BUILDER` argument +to the [`Makefile`](Makefile): `make BUILDER=`. Supported values are `docker`, `buildah`, and `podman`. + +### Build Against a Newer Version of Kubernetes + +First, you need to update module dependencies. The easiest way is to use +`scripts/upgrade_k8s.sh` copied [from a k/k issue](https://github.com/kubernetes/kubernetes/issues/79384#issuecomment-521493597): + +Just run it inside the repo's root, e.g. + +``` +$ ${INTEL_DEVICE_PLUGINS_SRC}/scripts/upgrade_k8s.sh +``` +Finally, run: + +``` +$ make generate +$ make test +``` + +and fix all new compilation issues. + +### Work with Intel Device Plugins Operator Modifications + +There are few useful steps when working with changes to Device Plugins CRDs and controllers: + +1. Install controller-gen: `GO111MODULE=on go get -u sigs.k8s.io/controller-tools/cmd/controller-gen@, e.g, v0.4.1` +2. Generate CRD and Webhook artifacts: `make generate` +3. Test local changes using [envtest](https://book.kubebuilder.io/reference/envtest.html): `make envtest` +4. Build a custom operator image: `make intel-deviceplugin-operator` +5. (Un)deploy operator: `kubectl [apply|delete] -k deployments/operator/default` + +### Publish a New Version of the Intel Device Plugins Operator to operatorhub.io + +Check if the fields mentioned below in the [base CSV manifest file](deployments/operator/manifests/bases/intel-device-plugins-operator.clusterserviceversion.yaml) have the correct values. If not, fix them manually (operator-sdk does not support updating these fields in any other way). +- spec.version +- spec.replaces +- metadata.annotations.containerImage +- metadata.annotations.createdAT + +Fork the [Community Operators](https://github.com/k8s-operatorhub/community-operators) repo and clone it: +``` +$ git clone https://github.com//community-operators +``` + +Generate bundle and build bundle image: +``` +$ make bundle TAG=0.X.Y CHANNELS=alpha DEFAULT_CHANNEL=alpha +$ make bundle-build +``` + +Push the image to a registry: +- If pushing to the Docker hub, specify `docker.io/` in front of the image name for running bundle. +- If pushing to the local registry, put the option `--use-http` for running bundle. + +Verify the operator deployment works OK via OLM in your development cluster: +``` +$ operator-sdk olm install +$ kubectl create namespace testoperator +$ operator-sdk run bundle : -n testoperator +# do verification checks +... +# do clean up +$ operator-sdk cleanup intel-device-plugins-operator --namespace testoperator +$ kubectl delete namespace testoperator +$ operator-sdk olm uninstall +``` + +Commit files: +``` +$ cd community-operators +$ git add operators/intel-device-plugins-operator/0.X.Y +$ git commit -am 'operators intel-device-plugins-operator (0.X.Y)' -s +``` + +Submit a PR to [Community Operators](https://github.com/k8s-operatorhub/community-operators) repo. + +Check operator page +https://operatorhub.io/operator/intel-device-plugins-operator +after PR is merged. + +### Run E2E Tests + +Currently the E2E tests require having a Kubernetes cluster already configured +on the nodes with the hardware required by the device plugins. Also all the +container images with the executables under test must be available in the +cluster. If these two conditions are satisfied, run the tests with: + +```bash +$ go test -v ./test/e2e/... +``` + +In case you want to run only certain tests, e.g., QAT ones, run: + +```bash +$ go test -v ./test/e2e/... -args -ginkgo.focus "QAT" +``` + +If you need to specify paths to your custom `kubeconfig` containing +embedded authentication info then add the `-kubeconfig` argument: + +```bash +$ go test -v ./test/e2e/... -args -kubeconfig /path/to/kubeconfig +``` + +The full list of available options can be obtained with: + +```bash +$ go test ./test/e2e/... -args -help +``` + +It is also possible to run the tests which don't depend on hardware +without a pre-configured Kubernetes cluster. Just make sure you have +[Kind](https://kind.sigs.k8s.io/) installed on your host and run: + +``` +$ make test-with-kind +``` + +### Run Controller Tests with a Local Control Plane + +The controller-runtime library provides a package for integration testing by +starting a local control plane. The package is called +[envtest](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest). The +operator uses this package for its integration testing. + +For setting up the environment for testing, `setup-envtest` can be used: + +```bash +$ go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest +$ setup-envtest use +$ KUBEBUILDER_ASSETS=$(setup-envtest use -i -p path ) make envtest +``` +## How to Develop Simple Device Plugins + +To create a simple device plugin without the hassle of developing your own gRPC +server, you can use a package included in this repository called +`github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin`. + +All you have to do is instantiate a `deviceplugin.Manager` and call +its `Run()` method: + +```go +func main() { + ... + + manager := dpapi.NewManager(namespace, plugin) + manager.Run() +} +``` + +The manager's constructor accepts two parameters: + +1. `namespace` which is a string like "color.example.com". All your devices + will be exposed under this name space, e.g. "color.example.com/yellow". + Please note that one device plugin can register many such "colors". + The manager will instantiate multiple gRPC servers for every registered "color". +2. `plugin` which is a reference to an object implementing one mandatory + interface `deviceplugin.Scanner`. + +`deviceplugin.Scanner` defines one method `Scan()` which is called only once +for every device plugin by `deviceplugin.Manager` in a goroutine and operates +in an infinite loop. A `Scan()` implementation scans the host for devices and +sends all found devices to a `deviceplugin.Notifier` instance. The +`deviceplugin.Notifier` is implemented and provided by the `deviceplugin` +package itself. The found devices are organized in an instance of +`deviceplugin.DeviceTree` object. The object is filled in with its +`AddDevice()` method: + +```go +func (dp *devicePlugin) Scan(notifier deviceplugin.Notifier) error { + for { + devTree := deviceplugin.NewDeviceTree() + ... + devTree.AddDevice("yellow", devID, deviceplugin.DeviceInfo{ + State: health, + Nodes: []pluginapi.DeviceSpec{ + { + HostPath: devPath, + ContainerPath: devPath, + Permissions: "rw", + }, + }, + }) + ... + notifier.Notify(devTree) + } +} +``` + +Optionally, your device plugin may also implement the +`deviceplugin.PostAllocator` interface. If implemented, its method +`PostAllocate()` modifies `pluginapi.AllocateResponse` responses just +before they are sent to `kubelet`. To see an example, refer to the FPGA +plugin which implements this interface to annotate its responses. + +In case you want to implement the whole allocation functionality in your +device plugin, you can implement the optional `deviceplugin.Allocator` +interface. In this case `PostAllocate()` is not called. But if you decide in your +implementation of `deviceplugin.Allocator` that you need to resort to the default +implementation of the allocation functionality then return an error of the type +`deviceplugin.UseDefaultMethodError`. + +### Logging + +The framework uses [`klog`](https://github.com/kubernetes/klog) as its logging +framework. It is encouraged for plugins to also use `klog` to maintain uniformity +in the logs and command line options. + +The framework initialises `klog`, so further calls to `klog.InitFlags()` by +plugins should not be necessary. This does add a number of log configuration +options to your plugin, which can be viewed with the `-h` command line option of your +plugin. + +The framework tries to adhere to the Kubernetes +[Logging Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md). +The advise is to use the `V()` levels for `Info()` calls, as calling `Info()` +with no set level will make configuration and filtering of logging via the command +line more difficult. + +The default is to not log `Info()` calls. This can be changed using the plugin command +line `-v` parameter. The additional annotations prepended to log lines by 'klog' can be disabled +with the `-skip_headers` option. + +### Error Conventions + +The framework has a convention for producing and logging errors. Ideally plugins will also adhere +to the convention. + +Errors generated within the framework and plugins are instantiated with the `New()` and +`Errorf()` functions of the [errors package](https://golang.org/pkg/errors/): + +```golang + return errors.New("error message") +``` + +Errors generated from outside the plugins and framework are augmented with their stack dump with code such as + +```golang + return errors.WithStack(err) +``` + +or + +```golang + return errors.Wrap(err, "some additional error message") +``` + +These errors are then logged using a default struct value format like: + +```golang + klog.Errorf("Example of an internal error death: %+v", err) +``` + +at the line where it's certain that the error cannot be passed out farther nor handled gracefully. +Otherwise, they can be logged as simple values: + +```golang + klog.Warningf("Example of a warning due to an external error: %v", err) +``` + +## Checklist for New Device Plugins + +For new device plugins contributed to this repository, below is a +checklist to get the plugin on par feature and quality wise with +others: + +1. Plugin binary available in [`cmd/`](cmd), its corresponding Dockerfile in [`build/docker/`](build/docker) and deployment Kustomization/YAMLs in [`deployments/`](deployments). +2. Plugin binary Go unit tests implemented and passing with >80% coverage: `make test WHAT=./cmd/`. +3. Plugin binary linter checks passing: `make lint`. +4. Plugin e2e tests implemented in [`test/e2e/`](test/e2e) and passing: `go test -v ./test/e2e/... -args -ginkgo.focus ""`. +5. Plugin CRD API added to [`pkg/apis/deviceplugin/v1`](pkg/apis/deviceplugin/v1) and CRDs generated: `make generate`. +6. Plugin CRD validation tests implemented in [`test/envtest/`](test/envtest) and passing: `make envtest`. +7. Plugin CRD controller implemented in [`pkg/controllers/`](pkg/controllers) and added to the manager in `cmd/operator/main.go`. +8. Plugin documentation written `cmd//README.md` and optionally end to end demos created in [`demo`](demo). diff --git a/0.26/_sources/README.md.txt b/0.26/_sources/README.md.txt new file mode 100644 index 000000000..7040c9320 --- /dev/null +++ b/0.26/_sources/README.md.txt @@ -0,0 +1,309 @@ +# Overview +[![Build Status](https://github.com/intel/intel-device-plugins-for-kubernetes/workflows/CI/badge.svg?branch=main)](https://github.com/intel/intel-device-plugins-for-kubernetes/actions?query=workflow%3ACI) +[![Go Report Card](https://goreportcard.com/badge/github.com/intel/intel-device-plugins-for-kubernetes)](https://goreportcard.com/report/github.com/intel/intel-device-plugins-for-kubernetes) +[![GoDoc](https://godoc.org/github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin?status.svg)](https://godoc.org/github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin) + +This repository contains a framework for developing plugins for the Kubernetes +[device plugins framework](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/), +along with a number of device plugin implementations utilizing that framework. + +The [v0.26 release](https://github.com/intel/intel-device-plugins-for-kubernetes/releases/latest) +is the latest feature release with its documentation available [here](https://intel.github.io/intel-device-plugins-for-kubernetes/0.26/). + +Table of Contents + +* [Prerequisites](#prerequisites) +* [Plugins](#plugins) + * [GPU device plugin](#gpu-device-plugin) + * [FPGA device plugin](#fpga-device-plugin) + * [QAT device plugin](#qat-device-plugin) + * [VPU device plugin](#vpu-device-plugin) + * [SGX device plugin](#sgx-device-plugin) + * [DSA device plugin](#dsa-device-plugin) + * [DLB device plugin](#dlb-device-plugin) + * [IAA device plugin](#iaa-device-plugin) +* [Device Plugins Operator](#device-plugins-operator) +* [XeLink XPU-Manager sidecar](#xelink-xpu-manager-sidecar) +* [Demos](#demos) +* [Workload Authors](#workload-authors) +* [Developers](#developers) +* [Supported Kubernetes versions](#supported-kubernetes-versions) +* [Pre-built plugin images](#pre-built-plugin-images) +* [License](#license) +* [Helm charts](#helm-charts) + +## Prerequisites + +Prerequisites for building and running these device plugins include: + +- Appropriate hardware and drivers +- A fully configured [Kubernetes cluster] +- A working [Go environment], of at least version v1.16. + +## Plugins + +The below sections detail existing plugins developed using the framework. + +### GPU Device Plugin + +The [GPU device plugin](cmd/gpu_plugin/README.md) provides access to +discrete and integrated Intel GPU device files. + +The demo subdirectory contains both a [GPU plugin demo video](demo/readme.md#intel-gpu-device-plugin-demo-video) +and an OpenCL sample deployment (`intelgpu-job.yaml`). + +### FPGA Device Plugin + +The [FPGA device plugin](cmd/fpga_plugin/README.md) supports FPGA passthrough for +the following hardware: + +- Intel® Arria® 10 devices +- Intel® Stratix® 10 devices + +The FPGA plugin comes as three parts. + +- the [device plugin](#device-plugin) +- the [admission controller](#admission-controller) +- the [CRIO-O prestart hook](#cri-o-prestart-hook) + +Refer to each individual sub-components documentation for more details. +Brief overviews of the sub-components are below. + +The demo subdirectory contains a +[video](demo/readme.md#intel-fpga-device-plugin-demo-video) showing deployment +and use of the FPGA plugin. Sources relating to the demo can be found in the +[opae-nlb-demo](demo/opae-nlb-demo) subdirectory. + +#### Device Plugin + +The [FPGA device plugin](cmd/fpga_plugin/README.md) is responsible for +discovering and reporting FPGA devices to `kubelet`. + +#### Admission Controller + +The [FPGA admission controller webhook](cmd/fpga_admissionwebhook/README.md) +is responsible for performing mapping from user-friendly function IDs to the +Interface ID and Bitstream ID that are required for FPGA programming. It also +implements access control by namespacing FPGA configuration information. + +#### CRI-O Prestart Hook + +The [FPGA prestart CRI-O hook](cmd/fpga_crihook/README.md) performs discovery +of the requested FPGA function bitstream and programs FPGA devices based on the +environment variables in the workload description. + +### [QAT](https://developer.intel.com/quickassist) Device Plugin + +The [QAT plugin](cmd/qat_plugin/README.md) supports device plugin for Intel QAT adapters, and includes +code [showing deployment](cmd/qat_plugin/dpdkdrv) via [DPDK](https://doc.dpdk.org/guides/cryptodevs/qat.html). + +The demo subdirectory includes details of both a +[QAT DPDK demo](demo/readme.md#intel-quickassist-technology-device-plugin-with-dpdk-demo-video) +and a [QAT OpenSSL demo](demo/readme.md#intel-quickassist-technology-device-plugin-openssl-demo-video). +Source for the OpenSSL demo can be found in the [relevant subdirectory](demo/openssl-qat-engine). + +Details for integrating the QAT device plugin into [Kata Containers](https://katacontainers.io/) +can be found in the +[Kata Containers documentation repository](https://github.com/kata-containers/kata-containers/blob/main/docs/use-cases/using-Intel-QAT-and-kata.md). + +### VPU Device Plugin + +The [VPU device plugin](cmd/vpu_plugin/README.md) supports Intel VCAC-A card +(https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/media-analytics-vcac-a-accelerator-card-by-celestica-datasheet.pdf) +the card has: +- 1 Intel Core i3-7100U processor +- 12 MyriadX VPUs +- 8GB DDR4 memory + +The demo subdirectory includes details of a OpenVINO deployment and use of the +VPU plugin. Sources can be found in [openvino-demo](demo/ubuntu-demo-openvino). + +### SGX Device Plugin + +The [SGX device plugin](cmd/sgx_plugin/README.md) allows workloads to use +Intel® Software Guard Extensions (Intel® SGX) on +platforms with SGX Flexible Launch Control enabled, e.g.,: + +- 3rd Generation Intel® Xeon® Scalable processor family, code-named “Ice Lake” +- Intel® Xeon® E3 processor +- Intel® NUC Kit NUC7CJYH + +The Intel SGX plugin comes in three parts. + +- the [device plugin](#sgx-plugin) +- the [admission webhook](#sgx-admission-webhook) +- the [SGX EPC memory registration](#sgx-epc-memory-registration) + +The demo subdirectory contains a [video](demo/readme.md#intel-sgx-device-plugin-demo-video) showing the deployment +and use of the Intel SGX device plugin. Sources relating to the demo can be found in the +[sgx-sdk-demo](demo/sgx-sdk-demo) and [sgx-aesmd-demo](demo/sgx-aesmd-demo) subdirectories. + +Brief overviews of the Intel SGX sub-components are given below. + + +#### device plugin + +The [SGX device plugin](cmd/sgx_plugin/README.md) is responsible for discovering +and reporting Intel SGX device nodes to `kubelet`. + +Containers requesting Intel SGX resources in the cluster should not use the +device plugins resources directly. + +#### Intel SGX Admission Webhook + +The Intel SGX admission webhook is responsible for performing Pod mutations based on +the `sgx.intel.com/quote-provider` pod annotation set by the user. The purpose +of the webhook is to hide the details of setting the necessary device resources +and volume mounts for using Intel SGX remote attestation in the cluster. Furthermore, +the Intel SGX admission webhook is responsible for writing a pod/sandbox +`sgx.intel.com/epc` annotation that is used by Kata Containers to dynamically +adjust its virtualized Intel SGX encrypted page cache (EPC) bank(s) size. + +The Intel SGX admission webhook is available as part of +[Intel Device Plugin Operator](cmd/operator/README.md) or +as a standalone [SGX Admission webhook image](cmd/sgx_admissionwebhook/README.md). + +#### Intel SGX EPC memory registration + +The Intel SGX EPC memory available on each node is registered as a Kubernetes extended resource using +node-feature-discovery (NFD). A custom NFD source hook is installed as part of +[SGX device plugin](cmd/sgx_plugin/README.md) +operator deployment and NFD is configured to register the Intel SGX EPC memory +extended resource reported by the hook. + +Containers requesting Intel SGX EPC resources in the cluster use +`sgx.intel.com/epc` resource which is of +type [memory](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory). + +### DSA Device Plugin + +The [DSA device plugin](cmd/dsa_plugin/README.md) supports acceleration using +the Intel Data Streaming accelerator(DSA). + +### DLB Device Plugin + +The [DLB device plugin](cmd/dlb_plugin/README.md) supports Intel Dynamic Load +Balancer accelerator(DLB). + +### IAA Device Plugin + +The [IAA device plugin](cmd/iaa_plugin/README.md) supports acceleration using +the Intel Analytics accelerator(IAA). + +## Device Plugins Operator + +To simplify the deployment of the device plugins, a unified device plugins +operator is implemented. + +Currently the operator has support for the DSA, DLB, FPGA, GPU, IAA, QAT, and +Intel SGX device plugins. Each device plugin has its own custom resource +definition (CRD) and the corresponding controller that watches CRUD operations +to those custom resources. + +The [Device plugins operator README](cmd/operator/README.md) gives the installation and usage details for the community operator available on [operatorhub.io](https://operatorhub.io/operator/intel-device-plugins-operator). + +The [Device plugins Operator for OCP](cmd/operator/ocp_quickstart_guide/README.md) gives the installation and usage details for the operator available on [Red Hat OpenShift Container Platform](https://catalog.redhat.com/software/operators/detail/61e9f2d7b9cdd99018fc5736). + +## XeLink XPU-Manager Sidecar + +To support interconnected GPUs in Kubernetes, XeLink sidecar is needed. + +The [XeLink XPU-Manager sidecar README](cmd/xpumanager_sidecar/README.md) gives information how the sidecar functions and how to use it. + +## Demos + +The [demo subdirectory](demo/readme.md) contains a number of demonstrations for +a variety of the available plugins. + +## Workload Authors + +For workloads to get accesss to devices managed by the plugins, the +`Pod` spec must specify the hardware resources needed: + +``` +spec: + containers: + - name: demo-container + image: /: + resources: + limits: + /: X +``` + +The summary of resources available via plugins in this repository is given in the list below. + +**Device Namespace : Registered Resource(s)** + * `dlb.intel.com` : `pf` or `vf` + * [dlb-libdlb-demo-pod.yaml](demo/dlb-libdlb-demo-pod.yaml) + * `dsa.intel.com` : `wq-user-[shared or dedicated]` + * [dsa-accel-config-demo-pod.yaml](demo/dsa-accel-config-demo-pod.yaml) + * `fpga.intel.com` : custom, see [mappings](cmd/fpga_admissionwebhook/README.md#mappings) + * [intelfpga-job.yaml](demo/intelfpga-job.yaml) + * `gpu.intel.com` : `i915` + * [intelgpu-job.yaml](demo/intelgpu-job.yaml) + * `iaa.intel.com` : `wq-user-[shared or dedicated]` + * [iaa-qpl-demo-pod.yaml](demo/iaa-qpl-demo-pod.yaml) + * `qat.intel.com` : `generic` or `cy`/`dc` + * [crypto-perf-dpdk-pod-requesting-qat.yaml](deployments/qat_dpdk_app/base/crypto-perf-dpdk-pod-requesting-qat.yaml) + * `sgx.intel.com` : `epc` + * [intelsgx-job.yaml](deployments/sgx_enclave_apps/base/intelsgx-job.yaml) + * `vpu.intel.com` : `hddl` + * [intelvpu-job.yaml](demo/intelvpu-job.yaml) + +## Developers + +For information on how to develop a new plugin using the framework or work on development task in +this repository, see the [Developers Guide](DEVEL.md). + +## Supported Kubernetes Versions + +Releases are made under the github [releases area](https://github.com/intel/intel-device-plugins-for-kubernetes/releases). Supported releases and +matching Kubernetes versions are listed below: + +| Branch | Kubernetes branch/version | Status | +|:------------------|:-------------------------------|:------------| +| release-0.26 | Kubernetes 1.26 branch v1.26.x | supported | +| release-0.25 | Kubernetes 1.25 branch v1.25.x | supported | +| release-0.24 | Kubernetes 1.24 branch v1.24.x | supported | +| release-0.23 | Kubernetes 1.23 branch v1.23.x | unsupported | +| release-0.22 | Kubernetes 1.22 branch v1.22.x | unsupported | +| release-0.21 | Kubernetes 1.21 branch v1.21.x | unsupported | +| release-0.20 | Kubernetes 1.20 branch v1.20.x | unsupported | +| release-0.19 | Kubernetes 1.19 branch v1.19.x | unsupported | +| release-0.18 | Kubernetes 1.18 branch v1.18.x | unsupported | +| release-0.17 | Kubernetes 1.17 branch v1.17.x | unsupported | +| release-0.15 | Kubernetes 1.15 branch v1.15.x | unsupported | +| release-0.11 | Kubernetes 1.11 branch v1.11.x | unsupported | + +[Go environment]: https://golang.org/doc/install +[Kubernetes cluster]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ + +## Pre-built plugin images + +Pre-built images of the plugins are available on the Docker hub. These images +are automatically built and uploaded to the hub from the latest main branch of +this repository. + +Release tagged images of the components are also available on the Docker hub, +tagged with their release version numbers in the format x.y.z, corresponding to +the branches and releases in this repository. + +**Note:** the default deployment files and operators are configured with +[imagePullPolicy](https://kubernetes.io/docs/concepts/containers/images/#updating-images) +```IfNotPresent``` and can be changed with ```scripts/set-image-pull-policy.sh```. + +## License + +All of the source code required to build intel-device-plugins-for-kubernetes +is available under Open Source licenses. The source code files identify external Go +modules used. Binaries are distributed as container images on +DockerHub*. Those images contain license texts and source code under `/licenses`. + +### Helm Charts + +Device Plugins Helm Charts are located in Intel Helm Charts repository [Intel Helm Charts](https://github.com/intel/helm-charts). This is another way of distributing Kubernetes resources of the device plugins framework. + +To add repo: +``` +helm repo add intel https://intel.github.io/helm-charts +``` diff --git a/0.26/_sources/SECURITY.md.txt b/0.26/_sources/SECURITY.md.txt new file mode 100644 index 000000000..e342bcacb --- /dev/null +++ b/0.26/_sources/SECURITY.md.txt @@ -0,0 +1,13 @@ +**Reporting a Potential Security Vulnerability**: If you have discovered +potential security vulnerability in this project, please send an e-mail to +secure@intel.com. Encrypt sensitive information using our +[PGP public key](https://www.intel.com/content/www/us/en/security-center/pgp-public-key.html). + +Please provide as much information as possible, including: + - The projects and versions affected + - Detailed description of the vulnerability + - Information on known exploits + +A member of the Intel Product Security Team will review your e-mail and +contact you to collaborate on resolving the issue. For more information on +how Intel works to resolve security issues, see [Vulnerability Handling Guidelines](https://www.intel.com/content/www/us/en/security-center/vulnerability-handling-guidelines.html). diff --git a/0.26/_sources/cmd/dlb_plugin/README.md.txt b/0.26/_sources/cmd/dlb_plugin/README.md.txt new file mode 100644 index 000000000..08ff0c51e --- /dev/null +++ b/0.26/_sources/cmd/dlb_plugin/README.md.txt @@ -0,0 +1,228 @@ +# Intel DLB device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Installation](#installation) + * [Pre-built Images](#pre-built-images) + * [Verify Plugin Registration](#verify-plugin-registration) +* [Testing and Demos](#testing-and-demos) + +## Introduction + +This Intel DLB device plugin provides support for [Intel DLB](https://builders.intel.com/docs/networkbuilders/SKU-343247-001US-queue-management-and-load-balancing-on-intel-architecture.pdf) devices under Kubernetes. + +### DLB2 driver configuration for PFs +The DLB device plugin requires a Linux Kernel DLB driver to be installed and enabled to operate. Get [DLB software release](https://www.intel.com/content/www/us/en/download/686372/intel-dynamic-load-balancer.html), build and load the dlb2 driver module following the instruction of 'DLB_Driver_User_Guide.pdf' in the directory 'dlb/docs'. + +After successfully loading the module, available dlb device nodes are visible in devfs. +```bash +$ ls -1 /dev/dlb* +/dev/dlb0 /dev/dlb1 /dev/dlb2 ... +``` + +### VF configuration using a DPDK tool (but with dlb2 driver) +If you configure SR-IOV/VF (virtual functions), continue the following configurations. This instruction uses DPDK tool to check eventdev devices, unbind a VF device, and bind dlb2 driver to a VF device. + +Patch dpdk sources to work with DLB: +```bash +$ wget -q https://fast.dpdk.org/rel/dpdk-21.11.tar.xz -O- | tar -Jx +$ wget -q https://downloadmirror.intel.com/734482/dlb_linux_src_release_7.7.0_2022_06_17.txz -O- | tar -Jx +$ cd ./dpdk-*/ && patch -p1 < ../dlb/dpdk/dpdk_dlb_*_diff.patch +$ sed -i 's/270b,2710,2714/270b,2710,2711,2714/g' ./usertools/dpdk-devbind.py +``` + +List eventdev devices: +```bash +$ ./usertools/dpdk-devbind.py -s | grep -A10 ^Eventdev +Eventdev devices using kernel driver +==================================== +0000:6d:00.0 'Device 2710' drv=dlb2 unused= +0000:72:00.0 'Device 2710' drv=dlb2 unused= +... +``` + +Enable virtual functions: +```bash +$ echo 4 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/sriov_numvfs +``` +> **Note:**: If it fails saying "No such file or directory," it may be bound to vfio-pci driver. Bind the device to dlb2 driver. + +Check if new dlb device nodes appear: +```bash +$ ls -1 /dev/dlb* +/dev/dlb0 /dev/dlb1 /dev/dlb10 /dev/dlb11 ... /dev/dlb8 /dev/dlb9 +``` + +Check that new eventdev devices appear: +```bash +$ ./usertools/dpdk-devbind.py -s | grep -A14 ^Eventdev +Eventdev devices using kernel driver +==================================== +0000:6d:00.0 'Device 2710' drv=dlb2 unused= +0000:6d:00.1 'Device 2711' drv=dlb2 unused= +0000:6d:00.2 'Device 2711' drv=dlb2 unused= +0000:6d:00.3 'Device 2711' drv=dlb2 unused= +0000:6d:00.4 'Device 2711' drv=dlb2 unused= +0000:72:00.0 'Device 2710' drv=dlb2 unused= +... +``` + +Assign PF resources to VF: +> **Note:**: The process below is only for the first vf resource among 4 resources. Repeat for other vfN_resources in /sys/bus/pci/devices/0000\:6d\:00.0/, and then bind dlb2 driver to 0000:6d:00.M that corresponds to vfN_resources. + +- Unbind driver from the VF device before configuring it. +```bash +$ sudo ./usertools/dpdk-devbind.py --unbind 0000:6d:00.1 +``` + +- Assign PF resources to VF: +```bash +$ echo 2048 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_atomic_inflights && + echo 2048 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_dir_credits && + echo 64 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_dir_ports && + echo 2048 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_hist_list_entries && + echo 8192 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_ldb_credits && + echo 64 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_ldb_ports && + echo 32 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_ldb_queues && + echo 32 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_sched_domains && + echo 2 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_sn0_slots && + echo 2 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_sn1_slots +``` + +- Bind driver back to the VF device: +```bash +$ sudo ./usertools/dpdk-devbind.py --bind dlb2 0000:6d:00.1 +``` + + +### Verification of well-configured devices: +Run libdlb example app: +> **Note:**: Alternative way is to use this [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/dlb-libdlb-demo/Dockerfile) for running tests. + +```bash +$ ls +dlb dpdk-21.11 +$ cd ./dlb/libdlb/ && make && sudo LD_LIBRARY_PATH=$PWD ./examples/dir_traffic -n 128 -d 1 +# For running test for /dev/dlbN, replace 1 with N. +``` + +Run dpdk example app: +> **Note:**: Alternative way is to use this [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/dlb-dpdk-demo/Dockerfile) for patching and building DPDK and running tests. + +- Install build dependencies and build dpdk: +```bash +$ sudo apt-get update && sudo apt-get install build-essential meson python3-pyelftools libnuma-dev python3-pip && sudo pip install ninja +# This configuration is based on Ubuntu/Debian distribution. For other distributions that do not use apt, install the dependencies using another way. +$ ls +dlb dpdk-21.11 +$ cd ./dpdk-* && meson setup --prefix $(pwd)/installdir builddir && ninja -C builddir install +``` + +- Run eventdev test +```bash +sudo ./builddir/app/dpdk-test-eventdev --no-huge --vdev='dlb2_event,dev_id=1' -- --test=order_queue --nb_flows 64 --nb_pkts 512 --plcores 1 --wlcores 2-7 +# For running test for /dev/dlbN, replace 1 with N. +``` + +## Installation + +The following sections detail how to obtain, build, deploy and test the DLB device plugin. + +Examples are provided showing how to deploy the plugin either using a DaemonSet or by hand on a per-node basis. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-dlb-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/dlb_plugin?ref=' +daemonset.apps/intel-dlb-plugin created +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +Nothing else is needed. See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin. + +### Verify Plugin Registration + +You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes: + +```bash +$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{" "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\( dlb\)' +master + dlb.intel.com/pf: 7 + dlb.intel.com/vf: 4 +``` + +## Testing and Demos + +We can test the plugin is working by deploying the provided example test images (dlb-libdlb-demo and dlb-dpdk-demo). + +1. Build a Docker image and create a pod running unit tests off the local Docker image: + + ```bash + $ make dlb-libdlb-demo + ... + Successfully tagged intel/dlb-libdlb-demo:devel + + $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/dlb-libdlb-demo-pod.yaml + pod/dlb-libdlb-demo-pod created + ``` + + ```bash + $ make dlb-dpdk-demo + ... + Successfully tagged intel/dlb-dpdk-demo:devel + + $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/dlb-dpdk-demo-pod.yaml + pod/dlb-dpdk-demo-pod created + ``` + +1. Wait until pod is completed: + + ```bash + $ kubectl get pods | grep dlb-.*-demo + NAME READY STATUS RESTARTS AGE + dlb-dpdk-demo 0/2 Completed 0 79m + dlb-libdlb-demo 0/2 Completed 0 18h + ``` + +1. Review the job's logs: + + ```bash + $ kubectl logs dlb-libdlb-demo + + ``` + + ```bash + $ kubectl logs dlb-dpdk-demo + + ``` + + If the pod did not successfully launch, possibly because it could not obtain the DLB + resource, it will be stuck in the `Pending` status: + + ```bash + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + dlb-dpdk-demo 0/2 Pending 0 3s + dlb-libdlb-demo 0/2 Pending 0 10s + ``` + + This can be verified by checking the Events of the pod: + + ```bash + $ kubectl describe pod dlb-libdlb-demo | grep -A3 Events: + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 85s default-scheduler 0/1 nodes are available: 1 Insufficient dlb.intel.com/pf, 1 Insufficient dlb.intel.com/vf. + ``` diff --git a/0.26/_sources/cmd/dsa_plugin/README.md.txt b/0.26/_sources/cmd/dsa_plugin/README.md.txt new file mode 100644 index 000000000..416ddf9cc --- /dev/null +++ b/0.26/_sources/cmd/dsa_plugin/README.md.txt @@ -0,0 +1,135 @@ +# Intel DSA device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Installation](#installation) + * [Pre-built Images](#pre-built-images) + * [Verify Plugin Registration](#verify-plugin-registration) +* [Testing and Demos](#testing-and-demos) + +## Introduction + +The DSA device plugin for Kubernetes supports acceleration using the Intel Data Streaming accelerator(DSA). + +The DSA plugin discovers DSA work queues and presents them as a node resources. + +The DSA plugin and operator optionally support provisioning of DSA devices and workqueues with the help of [accel-config](https://github.com/intel/idxd-config) utility through initcontainer. + +## Installation + +The following sections detail how to use the DSA device plugin. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-dsa-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/dsa_plugin?ref=' +daemonset.apps/intel-dsa-plugin created +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +Nothing else is needed. See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin. + +#### Automatic Provisioning + +There's a sample [idxd initcontainer](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/build/docker/intel-idxd-config-initcontainer.Dockerfile) included that provisions DSA devices and workqueues (1 engine / 1 group / 1 wq (user/dedicated)), to deploy: + +```bash +$ kubectl apply -k deployments/dsa_plugin/overlays/dsa_initcontainer/ +``` + +The provisioning [script](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/idxd-init.sh) and [template](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/master/demo/dsa.conf) are available for customization. + +The provisioning config can be optionally stored in the ProvisioningConfig configMap which is then passed to initcontainer through the volume mount. + +There's also a possibility for a node specific congfiguration through passing a nodename via NODE_NAME into initcontainer's environment and passing a node specific profile via configMap volume mount. + +To create a custom provisioning config: + +```bash +$ kubectl create configmap --namespace=inteldeviceplugins-system intel-dsa-config --from-file=demo/dsa.conf +``` + +### Verify Plugin Registration +You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes: + +```bash +$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{" "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\( dsa\)' +master + dsa.intel.com/wq-user-dedicated: 2 + dsa.intel.com/wq-user-shared: 8 +node1 + dsa.intel.com/wq-user-dedicated: 4 + dsa.intel.com/wq-user-shared: 20 +``` + +## Testing and Demos + +We can test the plugin is working by deploying the provided example accel-config test image. + +1. Build a Docker image with an accel-config tests: + + ```bash + $ make accel-config-demo + ... + Successfully tagged accel-config-demo:devel + ``` + +1. Create a pod running unit tests off the local Docker image: + + ```bash + $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/dsa-accel-config-demo-pod.yaml + pod/dsa-accel-config-demo created + ``` + +1. Wait until pod is completed: + + ```bash + $ kubectl get pods |grep dsa-accel-config-demo + dsa-accel-config-demo 0/1 Completed 0 31m + +1. Review the job's logs: + + ```bash + $ kubectl logs dsa-accel-config-demo | tail + [debug] PF in sub-task[6], consider as passed + [debug] PF in sub-task[7], consider as passed + [debug] PF in sub-task[8], consider as passed + [debug] PF in sub-task[9], consider as passed + [debug] PF in sub-task[10], consider as passed + [debug] PF in sub-task[11], consider as passed + [debug] PF in sub-task[12], consider as passed + [debug] PF in sub-task[13], consider as passed + [debug] PF in sub-task[14], consider as passed + [debug] PF in sub-task[15], consider as passed + ``` + + If the pod did not successfully launch, possibly because it could not obtain the DSA + resource, it will be stuck in the `Pending` status: + + ```bash + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + dsa-accel-config-demo 0/1 Pending 0 7s + ``` + + This can be verified by checking the Events of the pod: + + ```bash + + $ kubectl describe pod dsa-accel-config-demo | grep -A3 Events: + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 2m26s default-scheduler 0/1 nodes are available: 1 Insufficient dsa.intel.com/wq-user-dedicated, 1 Insufficient dsa.intel.com/wq-user-shared. + ``` diff --git a/0.26/_sources/cmd/fpga_admissionwebhook/README.md.txt b/0.26/_sources/cmd/fpga_admissionwebhook/README.md.txt new file mode 100644 index 000000000..89daa3b0d --- /dev/null +++ b/0.26/_sources/cmd/fpga_admissionwebhook/README.md.txt @@ -0,0 +1,186 @@ +# Intel FPGA admission controller for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Dependencies](#dependencies) +* [Installation](#installation) + * [Pre-requisites](#pre-requisites) + * [Mappings](#mappings) + * [Deployment](#deployment) + * [Webhook deployment](#webhook-deployment) + * [Mappings deployment](#mappings-deployment) +* [Next steps](#next-steps) + +## Introduction + +The FPGA admission controller is one of the components used to add support for Intel FPGA +devices to Kubernetes. + +> **NOTE:** Installation of the FPGA admission controller can be skipped if the +> [FPGA device plugin](../fpga_plugin/README.md) is operated with the Intel Device Plugins Operator +> since it integrates the controller's functionality. + +The FPGA admission controller webhook is responsible for performing mapping from user-friendly +function IDs to the Interface ID and Bitstream ID that are required for FPGA programming by +the [FPGA CRI-O hook](../fpga_crihook/README.md). + +Mappings are stored in namespaced custom resource definition (CRD) objects, therefore the admission +controller also performs access control, determining which bitstream can be used for which namespace. +More details can be found in the [Mappings](#mappings) section. + +The admission controller also keeps the user from bypassing namespaced mapping restrictions, +by denying admission of any pods that are trying to use internal knowledge of InterfaceID or +Bitstream ID environment variables used by the prestart hook. + +## Dependencies + +This component is one of a set of components that work together. You may also want to +install the following: + +- [FPGA device plugin](../fpga_plugin/README.md) +- [FPGA prestart CRI-O hook](../fpga_crihook/README.md) + +All components have the same basic dependencies as the +[generic plugin framework dependencies](../../README.md#about) + +## Installation + +The following sections detail how to obtain, build and deploy the admission +controller webhook plugin. + +### Pre-requisites + +The default webhook deployment depends on having [cert-manager](https://cert-manager.io/) +installed. See its installation instructions [here](https://cert-manager.io/docs/installation/kubectl/). + +Also if your cluster operates behind a corporate proxy make sure that the API +server is configured not to send requests to cluster services through the +proxy. You can check that with the following command: + +```bash +$ kubectl describe pod kube-apiserver --namespace kube-system | grep -i no_proxy | grep "\.svc" +``` + +In case there's no output and your cluster was deployed with `kubeadm` open +`/etc/kubernetes/manifests/kube-apiserver.yaml` at the control plane nodes and +append `.svc` and `.svc.cluster.local` to the `no_proxy` environment variable: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + ... +spec: + containers: + - command: + - kube-apiserver + - --advertise-address=10.237.71.99 + ... + env: + - name: http_proxy + value: http://proxy.host:8080 + - name: https_proxy + value: http://proxy.host:8433 + - name: no_proxy + value: 127.0.0.1,localhost,.example.com,10.0.0.0/8,.svc,.svc.cluster.local + ... +``` + +**Note:** To build clusters using `kubeadm` with the right `no_proxy` settings from the very beginning, +set the cluster service names to `$no_proxy` before `kubeadm init`: + +``` +$ export no_proxy=$no_proxy,.svc,.svc.cluster.local +``` + +## Mappings + +Mappings is a an essential part of the setup that gives a flexible instrument to a cluster +administrator to manage FPGA bitstreams and to control access to them. Being a set of +custom resource definitions they are used to configure the way FPGA resource requests get +translated into actual resources provided by the cluster. + +For the following mapping + +```yaml +apiVersion: fpga.intel.com/v2 +kind: AcceleratorFunction +metadata: + name: arria10.dcp1.2-nlb0-preprogrammed +spec: + afuId: d8424dc4a4a3c413f89e433683f9040b + interfaceId: 69528db6eb31577a8c3668f9faa081f6 + mode: af +``` + +requested FPGA resources are translated to AF resources. For example, +`fpga.intel.com/arria10.dcp1.2-nlb0-preprogrammed` is translated to +`fpga.intel.com/af-695.d84.aVKNtusxV3qMNmj5-qCB9thCTcSko8QT-J5DNoP5BAs` where the `af-` +prefix indicates the plugin's mode (`af`), `695` is the first three characters of +the region interface ID, `d84` is the first three characters of the accelerator function ID +and the last part `aVKNtusxV3qMNmj5-qCB9thCTcSko8QT-J5DNoP5BAs` is a base64-encoded concatenation +of the full region interface ID and accelerator function ID. +The format of resource names (e.g. `arria10.dcp1.2-nlb0-preprogrammed`) can be any and is up +to a cluster administrator. + +The same mapping, but with its mode field set to `region`, would translate +`fpga.intel.com/arria10.dcp1.2-nlb0-preprogrammed` to `fpga.intel.com/region-69528db6eb31577a8c3668f9faa081f6`, +and the corresponding AF IDs are set in environment variables for the container. +Though in this case the cluster administrator would probably want to rename +the mapping `arria10.dcp1.2-nlb0-preprogrammed` to something like `arria10.dcp1.2-nlb0-orchestrated` +to reflect its mode. The [FPGA CRI-O hook](../fpga_crihook/README.md) then loads the requested +bitstream to a region before the container is started. + +Mappings of resource names are configured with objects of `AcceleratorFunction` and +`FpgaRegion` custom resource definitions found respectively in +[`./deployment/fpga_admissionwebhook/crd/bases/fpga.intel.com_af.yaml`](/deployments/fpga_admissionwebhook/crd/bases/fpga.intel.com_acceleratorfunctions.yaml) +and [`./deployment/fpga_admissionwebhook/crd/bases/fpga.intel.com_region.yaml`](/deployments/fpga_admissionwebhook/crd/bases/fpga.intel.com_fpgaregions.yaml). + +Example mappings between 'names' and 'ID's are controlled by the admission controller mappings collection file found in +[`./deployments/fpga_admissionwebhook/mappings-collection.yaml`](/deployments/fpga_admissionwebhook/mappings-collection.yaml). + + +### Deployment + +#### Webhook deployment + +To deploy the webhook, run + +```bash +$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/fpga_admissionwebhook/default?ref=main +namespace/intelfpgawebhook-system created +customresourcedefinition.apiextensions.k8s.io/acceleratorfunctions.fpga.intel.com created +customresourcedefinition.apiextensions.k8s.io/fpgaregions.fpga.intel.com created +mutatingwebhookconfiguration.admissionregistration.k8s.io/intelfpgawebhook-mutating-webhook-configuration created +clusterrole.rbac.authorization.k8s.io/intelfpgawebhook-manager-role created +clusterrolebinding.rbac.authorization.k8s.io/intelfpgawebhook-manager-rolebinding created +service/intelfpgawebhook-webhook-service created +deployment.apps/intelfpgawebhook-webhook created +certificate.cert-manager.io/intelfpgawebhook-serving-cert created +issuer.cert-manager.io/intelfpgawebhook-selfsigned-issuer created +``` + +#### Mappings deployment + +Mappings deployment is a mandatory part of the webhook deployment. You should +prepare and deploy mappings that describe FPGA bitstreams available in your cluster. + +Example mappings collection [`./deployments/fpga_admissionwebhook/mappings-collection.yaml`](/deployments/fpga_admissionwebhook/mappings-collection.yaml) +can be used as an example for cluster mappings. This collection is not intended to be deployed as is, +it should be used as a reference and example of your own cluster mappings. + +To deploy the mappings, run + +```bash +$ kubectl apply -f + +``` + +Note that the mappings are scoped to the namespaces they were created in +and they are applicable to pods created in the corresponding namespaces. + + +## Next steps + +Continue with [FPGA prestart CRI-O hook](../fpga_crihook/README.md). diff --git a/0.26/_sources/cmd/fpga_crihook/README.md.txt b/0.26/_sources/cmd/fpga_crihook/README.md.txt new file mode 100644 index 000000000..59c26fd8a --- /dev/null +++ b/0.26/_sources/cmd/fpga_crihook/README.md.txt @@ -0,0 +1,49 @@ +# Intel FPGA prestart CRI-O webhook for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Dependencies](#dependencies) +* [Configuring CRI-O](#configuring-cri-o) + +## Introduction + +The FPGA CRI-O webhook is one of the components used to add support for Intel FPGA +devices to Kubernetes. + +The FPGA prestart CRI-O hook is triggered by container annotations, such as set by the +[FPGA device plugin](../fpga_plugin/README.md). It performs discovery of the requested FPGA +function bitstream and then programs FPGA devices based on the environment variables +in the workload description. + +The CRI-O prestart hook is only *required* when the +[FPGA admission webhook](../fpga_admissionwebhook/README.md) is configured for orchestration +programmed mode, and is benign (un-used) otherwise. + +> **Note:** The fpga CRI-O webhook is usually installed by the same DaemonSet as the +> FPGA device plugin. If building and installing the CRI-O webhook by hand, it is +> recommended you reference the +> [fpga plugin DaemonSet YAML](/deployments/fpga_plugin/base/intel-fpga-plugin-daemonset.yaml ) for +> more details. + +## Dependencies + +This component is one of a set of components that work together. You may also want to +install the following: + +- [FPGA device plugin](../fpga_plugin/README.md) +- [FPGA admission controller](../fpga_admissionwebhook/README.md) + +All components have the same basic dependencies as the +[generic plugin framework dependencies](../../README.md#about) + +See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the CRI hook. + +## Configuring CRI-O + +Recent versions of [CRI-O](https://github.com/cri-o/cri-o) are shipped with default configuration +file that prevents CRI-O to discover and configure hooks automatically. +For FPGA orchestration programmed mode, the OCI hooks are the key component. +Please ensure that your `/etc/crio/crio.conf` parameter `hooks_dir` is either unset +(to enable default search paths for OCI hooks configuration) or contains the directory +`/etc/containers/oci/hooks.d`. diff --git a/0.26/_sources/cmd/fpga_plugin/README.md.txt b/0.26/_sources/cmd/fpga_plugin/README.md.txt new file mode 100644 index 000000000..b726e461c --- /dev/null +++ b/0.26/_sources/cmd/fpga_plugin/README.md.txt @@ -0,0 +1,228 @@ +# Intel FPGA device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) + * [Component Overview](#component-overview) +* [Modes and Configuration Options](#modes-and-configuration-options) +* [Installation](#installation) + * [Prerequisites](#prerequisites) + * [Pre-built Images](#pre-built-images) + * [Verify Plugin Registration](#verify-plugin-registration) + +## Introduction + +This FPGA device plugin is part of a collection of Kubernetes components found within this +repository that enable integration of Intel FPGA hardware into Kubernetes. + +The following hardware platforms are supported: + +- Intel Arria 10 +- Intel Stratix 10 + +The components support the [Open Programmable Acceleration Engine (OPAE)](https://opae.github.io/latest/index.html) +interface. + +The components together implement the following features: + +- discovery of pre-programmed accelerator functions +- discovery of programmable regions +- orchestration of FPGA programming +- access control for FPGA hardware + +### Component Overview + +The following components are part of this repository, and work together to support Intel FPGAs under +Kubernetes: + +- [FPGA device plugin](README.md) (this component) + + A Kubernetes [device plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) + that discovers available FPGA resources on a node and advertises them to the Kubernetes control plane + via the node kubelet. + +- [FPGA admission controller webhook](../fpga_admissionwebhook/README.md) + + A Kubernetes [admission controller webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) + which can be used to dynamically convert logical resource names in pod specifications into actual FPGA + resource names, as advertised by the device plugin. + + The webhook can also set environment variables to instruct the CRI-O prestart hook to program the FPGA + before launching the container. + + > **NOTE:** Installation of the [FPGA admission controller webhook](../fpga_admissionwebhook/README.md) can be skipped if the + > FPGA device plugin is operated with the Intel Device Plugins Operator + > since it integrates the controller's functionality. + > However, [the mappings](../fpga_admissionwebhook/README.md#mappings-deployment) still must be deployed." + +- [FPGA CRI-O prestart hook](../fpga_crihook/README.md) + + A [CRI-O](https://github.com/cri-o/cri-o) prestart hook that, upon instruction from the FPGA admission + controller, allocates and programs the FPGA before the container is launched. + +The repository also contains an [FPGA helper tool](../fpga_tool/README.md) that may be useful during +development, initial deployment and debugging. + +### Modes and Configuration Options + +The FPGA plugin set can run in one of two modes: + +- `region` mode, where the plugins locate and advertise + regions of the FPGA, and facilitate programing of those regions with the + requested bistreams. +- `af` mode, where the FPGA bitstreams are already loaded + onto the FPGA, and the plugins discover and advertises the existing + Accelerator Functions (AF). + +The example YAML deployments described in this document only currently support +`af` mode. To utilise `region` mode, either modify the existing YAML appropriately, +or deploy 'by hand'. + +Overview diagrams of `af` and `region` modes are below: + +region mode: + +![Overview of `region` mode](pictures/FPGA-region.png) + +af mode: + +![Overview of `af` mode](pictures/FPGA-af.png) + +## Installation + +The below sections cover how to use this component. + +### Prerequisites + +All components have the same basic dependencies as the +[generic plugin framework dependencies](../../README.md#about) + +To obtain a fully operational FPGA enabled cluster, you must install all three +major components: + +- [FPGA device plugin](README.md) (this component) +- [FPGA admission controller webhook](../fpga_admissionwebhook/README.md) +- [FPGA prestart CRI-O hook](../fpga_crihook/README.md) + +The CRI-O hook is only *required* if `region` mode is being used, but is installed by default by the +[FPGA plugin DaemonSet YAML](/deployments/fpga_plugin/base/intel-fpga-plugin-daemonset.yaml), and is benign +in `af` mode. + +If using the `af` mode, and therefore *not* using the +CRI-O prestart hook, runtimes other than CRI-O can be used (that is, the CRI-O hook presently +*only* works with the CRI-O runtime). + +The FPGA device plugin requires a Linux Kernel FPGA driver to be installed and enabled to +operate. The plugin supports the use of either of following two drivers, and auto detects +which is present and thus to use: + +- The Linux Kernel in-tree [DFL](https://www.kernel.org/doc/html/latest/fpga/dfl.html) driver +- The out of tree [OPAE](https://opae.github.io/latest/docs/drv_arch/drv_arch.html) driver + +Install this component (FPGA device plugin) first, and then follow the links +and instructions to install the other components. + +The FPGA webhook deployment depends on having [cert-manager](https://cert-manager.io/) +installed. See its installation instructions [here](https://cert-manager.io/docs/installation/kubectl/). + +```bash +$ kubectl get pods -n cert-manager +NAME READY STATUS RESTARTS AGE +cert-manager-7747db9d88-bd2nl 1/1 Running 0 1m +cert-manager-cainjector-87c85c6ff-59sb5 1/1 Running 0 1m +cert-manager-webhook-64dc9fff44-29cfc 1/1 Running 0 1m + +``` + +### Pre-built Images + +Pre-built images of the components are available on the [Docker hub](https://hub.docker.com/u/intel). +These images are automatically built and uploaded to the hub from the latest `main` branch of +this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers (of the form `x.y.z`, matching the branch/tag release number in this repo). + +The following images are available on the Docker hub: + +- [The FPGA plugin](https://hub.docker.com/r/intel/intel-fpga-plugin) +- [The FPGA admisson webhook](https://hub.docker.com/r/intel/intel-fpga-admissionwebhook) +- [The FPGA CRI-O prestart hook (in the `initcontainer` image)](https://hub.docker.com/r/intel/intel-fpga-initcontainer) + +Depending on the FPGA mode, run either +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/fpga_plugin/overlays/af?ref=' +namespace/intelfpgaplugin-system created +customresourcedefinition.apiextensions.k8s.io/acceleratorfunctions.fpga.intel.com created +customresourcedefinition.apiextensions.k8s.io/fpgaregions.fpga.intel.com created +mutatingwebhookconfiguration.admissionregistration.k8s.io/intelfpgaplugin-mutating-webhook-configuration created +clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-manager-role created +clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-node-getter created +clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-get-nodes created +clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-manager-rolebinding created +service/intelfpgaplugin-webhook-service created +deployment.apps/intelfpgaplugin-webhook created +daemonset.apps/intelfpgaplugin-fpgadeviceplugin created +certificate.cert-manager.io/intelfpgaplugin-serving-cert created +issuer.cert-manager.io/intelfpgaplugin-selfsigned-issuer created +``` +or +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/fpga_plugin/overlays/region?ref=' +namespace/intelfpgaplugin-system created +customresourcedefinition.apiextensions.k8s.io/acceleratorfunctions.fpga.intel.com created +customresourcedefinition.apiextensions.k8s.io/fpgaregions.fpga.intel.com created +mutatingwebhookconfiguration.admissionregistration.k8s.io/intelfpgaplugin-mutating-webhook-configuration created +clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-manager-role created +clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-node-getter created +clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-get-nodes created +clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-manager-rolebinding created +service/intelfpgaplugin-webhook-service created +deployment.apps/intelfpgaplugin-webhook created +daemonset.apps/intelfpgaplugin-fpgadeviceplugin created +certificate.cert-manager.io/intelfpgaplugin-serving-cert created +issuer.cert-manager.io/intelfpgaplugin-selfsigned-issuer created +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +The command should result in two pods running: +```bash +$ kubectl get pods -n intelfpgaplugin-system +NAME READY STATUS RESTARTS AGE +intelfpgaplugin-fpgadeviceplugin-skcw5 1/1 Running 0 57s +intelfpgaplugin-webhook-7d6bcb8b57-k52b9 1/1 Running 0 57s +``` + +If you need the FPGA plugin on some nodes to operate in a different mode then add this +annotation to the nodes: + +```bash +$ kubectl annotate node 'fpga.intel.com/device-plugin-mode=region' +``` +or +```bash +$ kubectl annotate node 'fpga.intel.com/device-plugin-mode=af' +``` +And restart the pods on the nodes. + +> **Note:** The FPGA plugin [DaemonSet YAML](/deployments/fpga_plugin/base/intel-fpga-plugin-daemonset.yaml) +> also deploys the [FPGA CRI-O hook](../fpga_crihook/README.md) `initcontainer` image, but it will be +> benign (un-used) when running the FPGA plugin in `af` mode. + +#### Verify Plugin Registration + +Verify the FPGA plugin has been deployed on the nodes. The below shows the output +you can expect in `region` mode, but similar output should be expected for `af` +mode: + +```bash +$ kubectl describe nodes | grep fpga.intel.com +fpga.intel.com/region-ce48969398f05f33946d560708be108a: 1 +fpga.intel.com/region-ce48969398f05f33946d560708be108a: 1 +``` + +> **Note:** The FPGA plugin [DaemonSet YAML](/deployments/fpga_plugin/fpga_plugin.yaml) +> also deploys the [FPGA CRI-O hook](../fpga_crihook/README.md) `initcontainer` image as well. You may +> also wish to build that image locally before deploying the FPGA plugin to avoid deploying +> the Docker hub default image. diff --git a/0.26/_sources/cmd/fpga_tool/README.md.txt b/0.26/_sources/cmd/fpga_tool/README.md.txt new file mode 100644 index 000000000..39752c890 --- /dev/null +++ b/0.26/_sources/cmd/fpga_tool/README.md.txt @@ -0,0 +1,29 @@ +# Intel FPGA test tool + +## Introduction + +This directory contains an FPGA test tool that can be used to locate, examine and program Intel +FPGAs. + +### Command line and usage + +The tool has the following command line arguments: + +```bash +info, fpgainfo, install, list, fmeinfo, portinfo, list-fme, list-port, pr, release, assign +``` + +and the following command line options: + +```bash +Usage of ./fpga_tool: + -b string + Path to bitstream file (GBS or AOCX) + -d string + Path to device node (FME or Port) + -dry-run + Don't write/program, just validate and log + -force + Force overwrite operation for installing bitstreams + -q Quiet mode. Only errors will be reported +``` \ No newline at end of file diff --git a/0.26/_sources/cmd/gpu_fakedev/README.md.txt b/0.26/_sources/cmd/gpu_fakedev/README.md.txt new file mode 100644 index 000000000..94938ea41 --- /dev/null +++ b/0.26/_sources/cmd/gpu_fakedev/README.md.txt @@ -0,0 +1,47 @@ +# Fake (GPU) device file generator + +Table of Contents +* [Introduction](#introduction) +* [Configuration](#configuration) +* [Potential improvements](#potential-improvements) +* [Related tools](#related-tools) + +## Introduction + +This is a tool for generating (large number of) fake device files for +k8s device scheduling scalability testing. But it can also be used +just to test (GPU) device plugin functionality without having +corresponding device HW. + +Its "intel-gpu-fakedev" container is intended to be run as first init +container in a device plugin pod, so that device plugin (and its NFD +labeler) see the fake (sysfs + devfs) files generated by the tool, +instead of real host sysfs and devfs content. + +## Configuration + +[Configs](configs/) subdirectory contains example JSON configuration +file(s) for the generator. Currently there's only one example JSON +file, but each new device variant adding feature(s) that have specific +support in device plugin, could have their own fake device config. + +## Potential improvements + +If support for mixed device environment is needed, tool can be updated +to use node / configuration file mapping. Such mappings could be e.g. +in configuration files themselves as node name include / exlude lists, +and tool would use first configuration file matching the node it's +running on. For now, one would need to use different pod / config +specs for different nodes to achieve that... + +Currently JSON config file options and the generated files are tied to +what GPU plugin uses, but if needed, they could be changed to fake +also sysfs + devfs device files used by other plugins. + +## Related tools + +[fakedev-exporter](#https://github.com/intel/fakedev-exporter) project +can be used to schedule suitably configured fake workloads on the fake +devices, and to provide provide fake activity metrics for them to +Prometheus, that look like they were reported by real Prometheus +metric exporters for real workloads running on real devices. diff --git a/0.26/_sources/cmd/gpu_nfdhook/README.md.txt b/0.26/_sources/cmd/gpu_nfdhook/README.md.txt new file mode 100644 index 000000000..2735f8470 --- /dev/null +++ b/0.26/_sources/cmd/gpu_nfdhook/README.md.txt @@ -0,0 +1,92 @@ +# Intel GPU NFD hook + +Table of Contents + +* [Introduction](#introduction) +* [GPU memory](#gpu-memory) +* [Default labels](#default-labels) +* [PCI-groups (optional)](#pci-groups-optional) +* [Capability labels (optional)](#capability-labels-optional) +* [Limitations](#limitations) + +## Introduction + +This is the [Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery) +binary hook implementation for the Intel GPUs. The intel-gpu-initcontainer (which +is built with the other images) can be used as part of the gpu-plugin deployment +to copy hook to the host systems on which gpu-plugin itself is deployed. + +When NFD worker runs this hook, it will add a number of labels to the nodes, +which can be used for example to deploy services to nodes with specific GPU +types. Selected numeric labels can be turned into kubernetes extended resources +by the NFD, allowing for finer grained resource management for GPU-using PODs. + +In the NFD deployment, the hook requires `/host-sys` -folder to have the host `/sys`-folder content mounted. Write access is not necessary. + +## GPU memory + +GPU memory amount is read from sysfs `gt/gt*` files and turned into a label. +There are two supported environment variables named `GPU_MEMORY_OVERRIDE` and +`GPU_MEMORY_RESERVED`. Both are supposed to hold numeric byte amounts. For systems with +older kernel drivers or GPUs which do not support reading the GPU memory +amount, the `GPU_MEMORY_OVERRIDE` environment variable value is turned into a GPU +memory amount label instead of a read value. `GPU_MEMORY_RESERVED` value will be +scoped out from the GPU memory amount found from sysfs. + +## Default labels + +Following labels are created by default. You may turn numeric labels into extended resources with NFD. + +name | type | description| +-----|------|------| +|`gpu.intel.com/millicores`| number | node GPU count * 1000. Can be used as a finer grained shared execution fraction. +|`gpu.intel.com/memory.max`| number | sum of detected [GPU memory amounts](#gpu-memory) in bytes OR environment variable value * GPU count +|`gpu.intel.com/cards`| string | list of card names separated by '`.`'. The names match host `card*`-folders under `/sys/class/drm/`. Deprecated, use `gpu-numbers`. +|`gpu.intel.com/gpu-numbers`| string | list of numbers separated by '`.`'. The numbers correspond to device file numbers for the primary nodes of given GPUs in kernel DRI subsystem, listed as `/dev/dri/card` in devfs, and `/sys/class/drm/card` in sysfs. +|`gpu.intel.com/tiles`| number | sum of all detected GPU tiles in the system. +|`gpu.intel.com/numa-gpu-map`| string | list of numa node to gpu mappings. + +If the value of the `gpu-numbers` label would not fit into the 63 character length limit, you will also get labels `gpu-numbers2`, +`gpu-numbers3`... until all the gpu numbers have been labeled. + +The tile count `gpu.intel.com/tiles` describes the total amount of tiles on the system. System is expected to be homogeneous, and thus the number of tiles per GPU can be calculated by dividing the tile count with GPU count. + +The `numa-gpu-map` label is a list of numa to gpu mapping items separated by `_`. Each list item has a numa node id combined with a list of gpu indices. e.g. 0-1.2.3 would mean: numa node 0 has gpus 1, 2 and 3. More complex example would be: 0-0.1_1-3.4 where numa node 0 would have gpus 0 and 1, and numa node 1 would have gpus 3 and 4. As with `gpu-numbers`, this label will be extended to multiple labels if the length of the value exceeds the max label length. + +## PCI-groups (optional) + +GPUs which share the same pci paths under `/sys/devices/pci*` can be grouped into a label. GPU nums are separated by '`.`' and +groups are separated by '`_`'. The label is created only if environment variable named `GPU_PCI_GROUPING_LEVEL` has a value greater +than zero. GPUs are considered to belong to the same group, if as many identical folder names are found for the GPUs, as is the value +of the environment variable. Counting starts from the folder name which starts with `pci`. + +For example, the SG1 card has 4 GPUs, which end up sharing pci-folder names under `/sys/devices`. With a `GPU_PCI_GROUPING_LEVEL` +of 3, a node with two such SG1 cards could produce a `pci-groups` label with a value of `0.1.2.3_4.5.6.7`. + +name | type | description| +-----|------|------| +|`gpu.intel.com/pci-groups`| string | list of pci-groups separated by '`_`'. GPU numbers in the groups are separated by '`.`'. The numbers correspond to device file numbers for the primary nodes of given GPUs in kernel DRI subsystem, listed as `/dev/dri/card` in devfs, and `/sys/class/drm/card` in sysfs. + +If the value of the `pci-groups` label would not fit into the 63 character length limit, you will also get labels `pci-groups2`, +`pci-groups3`... until all the pci groups have been labeled. + +## Capability labels (optional) + +Capability labels are created from information found inside debugfs, and therefore +unfortunately require running the NFD worker as root. Due to coming from debugfs, +which is not guaranteed to be stable, these are not guaranteed to be stable either. +If you do not need these, simply do not run NFD worker as root, that is also more secure. +Depending on your kernel driver, running the NFD hook as root may introduce following labels: + +name | type | description| +-----|------|------| +|`gpu.intel.com/platform_gen`| string | GPU platform generation name, typically an integer. Deprecated. +|`gpu.intel.com/media_version`| string | GPU platform Media pipeline generation name, typically a number. Deprecated. +|`gpu.intel.com/graphics_version`| string | GPU platform graphics/compute pipeline generation name, typically a number. Deprecated. +|`gpu.intel.com/platform_.count`| number | GPU count for the named platform. +|`gpu.intel.com/platform_.tiles`| number | GPU tile count in the GPUs of the named platform. +|`gpu.intel.com/platform_.present`| string | "true" for indicating the presense of the GPU platform. + +## Limitations + +For the above to work as intended, GPUs on the same node must be identical in their capabilities. diff --git a/0.26/_sources/cmd/gpu_plugin/README.md.txt b/0.26/_sources/cmd/gpu_plugin/README.md.txt new file mode 100644 index 000000000..13e345560 --- /dev/null +++ b/0.26/_sources/cmd/gpu_plugin/README.md.txt @@ -0,0 +1,375 @@ +# Intel GPU device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Modes and Configuration Options](#modes-and-configuration-options) +* [Installation](#installation) + * [Prerequisites](#prerequisites) + * [Drivers for discrete GPUs](#drivers-for-discrete-gpus) + * [Kernel driver](#kernel-driver) + * [Intel DKMS packages](#intel-dkms-packages) + * [Upstream kernel](#upstream-kernel) + * [User-space drivers](#user-space-drivers) + * [Drivers for older (integrated) GPUs](#drivers-for-older-integrated-gpus) + * [Pre-built Images](#pre-built-images) + * [Install to all nodes](#install-to-all-nodes) + * [Install to nodes with Intel GPUs with NFD](#install-to-nodes-with-intel-gpus-with-nfd) + * [Install to nodes with NFD, Monitoring and Shared-dev](#install-to-nodes-with-nfd-monitoring-and-shared-dev) + * [Install to nodes with Intel GPUs with Fractional resources](#install-to-nodes-with-intel-gpus-with-fractional-resources) + * [Fractional resources details](#fractional-resources-details) + * [Verify Plugin Registration](#verify-plugin-registration) +* [Testing and Demos](#testing-and-demos) +* [Issues with media workloads on multi-GPU setups](#issues-with-media-workloads-on-multi-gpu-setups) + * [Workaround for QSV and VA-API](#workaround-for-qsv-and-va-api) + + +## Introduction + +Intel GPU plugin facilitates Kubernetes workload offloading by providing access to +discrete (including Intel® Data Center GPU Flex Series) and integrated Intel GPU devices +supported by the host kernel. + +Use cases include, but are not limited to: +- Media transcode +- Media analytics +- Cloud gaming +- High performance computing +- AI training and inference + +For example containers with Intel media driver (and components using that), can offload +video transcoding operations, and containers with the Intel OpenCL / oneAPI Level Zero +backend libraries can offload compute operations to GPU. + +## Modes and Configuration Options + +| Flag | Argument | Default | Meaning | +|:---- |:-------- |:------- |:------- | +| -enable-monitoring | - | disabled | Enable 'i915_monitoring' resource that provides access to all Intel GPU devices on the node | +| -resource-manager | - | disabled | Enable fractional resource management, [see also dependencies](#fractional-resources) | +| -shared-dev-num | int | 1 | Number of containers that can share the same GPU device | +| -allocation-policy | string | none | 3 possible values: balanced, packed, none. It is meaningful when shared-dev-num > 1, balanced mode is suitable for workload balance among GPU devices, packed mode is suitable for making full use of each GPU device, none mode is the default. Allocation policy does not have effect when resource manager is enabled. | + +The plugin also accepts a number of other arguments (common to all plugins) related to logging. +Please use the -h option to see the complete list of logging related options. + +## Installation + +The following sections detail how to obtain, build, deploy and test the GPU device plugin. + +Examples are provided showing how to deploy the plugin either using a DaemonSet or by hand on a per-node basis. + +### Prerequisites + +Access to a GPU device requires firmware, kernel and user-space +drivers supporting it. Firmware and kernel driver need to be on the +host, user-space drivers in the GPU workload containers. + +Intel GPU devices supported by the current kernel can be listed with: +``` +$ grep i915 /sys/class/drm/card?/device/uevent +/sys/class/drm/card0/device/uevent:DRIVER=i915 +/sys/class/drm/card1/device/uevent:DRIVER=i915 +``` + +#### Drivers for discrete GPUs + +##### Kernel driver + +###### Intel DKMS packages + +`i915` GPU driver DKMS[^dkms] package is recommended until Intel +discrete GPU support in upstream is complete. It can be installed +from Intel package repositories for a subset of older kernel versions +used in enterprise / LTS distributions: +https://dgpu-docs.intel.com/installation-guides/index.html + +[^dkms]: [intel-gpu-i915-backports](https://github.com/intel-gpu/intel-gpu-i915-backports). + +###### Upstream kernel + +With upstream 6.x kernels, discrete GPU support needs to be enabled using +kernel `i915.force_probe=` command line option until relevant kernel +driver features have been completed also in upstream: +https://www.kernel.org/doc/html/latest/gpu/rfc/index.html + +PCI IDs for the Intel GPUs on given host can be listed with: +``` +$ lspci | grep -e VGA -e Display | grep Intel +88:00.0 Display controller: Intel Corporation Device 56c1 (rev 05) +8d:00.0 Display controller: Intel Corporation Device 56c1 (rev 05) +``` + +(`lspci` lists GPUs with display support as "VGA compatible controller", +and server GPUs without display support, as "Display controller".) + +Mesa "Iris" 3D driver header provides a mapping between GPU PCI IDs and their Intel brand names: +https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/include/pci_ids/iris_pci_ids.h + +If your kernel build does not find the correct firmware version for +a given GPU from the host (see `dmesg | grep i915` output), latest +firmware versions are available in upstream: +https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/i915 + +##### User-space drivers + +Until new enough user-space drivers (supporting also discrete GPUs) +are available directly from distribution package repositories, they +can be installed to containers from Intel package repositories. See: +https://dgpu-docs.intel.com/installation-guides/index.html + +Example container is listed in [Testing and demos](#testing-and-demos). + +Validation status against *upstream* kernel is listed in the user-space drivers release notes: +* Media driver: https://github.com/intel/media-driver/releases +* Compute driver: https://github.com/intel/compute-runtime/releases + +#### Drivers for older (integrated) GPUs + +For the older (integrated) GPUs, new enough firmware and kernel driver +are typically included already with the host OS, and new enough +user-space drivers (for the GPU containers) are in the host OS +repositories. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-gpu-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +Note: Replace `` with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin. + +#### Install to all nodes + +Simplest option to enable use of Intel GPUs in Kubernetes Pods. + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin?ref=' +``` + +#### Install to nodes with Intel GPUs with NFD + +Deploying GPU plugin to only nodes that have Intel GPU attached. [Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery) is required to detect the presence of Intel GPUs. + +```bash +# Start NFD - if your cluster doesn't have NFD installed yet +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=' + +# Create NodeFeatureRules for detecting GPUs on nodes +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=' + +# Create GPU plugin daemonset +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/nfd_labeled_nodes?ref=' +``` + +#### Install to nodes with NFD, Monitoring and Shared-dev + +Same as above, but configures GPU plugin with logging, [monitoring and shared-dev](#modes-and-configuration-options) features enabled. This option is useful when there is a desire to retrieve GPU metrics from nodes. For example with [XPU-Manager](https://github.com/intel/xpumanager/) or [collectd](https://github.com/collectd/collectd/tree/collectd-6.0). + +```bash +# Start NFD - if your cluster doesn't have NFD installed yet +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=' + +# Create NodeFeatureRules for detecting GPUs on nodes +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=' + +# Create GPU plugin daemonset +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/monitoring_shared-dev_nfd/?ref=' +``` + +#### Install to nodes with Intel GPUs with Fractional resources + +With the experimental fractional resource feature you can use additional kubernetes extended +resources, such as GPU memory, which can then be consumed by deployments. PODs will then only +deploy to nodes where there are sufficient amounts of the extended resources for the containers. + +(For this to work properly, all GPUs in a given node should provide equal amount of resources +i.e. heteregenous GPU nodes are not supported.) + +Enabling the fractional resource feature isn't quite as simple as just enabling the related +command line flag. The DaemonSet needs additional RBAC-permissions +and access to the kubelet podresources gRPC service, plus there are other dependencies to +take care of, which are explained below. For the RBAC-permissions, gRPC service access and +the flag enabling, it is recommended to use kustomization by running: + +```bash +# Start NFD with GPU related configuration changes +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/gpu?ref=' + +# Create NodeFeatureRules for detecting GPUs on nodes +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=' + +# Create GPU plugin daemonset +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/fractional_resources?ref=' +``` + +##### Fractional resources details + +Usage of these fractional GPU resources requires that the cluster has node +extended resources with the name prefix `gpu.intel.com/`. Those can be created with NFD +by running the [hook](/cmd/gpu_nfdhook/) installed by the plugin initcontainer. When fractional resources are +enabled, the plugin lets a [scheduler extender](https://github.com/intel/platform-aware-scheduling/tree/master/gpu-aware-scheduling) +do card selection decisions based on resource availability and the amount of extended +resources requested in the [pod spec](https://github.com/intel/platform-aware-scheduling/blob/master/gpu-aware-scheduling/docs/usage.md#pods). + +The scheduler extender then needs to annotate the pod objects with unique +increasing numeric timestamps in the annotation `gas-ts` and container card selections in +`gas-container-cards` annotation. The latter has container separator '`|`' and card separator +'`,`'. Example for a pod with two containers and both containers getting two cards: +`gas-container-cards:card0,card1|card2,card3`. Enabling the fractional-resource support +in the plugin without running such an annotation adding scheduler extender in the cluster +will only slow down GPU-deployments, so do not enable this feature unnecessarily. + +In multi-tile systems, containers can request individual tiles to improve GPU resource usage. +Tiles targeted for containers are specified to pod via `gas-container-tiles` annotation where the the annotation +value describes a set of card and tile combinations. For example in a two container pod, the annotation +could be `gas-container-tiles:card0:gt0+gt1|card1:gt1,card2:gt0`. Similarly to `gas-container-cards`, the container +details are split via `|`. In the example above, the first container gets tiles 0 and 1 from card 0, +and the second container gets tile 1 from card 1 and tile 0 from card 2. + +> **Note**: It is also possible to run the GPU device plugin using a non-root user. To do this, +the nodes' DAC rules must be configured to device plugin socket creation and kubelet registration. +Furthermore, the deployments `securityContext` must be configured with appropriate `runAsUser/runAsGroup`. + +### Verify Plugin Registration + +You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes: + +```bash +$ kubectl get nodes -o=jsonpath="{range .items[*]}{.metadata.name}{'\n'}{' i915: '}{.status.allocatable.gpu\.intel\.com/i915}{'\n'}" +master + i915: 1 +``` + +## Testing and Demos + +The GPU plugin functionality can be verified by deploying an [OpenCL image](../../demo/intel-opencl-icd/) which runs `clinfo` outputting the GPU capabilities (detected by driver installed to the image). + +1. Make the image available to the cluster: + + Build image: + + ```bash + $ make intel-opencl-icd + ``` + + Tag and push the `intel-opencl-icd` image to a repository available in the cluster. Then modify the `intelgpu-job.yaml`'s image location accordingly: + + ```bash + $ docker tag intel/intel-opencl-icd:devel /intel/intel-opencl-icd:latest + $ docker push /intel/intel-opencl-icd:latest + $ $EDITOR ${INTEL_DEVICE_PLUGINS_SRC}/demo/intelgpu-job.yaml + ``` + + If you are running the demo on a single node cluster, and do not have your own registry, you can add image to node image cache instead. For example, to import docker image to containerd cache: + + ```bash + $ IMAGE_NAME=opencl-icd.tar + $ docker save -o $IMAGE_NAME intel/intel-opencl-icd:devel + $ ctr -n=k8s.io images import $IMAGE_NAME + $ rm $IMAGE_NAME + ``` + +1. Create a job: + + ```bash + $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/intelgpu-job.yaml + job.batch/intelgpu-demo-job created + ``` + +1. Review the job's logs: + + ```bash + $ kubectl get pods | fgrep intelgpu + # substitute the 'xxxxx' below for the pod name listed in the above + $ kubectl logs intelgpu-demo-job-xxxxx + + ``` + + If the pod did not successfully launch, possibly because it could not obtain + the requested GPU resource, it will be stuck in the `Pending` status: + + ```bash + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + intelgpu-demo-job-xxxxx 0/1 Pending 0 8s + ``` + + This can be verified by checking the Events of the pod: + + ```bash + $ kubectl describe pod intelgpu-demo-job-xxxxx + ... + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling default-scheduler 0/1 nodes are available: 1 Insufficient gpu.intel.com/i915. + ``` + + +## Issues with media workloads on multi-GPU setups + +Unlike with 3D & compute, and OneVPL media API, QSV (MediaSDK) & VA-API +media APIs do not offer device discovery functionality for applications. +There is nothing (e.g. environment variable) with which the default +device could be overridden either. + +As result, most (all?) media applications using VA-API or QSV, fail to +locate the correct GPU device file unless it is the first ("renderD128") +one, or device file name is explictly specified with an application option. + +Kubernetes device plugins expose only requested number of device +files, and their naming matches host device file names (for several +reasons unrelated to media). Therefore, on multi-GPU hosts, the only +GPU device file mapped to the media container can be some other one +than "renderD128", and media applications using VA-API or QSV need to +be explicitly told which one to use. + +These options differ from application to application. Relevant FFmpeg +options are documented here: +* VA-API: https://trac.ffmpeg.org/wiki/Hardware/VAAPI +* QSV: https://github.com/Intel-Media-SDK/MediaSDK/wiki/FFmpeg-QSV-Multi-GPU-Selection-on-Linux + + +### Workaround for QSV and VA-API + +[Render device](render-device.sh) shell script locates and outputs the +correct device file name. It can be added to the container and used +to give device file name for the application. + +Use it either from another script invoking the application, or +directly from the Pod YAML command line. In latter case, it can be +used either to add the device file name to the end of given command +line, like this: + +```bash +command: ["render-device.sh", "vainfo", "--display", "drm", "--device"] + +=> /usr/bin/vainfo --display drm --device /dev/dri/renderDXXX +``` + +Or inline, like this: + +```bash +command: ["/bin/sh", "-c", + "vainfo --device $(render-device.sh 1) --display drm" + ] +``` + +If device file name is needed for multiple commands, one can use shell variable: + +```bash +command: ["/bin/sh", "-c", + "dev=$(render-device.sh 1) && vainfo --device $dev && " + ] +``` + +With argument N, script outputs name of the Nth suitable GPU device +file, which can be used when more than one GPU resource was requested. diff --git a/0.26/_sources/cmd/iaa_plugin/README.md.txt b/0.26/_sources/cmd/iaa_plugin/README.md.txt new file mode 100644 index 000000000..d2f067a1f --- /dev/null +++ b/0.26/_sources/cmd/iaa_plugin/README.md.txt @@ -0,0 +1,120 @@ +# Intel IAA device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Installation](#installation) + * [Pre-built images](#pre-built-images) + * [Verify plugin registration](#verify-plugin-registration) +* [Testing and Demos](#testing-and-demos) + +## Introduction + +The IAA device plugin for Kubernetes supports acceleration using the Intel Analytics accelerator(IAA). + +The IAA plugin discovers IAA work queues and presents them as a node resources. + +The IAA plugin and operator optionally support provisioning of IAA devices and workqueues with the help of [accel-config](https://github.com/intel/idxd-config) utility through initcontainer. + +## Installation + +The following sections detail how to use the IAA device plugin. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-iaa-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/iaa_plugin?ref=' +daemonset.apps/intel-iaa-plugin created +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +Nothing else is needed. See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin. + +#### Automatic Provisioning + +There's a sample [idxd initcontainer](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/build/docker/intel-idxd-config-initcontainer.Dockerfile) included that provisions IAA devices and workqueues (1 engine / 1 group / 1 wq (user/dedicated)), to deploy: + +```bash +$ kubectl apply -k deployments/iaa_plugin/overlays/iaa_initcontainer/ +``` + +The provisioning [script](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/idxd-init.sh) and [template](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/iaa.conf) are available for customization. + +The provisioning config can be optionally stored in the ProvisioningConfig configMap which is then passed to initcontainer through the volume mount. + +There's also a possibility for a node specific congfiguration through passing a nodename via NODE_NAME into initcontainer's environment and passing a node specific profile via configMap volume mount. + +To create a custom provisioning config: + +```bash +$ kubectl create configmap --namespace=inteldeviceplugins-system intel-iaa-config --from-file=demo/iaa.conf +``` + +### Verify Plugin Registration + +You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes: + +```bash +$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{" "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\( iaa\)' +master + iaa.intel.com/wq-user-dedicated: 2 + iaa.intel.com/wq-user-shared: 10 +node1 + iaa.intel.com/wq-user-dedicated: 4 + iaa.intel.com/wq-user-shared: 30 +``` + +## Testing and Demos + +We can test the plugin is working by deploying the provided example iaa-qpl-demo test image. + +1. Build a Docker image with an accel-config tests: + + ```bash + $ make iaa-qpl-demo + ... + Successfully tagged intel/iaa-qpl-demo:devel + ``` + +1. Create a pod running unit tests off the local Docker image: + + ```bash + $ kubectl apply -f ./demo/iaa-qpl-demo-pod.yaml + pod/iaa-qpl-demo created + ``` + +1. Wait until pod is completed: + + ```bash + $ kubectl get pods |grep iaa-qpl-demo + iaa-qpl-demo 0/1 Completed 0 31m + + If the pod did not successfully launch, possibly because it could not obtain the IAA + resource, it will be stuck in the `Pending` status: + + ```bash + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + iaa-qpl-demo 0/1 Pending 0 7s + ``` + + This can be verified by checking the Events of the pod: + + ```bash + + $ kubectl describe pod iaa-qpl-demo | grep -A3 Events: + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 2m26s default-scheduler 0/1 nodes are available: 1 Insufficient iaa.intel.com/wq-user-dedicated, 1 Insufficient iaa.intel.com/wq-user-shared. + ``` diff --git a/0.26/_sources/cmd/operator/README.md.txt b/0.26/_sources/cmd/operator/README.md.txt new file mode 100644 index 000000000..3650132ff --- /dev/null +++ b/0.26/_sources/cmd/operator/README.md.txt @@ -0,0 +1,175 @@ +# Intel Device Plugins Operator + +Table of Contents + +* [Introduction](#introduction) +* [Installation](#installation) +* [Upgrade](#upgrade) +* [Known issues](#known-issues) + +## Introduction + +Intel Device Plugins Operator is a Kubernetes custom controller whose goal is to serve the +installation and lifecycle management of Intel device plugins for Kubernetes. +It provides a single point of control for GPU, QAT, SGX, FPGA, DSA and DLB devices to a cluster +administrators. + +## Installation + +Install NFD (if it's not already installed) and node labelling rules (requires NFD v0.10+): + +``` +# either with default NFD installation +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=' +# or when setting up with SGX +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/sgx?ref=' +# and finally, NodeFeatureRules +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=' +``` +Make sure both NFD master and worker pods are running: + +``` +$ kubectl get pods -n node-feature-discovery +NAME READY STATUS RESTARTS AGE +nfd-master-599c58dffc-9wql4 1/1 Running 0 25h +nfd-worker-qqq4h 1/1 Running 0 25h +``` + +Note that labelling is not performed immediately. Give NFD 1 minute to pick up the rules and label nodes. + +As a result all found devices should have correspondent labels, e.g. for Intel DLB devices the label is +intel.feature.node.kubernetes.io/dlb: +``` +$ kubectl get no -o json | jq .items[].metadata.labels |grep intel.feature.node.kubernetes.io/dlb + "intel.feature.node.kubernetes.io/dlb": "true", +``` + +Full list of labels can be found in the deployments/operator/samples directory: +``` +$ grep -r feature.node.kubernetes.io/ deployments/operator/samples/ +deployments/operator/samples/deviceplugin_v1_dlbdeviceplugin.yaml: intel.feature.node.kubernetes.io/dlb: 'true' +deployments/operator/samples/deviceplugin_v1_qatdeviceplugin.yaml: intel.feature.node.kubernetes.io/qat: 'true' +deployments/operator/samples/deviceplugin_v1_sgxdeviceplugin.yaml: intel.feature.node.kubernetes.io/sgx: 'true' +deployments/operator/samples/deviceplugin_v1_gpudeviceplugin.yaml: intel.feature.node.kubernetes.io/gpu: "true" +deployments/operator/samples/deviceplugin_v1_fpgadeviceplugin.yaml: intel.feature.node.kubernetes.io/fpga-arria10: 'true' +deployments/operator/samples/deviceplugin_v1_dsadeviceplugin.yaml: intel.feature.node.kubernetes.io/dsa: 'true' +``` + +The default operator deployment depends on [cert-manager](https://cert-manager.io/) running in the cluster. +See installation instructions [here](https://cert-manager.io/docs/installation/kubectl/). + +Make sure all the pods in the `cert-manager` namespace are up and running: + +``` +$ kubectl get pods -n cert-manager +NAME READY STATUS RESTARTS AGE +cert-manager-7747db9d88-bd2nl 1/1 Running 0 21d +cert-manager-cainjector-87c85c6ff-59sb5 1/1 Running 0 21d +cert-manager-webhook-64dc9fff44-29cfc 1/1 Running 0 21d +``` + +Also if your cluster operates behind a corporate proxy make sure that the API +server is configured not to send requests to cluster services through the +proxy. You can check that with the following command: + +```bash +$ kubectl describe pod kube-apiserver --namespace kube-system | grep -i no_proxy | grep "\.svc" +``` + +In case there's no output and your cluster was deployed with `kubeadm` open +`/etc/kubernetes/manifests/kube-apiserver.yaml` at the control plane nodes and +append `.svc` and `.svc.cluster.local` to the `no_proxy` environment variable: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + ... +spec: + containers: + - command: + - kube-apiserver + - --advertise-address=10.237.71.99 + ... + env: + - name: http_proxy + value: http://proxy.host:8080 + - name: https_proxy + value: http://proxy.host:8433 + - name: no_proxy + value: 127.0.0.1,localhost,.example.com,10.0.0.0/8,.svc,.svc.cluster.local + ... +``` + +**Note:** To build clusters using `kubeadm` with the right `no_proxy` settings from the very beginning, +set the cluster service names to `$no_proxy` before `kubeadm init`: + +``` +$ export no_proxy=$no_proxy,.svc,.svc.cluster.local +``` + +Finally deploy the operator itself: + +``` +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/operator/default?ref=' +``` + +Now you can deploy the device plugins by creating corresponding custom resources. +The samples for them are available [here](/deployments/operator/samples/). + +## Usage + +Deploy your device plugin by applying its custom resource, e.g. +`GpuDevicePlugin` with + +```bash +$ kubectl apply -f https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/main/deployments/operator/samples/deviceplugin_v1_gpudeviceplugin.yaml +``` + +Observe it is up and running: + +```bash +$ kubectl get GpuDevicePlugin +NAME DESIRED READY NODE SELECTOR AGE +gpudeviceplugin-sample 1 1 5s +``` + +In order to limit the deployment to a specific device type, +use one of kustomizations under deployments/operator/device. + +For example, to limit the deployment to FPGA, use: + +```bash +$ kubectl apply -k deployments/operator/device/fpga +``` + +Operator also supports deployments with multiple selected device types. +In this case, create a new kustomization with the necessary resources +that passes the desired device types to the operator using `--device` +command line argument multiple times. + +## Upgrade + +The upgrade of the deployed plugins can be done by simply installing a new release of the operator. + +The operator auto-upgrades operator-managed plugins (CR images and thus corresponding deployed daemonsets) to the current release of the operator. + +The [registry-url]/[namespace]/[image] are kept intact on the upgrade. + +No upgrade is done for: + +- Non-operator managed deployments +- Operator deployments without numeric tags + +## Known issues + +When the operator is run with leader election enabled, that is with the option +`--leader-elect`, make sure the cluster is not overloaded with excessive +number of pods. Otherwise a heart beat used by the leader election code may trigger +a timeout and crash. We are going to use different clients for the controller and +leader election code to alleviate the issue. See more details in +https://github.com/intel/intel-device-plugins-for-kubernetes/issues/476. + +In case the deployment is limited to specific device type(s), +the CRDs for other device types are still created, but no controllers +for them are registered. diff --git a/0.26/_sources/cmd/operator/ocp_quickstart_guide/README.md.txt b/0.26/_sources/cmd/operator/ocp_quickstart_guide/README.md.txt new file mode 100644 index 000000000..6c6bf0ebd --- /dev/null +++ b/0.26/_sources/cmd/operator/ocp_quickstart_guide/README.md.txt @@ -0,0 +1,66 @@ +# Intel® Device Plugins Operator for Red Hat OpenShift Container Platform + +## Table of Contents +* [Introduction](#introduction) +* [Minimum Hardware Requirements](#minimum-hardware-requirements) + * [Intel SGX Enabled Server](#intel-sgx-enabled-server) +* [Installation](#installation) + * [Prerequisites](#prerequisites) + * [Install Operator using OpenShift Web Console](#install-operator-using-openshift-web-console) + * [Verify Operator installation](#verify-operator-installation) +* [Deploying Intel Device Plugins](#deploying-intel-device-plugins) + * [Intel SGX Device Plugin](#intel-sgx-device-plugin) + +## Introduction +The Intel Device Plugins Operator for OpenShift Container Platform is a collection of device plugins advertising Intel specific hardware resources to the kubelet. It provides a single point of control for Intel® Software Guard Extensions (Intel® SGX), Intel GPUs, Intel® QuickAccess Technology (Intel® QAT), Intel® Data Streaming Accelerator (Intel® DSA), and Intel® In-Memory Analytics Accelerator (Intel® IAA) devices to cluster administrators. The [`v0.24.0`](https://github.com/intel/intel-device-plugins-for-kubernetes/releases/tag/v0.24.0) release of the operator only supports Intel SGX and Intel QAT device plugins. GPU, Intel DSA, Intel IAA, and other device plugins will be supported in future releases. + +## Minimum Hardware Requirements +### Intel SGX Enabled Server +- Third Generation Intel® Xeon® Scalable Platform, code-named “Ice Lake” or later +- Configure BIOS using below details + ![SGX Server BIOS](images/SGX-BIOS.PNG) + [**Note:** The BIOS configuration shown above is just for the reference. Please contact your BIOS vendor for details] + +## Installation +### Prerequisites +- Make sure Red Hat OpenShift Cluster is ready to use and the developing machine is RHEL and `oc` command is installed and configured properly. Please note that the following operation is verified on Red Hat OpenShift Cluster 4.11 and working machine RHEL-8.6 +- Install the `oc` command to your development machine +- Follow the [link](https://docs.openshift.com/container-platform/4.11/hardware_enablement/psap-node-feature-discovery-operator.html) to install **NFD operator** (if it's not already installed). + **Note:** Please only install the NFD operator and use steps below to create the NodeFeatureDiscovery instance. + - Create the NodeFeatureDiscovery instance + ``` + $ oc apply -f https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/v0.24.0/deployments/nfd/overlays/node-feature-discovery/node-feature-discovery-openshift.yaml + ``` + - Create the NodeFeatureRule instance + ``` + $ oc apply -f https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/v0.24.0/deployments/nfd/overlays/node-feature-rules/node-feature-rules-openshift.yaml + ``` +- Deploy SELinux Policy for OCP 4.10 - + The SGX device plugin and Init container run as a label `container_device_plugin_t` and `container_device_plugin_init_t` respectively. This requires a custom SELinux policy to be deployed before the SGX plugin can be run. To deploy this policy, run + ``` + $ oc apply -f https://raw.githubusercontent.com/intel/user-container-selinux/main/policy-deployment.yaml + ``` + +### Install Operator using OpenShift Web Console +1. In OpenShift web console navigate to **Operator** -> **OperatorHub** +2. Search for **Intel Device Plugins Operator ->** Click **Install** + + +### Verify Operator installation +1. Go to **Operator** -> **Installed Operators** +2. Verify the status of operator as **Succeeded** +3. Click **Intel Device Plugins Operator** to view the details + ![Verify Operator](images/verify-operator.PNG) + + +## Deploying Intel Device Plugins + +### Intel SGX Device Plugin +Follow the steps below to deploy Intel SGX Device Plugin Custom Resource +1. Go to **Operator** -> **Installed Operators** +2. Open **Intel Device Plugins Operator** +3. Navigate to tab **Intel Software Guard Extensions Device Plugin** +4. Click **Create SgxDevicePlugin ->** set correct parameters -> Click **Create** + OR for any customizations, please select `YAML view` and edit details. Once done, click **Create** +5. Verify CR by checking the status of DaemonSet **`intel-sgx-plugin`** +6. Now `SgxDevicePlugin` is ready to deploy any workloads diff --git a/0.26/_sources/cmd/qat_plugin/README.md.txt b/0.26/_sources/cmd/qat_plugin/README.md.txt new file mode 100644 index 000000000..b8bc5688e --- /dev/null +++ b/0.26/_sources/cmd/qat_plugin/README.md.txt @@ -0,0 +1,253 @@ +# Intel QuickAssist Technology (QAT) device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Modes and Configuration Options](#modes-and-configuration-options) +* [Installation](#installation) + * [Prerequisites](#prerequisites) + * [Pre-built Images](#pre-built-images) + * [Verify Plugin Registration](#verify-plugin-registration) +* [Demos and Testing](#demos-and-testing) + * [DPDK QAT Demos](#dpdk-qat-demos) + * [DPDK Prerequisites](#dpdk-prerequisites) + * [Deploy the pod](#deploy-the-pod) + * [Manual test run](#manual-test-run) + * [Automated test run](#automated-test-run) + * [OpenSSL QAT Demo](#openssl-qat-demo) +* [Checking for Hardware](#checking-for-hardware) + +## Introduction + +This Intel QAT device plugin provides support for Intel QAT devices under Kubernetes. +The supported devices are determined by the VF device drivers available in your Linux +Kernel. See the [Prerequisites](#prerequisites) section for more details. + +Supported Devices include, but may not be limited to, the following: + +- [Intel® Xeon® with Intel® C62X Series Chipset][1] +- Intel® Xeon® with Intel® QAT Gen4 devices +- [Intel® Atom™ Processor C3000][2] +- [Intel® Communications Chipset 8925 to 8955 Series][3] + +The QAT device plugin provides access to QAT hardware accelerated cryptographic and compression features. +Demonstrations are provided utilising [DPDK](https://doc.dpdk.org/) and [OpenSSL](https://www.openssl.org/). + +[Kata Containers](https://katacontainers.io/) QAT integration is documented in the +[Kata Containers documentation repository][6]. + +## Modes and Configuration Options + +The QAT plugin can take a number of command line arguments, summarised in the following table: + +| Flag | Argument | Meaning | +|:---- |:-------- |:------- | +| -dpdk-driver | string | DPDK Device driver for configuring the QAT device (default: `vfio-pci`) | +| -kernel-vf-drivers | string | Comma separated VF Device Driver of the QuickAssist Devices in the system. Devices supported: DH895xCC, C62x, C3xxx, 4xxx/401xx, C4xxx and D15xx (default: `c6xxvf,4xxxvf`) | +| -max-num-devices | int | maximum number of QAT devices to be provided to the QuickAssist device plugin (default: `32`) | +| -mode | string | plugin mode which can be either `dpdk` or `kernel` (default: `dpdk`) | +| -allocation-policy | string | 2 possible values: balanced and packed. Balanced mode spreads allocated QAT VF resources balanced among QAT PF devices, and packed mode packs one QAT PF device full of QAT VF resources before allocating resources from the next QAT PF. (There is no default.) | + +The plugin also accepts a number of other arguments related to logging. Please use the `-h` option to see +the complete list of logging related options. + +For more details on the `-dpdk-driver` choice, see +[DPDK Linux Driver Guide](http://dpdk.org/doc/guides/linux_gsg/linux_drivers.html). + +> **Note:**: With Linux 5.9+ kernels the `vfio-pci` module must be loaded with +> `disable_denylist=1` parameter for the QAT device plugin to work correctly with +> devices prior to Gen4 (`4xxx`). + +For more details on the available options to the `-kernel-vf-drivers` option, see the list of +vf drivers available in the [Linux Kernel](https://github.com/torvalds/linux/tree/master/drivers/crypto/qat). + +If the `-mode` parameter is set to `kernel`, no other parameter documented above are valid, +except the `klog` logging related parameters. +`kernel` mode implements resource allocation based on system configured [logical instances][7]. + +> **Note**: `kernel` mode is excluded by default from all builds (including those hosted on the Docker hub), +> by default. See the [Build the plugin image](#build-the-plugin-image) section for more details. + +The `kernel` mode does not guarantee full device isolation between containers +and therefore it's not recommended. This mode will be deprecated and removed once `libqat` +implements non-UIO based device access. + +## Installation + +The below sections cover how to obtain, build and install this component. + +The component can be installed either using a DaemonSet or running 'by hand' on each node. + +### Prerequisites + +The component has the same basic dependancies as the +[generic plugin framework dependencies](../../README.md#about). + +You will also need [appropriate hardware installed](#checking-for-hardware). + +The QAT plugin requires Linux Kernel VF QAT drivers to be available. These drivers +are available via two methods. One of them must be installed and enabled: + +- [Linux Kernel upstream drivers](https://github.com/torvalds/linux/tree/master/drivers/crypto/qat) +- [Intel QuickAssist Technology software for Linux][9] + +The demonstrations have their own requirements, listed in their own specific sections. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-qat-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_plugin?ref=' +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +An alternative kustomization for deploying the plugin is with the debug mode switched on: + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_plugin/overlays/debug?ref=' +``` + +> **Note**: It is also possible to run the QAT device plugin using a non-root user. To do this, +> the nodes' DAC rules must be configured to allow PCI driver unbinding/binding, device plugin +> socket creation and kubelet registration. Furthermore, the deployments `securityContext` must +> be configured with appropriate `runAsUser/runAsGroup`. + +#### Automatic Provisioning + +There's a sample [qat initcontainer](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/build/docker/intel-qat-initcontainer.Dockerfile). Regardless of device types, the script running inside the initcontainer enables QAT SR-IOV VFs. + +To deploy, run as follows: + +```bash +$ kubectl apply -k deployments/qat_plugin/overlays/qat_initcontainer/ +``` + +In addition to the default configuration, you can add device-specific configurations via ConfigMap. + +| Device | Possible Configuration | How To Customize | Options | Notes | +|:-------|:-----------------------|:-----------------|:--------|:------| +| 4xxx, 401xx | [cfg_services](https://github.com/torvalds/linux/blob/42e66b1cc3a070671001f8a1e933a80818a192bf/Documentation/ABI/testing/sysfs-driver-qat) reports the configured services (crypto services or compression services) of the QAT device. | `ServicesEnabled=` | compress:`dc`, crypto:`sym;asym` | Linux 6.0+ kernel is required. | + +To create a provisioning config after customizing, run as follows: + +```bash +$ kubectl create configmap --namespace=inteldeviceplugins-system qat-config --from-file=deployments/qat_plugin/overlays/qat_initcontainer/qat.conf +``` +> **Note**: When deploying the overlay qat_initcontainer, such a manual creation is not necessary since ConfigMap is generated automatically. Just set the values in the config file and deploy the overlay. + +When using the operator for deploying the plugin with provisioning config, use `provisioningConfig` field for the name of the ConfigMap, then the config is passed to initcontainer through the volume mount. + +There's also a possibility for a node specific congfiguration through passing a nodename via `NODE_NAME` into initcontainer's environment and passing a node specific profile (`qat-$NODE_NAME.conf`) via ConfigMap volume mount. + + +#### Verify Plugin Registration + +Verification of the plugin deployment and detection of QAT hardware can be confirmed by +examining the resource allocations on the nodes: + +```bash +$ kubectl describe node | grep qat.intel.com/generic + qat.intel.com/generic: 10 + qat.intel.com/generic: 10 +``` + +## Demos and Testing + +The below sections cover `DPDK` and `OpenSSL` demos, both of which utilise the +QAT device plugin under Kubernetes. + +### DPDK QAT demos + +The Data Plane Development Kit (DPDK) QAT demos use DPDK +[crypto-perf](https://doc.dpdk.org/guides/tools/cryptoperf.html) and +[compress-perf](https://doc.dpdk.org/guides/tools/comp_perf.html) utilities to exercise +DPDK QAT Poll-Mode Drivers (PMD). For more information on the tools' parameters, refer to the +website links. + +#### DPDK Prerequisites + +For the DPDK QAT demos to work, the DPDK drivers must be loaded and configured. +For more information, refer to: +[DPDK Getting Started Guide for Linux](https://doc.dpdk.org/guides/linux_gsg/index.html) and +[DPDK Getting Started Guide, Linux Drivers section](http://dpdk.org/doc/guides/linux_gsg/linux_drivers.html) + +#### Deploy the pod + +In the pod specification file, add container resource request and limit. +For example, `qat.intel.com/generic: ` for a container requesting QAT devices. + +For a DPDK-based workload, you may need to add hugepage request and limit. + +```bash +$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_dpdk_app/base/ +$ kubectl get pods + NAME READY STATUS RESTARTS AGE + qat-dpdk 1/1 Running 0 27m + intel-qat-plugin-5zgvb 1/1 Running 0 3h + +``` + +> **Note**: If the `igb_uio` VF driver is used with the QAT device plugin, +> the workload be deployed with `SYS_ADMIN` capabilities added. + +#### Manual Test Run + +Manually execute the `dpdk-test-crypto-perf` application to review the logs: + +```bash +$ kubectl exec -it qat-dpdk bash + +$ dpdk-test-crypto-perf -l 6-7 -w $QAT1 \ +-d /usr/lib64/librte_mempool_ring.so.1.1 \ +-d /usr/lib64/librte_pmd_qat.so.1.1 -- \ +--ptest throughput --devtype crypto_qat \ +--optype cipher-only --cipher-algo aes-cbc --cipher-op encrypt \ +--cipher-key-sz 16 --total-ops 10000000 --burst-sz 32 --buffer-sz 64 +``` + +> **Note**: Adapt the `.so` versions to what the DPDK version in the container provides. + +#### Automated Test Run + +It is also possible to deploy and run `crypto-perf` using the following +`kustomize` overlays: + +```bash +$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_dpdk_app/test-crypto1 +$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_dpdk_app/test-compress1 +$ kubectl logs qat-dpdk-test-crypto-perf-tc1 +$ kubectl logs qat-dpdk-test-compress-perf-tc1 +``` + +> **Note**: for `test-crypto1` and `test-compress1` to work, the cluster must enable +[Kubernetes CPU manager's](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/) `static` policy. + +### OpenSSL QAT Demo + +Please refer to the [Kata Containers documentation][8] for details on the OpenSSL +QAT acceleration demo. + +## Checking for Hardware + +In order to utilise the QAT device plugin, QuickAssist SR-IOV virtual functions must be configured. +You can verify this on your nodes by checking for the relevant PCI identifiers: + +```bash +for i in 0442 0443 18a1 37c9 6f55 19e3 4941 4943; do lspci -d 8086:$i; done +``` + +[1]:https://www-ssl.intel.com/content/www/us/en/design/products-and-solutions/processors-and-chipsets/purley/intel-xeon-scalable-processors.html +[2]:https://www.intel.com/content/www/us/en/design/products-and-solutions/processors-and-chipsets/denverton/ns/atom-processor-c3000-series.html +[3]:https://www.intel.com/content/www/us/en/ethernet-products/gigabit-server-adapters/quickassist-adapter-8950-brief.html +[6]:https://github.com/kata-containers/documentation/blob/master/use-cases/using-Intel-QAT-and-kata.md +[7]:https://www.intel.com/content/www/us/en/content-details/710060/intel-quickassist-technology-software-for-linux-programmer-s-guide-hw-version-1-7.html +[8]:https://github.com/kata-containers/documentation/blob/master/use-cases/using-Intel-QAT-and-kata.md#build-openssl-intel-qat-engine-container +[9]:https://www.intel.com/content/www/us/en/content-details/710059/intel-quickassist-technology-software-for-linux-getting-started-guide-hw-version-1-7.html diff --git a/0.26/_sources/cmd/sgx_admissionwebhook/README.md.txt b/0.26/_sources/cmd/sgx_admissionwebhook/README.md.txt new file mode 100644 index 000000000..094fa5661 --- /dev/null +++ b/0.26/_sources/cmd/sgx_admissionwebhook/README.md.txt @@ -0,0 +1,81 @@ +# Intel SGX admission controller for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Installation](#installation) + * [Pre-requisites](#pre-requisites) + * [Deployment](#deployment) + +## Introduction + +The SGX admission controller is one of the components used to add support for Intel SGX +functionality to Kubernetes. + +> **NOTE:** Installation of the SGX admission controller can be skipped if the +> [SGX device plugin](../sgx_plugin/README.md) is operated with the Intel Device Plugins Operator +> since it integrates the controller's functionality. + +The SGX admission webhook is responsible for performing Pod mutations based on the `sgx.intel.com/quote-provider` +pod annotation set by the user. The purpose of the webhook is to hide the details of setting the necessary +device resources and volume mounts for using SGX remote attestation in the cluster. Furthermore, +the SGX admission webhook is responsible for writing a pod/sandbox `sgx.intel.com/epc` annotation that is used by +Kata Containers to dynamically adjust its virtualized SGX encrypted page cache (EPC) bank(s) size. + +## Installation + +The following sections detail how to obtain, build and deploy the admission +controller webhook plugin. + +### Pre-requisites + +The default webhook deployment depends on having [cert-manager](https://cert-manager.io/) +installed. See its installation instructions [here](https://cert-manager.io/docs/installation/kubectl/). + +Also if your cluster operates behind a corporate proxy make sure that the API +server is configured not to send requests to cluster services through the +proxy. You can check that with the following command: + +```bash +$ kubectl describe pod kube-apiserver --namespace kube-system | grep -i no_proxy | grep "\.svc" +``` + +In case there's no output and your cluster was deployed with `kubeadm` open +`/etc/kubernetes/manifests/kube-apiserver.yaml` at the control plane nodes and +append `.svc` and `.svc.cluster.local` to the `no_proxy` environment variable: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + ... +spec: + containers: + - command: + - kube-apiserver + - --advertise-address=10.237.71.99 + ... + env: + - name: http_proxy + value: http://proxy.host:8080 + - name: https_proxy + value: http://proxy.host:8433 + - name: no_proxy + value: 127.0.0.1,localhost,.example.com,10.0.0.0/8,.svc,.svc.cluster.local + ... +``` + +**Note:** To build clusters using `kubeadm` with the right `no_proxy` settings from the very beginning, +set the cluster service names to `$no_proxy` before `kubeadm init`: + +``` +$ export no_proxy=$no_proxy,.svc,.svc.cluster.local +``` + +### Deployment + +To deploy the webhook with cert-manager, run + +```bash +$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_admissionwebhook/overlays/default-with-certmanager?ref=main +``` diff --git a/0.26/_sources/cmd/sgx_plugin/README.md.txt b/0.26/_sources/cmd/sgx_plugin/README.md.txt new file mode 100644 index 000000000..7632c7180 --- /dev/null +++ b/0.26/_sources/cmd/sgx_plugin/README.md.txt @@ -0,0 +1,237 @@ +# Intel Software Guard Extensions (SGX) device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Modes and Configuration Options](#modes-and-configuration-options) +* [Installation](#installation) + * [Prerequisites](#prerequisites) + * [Pre-built Images](#pre-built-images) + * [Verify Plugin Registration](#verify-plugin-registration) +* [Testing and Demos](#testing-and-demos) + * [SGX ECDSA Remote Attestation](#sgx-ecdsa-remote-attestation) + * [Remote Attestation Prerequisites](#remote-attestation-prerequisites) + * [Build the images](#build-the-image) + * [Deploy the pod](#deploy-the-pod) + +## Introduction + +The Intel SGX device plugin and related components allow workloads to use Intel SGX on +platforms with SGX Flexible Launch Control enabled, e.g.,: + +- 3rd Generation Intel® Xeon® Scalable Platform, code-named “Ice Lake” +- Intel® Xeon® E3 +- Intel® NUC Kit NUC7CJYH + +The SGX solution comes in three parts: + +- the [SGX Device plugin](/README.md#sgx-device-plugin) +- the [SGX Admission webhook](/README.md#sgx-admission-webhook) +- the [SGX EPC memory registration](/README.md#sgx-epc-memory-registration) + +This README covers setting up all three components. + +### Modes and Configuration options + +The SGX plugin can take a number of command line arguments, summarised in the following table: + +| Flag | Argument | Meaning | +|:---- |:-------- |:------- | +| -enclave-limit | int | the number of containers per worker node allowed to use `/dev/sgx_enclave` device node (default: `20`) | +| -provision-limit | int | the number of containers per worker node allowed to use `/dev/sgx_provision` device node (default: `20`) | + +The plugin also accepts a number of other arguments related to logging. Please use the `-h` option to see +the complete list of logging related options. + +## Installation + +The following sections cover how to use the necessary Kubernetes SGX specific +components. + +### Prerequisites + +The component has the same basic dependencies as the +[generic plugin framework dependencies](../../README.md#about). + +The SGX device plugin requires Linux Kernel SGX drivers to be available. These drivers +are available in Linux since 5.11. [The SGX DCAP out-of-tree driver](https://github.com/intel/SGXDataCenterAttestationPrimitives) +(v1.41 and later) is also known to work. + +The hardware platform must support SGX Flexible Launch Control. + +The SGX deployment depends on having [cert-manager](https://cert-manager.io/) +installed. See its installation instructions [here](https://cert-manager.io/docs/installation/kubectl/). + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/u/intel/) +are available on Docker Hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on Docker Hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy Intel SGX components in your cluster is to follow the steps +below. + +The deployment YAML files supplied with the components in this repository use the images with the `devel` +tag by default. If you do not build your own local images, your Kubernetes cluster may pull down +the devel images from Docker Hub by default. + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +### Installation Using the Operator + +First, deploy `node-feature-discovery`: + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/sgx?ref=' +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=' +``` + +**Note:** The [default configuration](/deployments/nfd/overlays/node-feature-rules/node-feature-rules.yaml) assumes that the in-tree driver is used and enabled (`CONFIG_X86_SGX=y`). If +the SGX DCAP out-of-tree driver is used, the `kernel.config` match expression must be removed. + +Next, deploy the Intel Device plugin operator: + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/operator/default?ref=' +``` + +**Note:** See the operator [deployment details](/cmd/operator/README.md) for its dependencies and for setting it up on systems behind proxies. + +Finally, deploy the SGX device plugin with the operator + +```bash +$ kubectl apply -f 'https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes//deployments/operator/samples/deviceplugin_v1_sgxdeviceplugin.yaml' +``` + +### Installation Using kubectl + +There are two alternative ways to deploy SGX device plugin using `kubectl`. + +The first approach involves deployment of the [SGX DaemonSet YAML](/deployments/sgx_plugin/base/intel-sgx-plugin.yaml) +and [node-feature-discovery](/deployments/nfd/overlays/sgx/kustomization.yaml) +with the necessary configuration. + +There is a kustomization for deploying everything: +```bash +$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_plugin/overlays/epc-nfd/ +``` + +The second approach has a lesser deployment footprint. It does not require NFD, but a helper daemonset that creates `sgx.intel.com/capable='true'` node label and advertises EPC capacity to the API server. + +The following kustomization is used for this approach: +```bash +$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_plugin/overlays/epc-register/ +``` + +Additionally, SGX admission webhook must be deployed +```bash +$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_admissionwebhook/ +``` + +### Verify Plugin Registration + +Verification of the plugin deployment and detection of SGX hardware can be confirmed by +examining the resource allocations on the nodes: + +```bash +$ kubectl describe node | grep sgx.intel.com + nfd.node.kubernetes.io/extended-resources: sgx.intel.com/epc + sgx.intel.com/enclave: 20 + sgx.intel.com/epc: 98566144 + sgx.intel.com/provision: 20 + sgx.intel.com/enclave: 20 + sgx.intel.com/epc: 98566144 + sgx.intel.com/provision: 20 + sgx.intel.com/enclave 1 1 + sgx.intel.com/epc 400 400 + sgx.intel.com/provision 1 1 +``` + +## Testing and Demos +### SGX ECDSA Remote Attestation + +The SGX remote attestation allows a relying party to verify that the software is running inside an Intel® SGX enclave on a platform +that has the trusted computing base up to date. + +The demo guides to run an SGX DCAP/ECDSA quote generation in on a single-node kubernetes cluster using Intel® reference +SGX PCK Certificate Cache Service (PCCS) that is configured to service localhost connections. + +Read more about [SGX Remote Attestation](https://software.intel.com/content/www/us/en/develop/topics/software-guard-extensions/attestation-services.html). + +#### Remote Attestation Prerequisites + +For the SGX ECDSA Remote Attestation demo to work, the platform must be correctly registered and a PCCS running. + +For documentation to set up Intel® reference PCCS, refer to: +[Intel® Software Guard Extensions (Intel® SGX) Services](https://api.portal.trustedservices.intel.com/) and +[Intel® Software Guard Extensions SDK for Linux](https://www.intel.com/content/www/us/en/developer/tools/software-guard-extensions/overview.html) + +Furthermore, the Kubernetes cluster must be set up according the [instructions above](#deploying-with-pre-built-images). + +#### Build the image + +The demo uses container images build from Intel® SGX SDK and DCAP releases. + +To build the demo images: + +```bash +$ cd ${INTEL_DEVICE_PLUGINS_SRC} +$ make sgx-aesmd-demo +... +Successfully tagged intel/sgx-aesmd-demo:devel +$ make sgx-sdk-demo +... +Successfully tagged intel/sgx-sdk-demo:devel +``` + +#### Deploy the pods + +The demo runs Intel aesmd (architectural enclaves service daemon) that is responsible +for generating SGX quotes for workloads. It is deployed with `hostNetwork: true` +to allow connections to localhost PCCS. + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_aesmd?ref=' +$ kubectl get pods + NAME READY STATUS RESTARTS AGE + intel-sgx-aesmd-mrnm8 1/1 Running 0 3h47m + sgxdeviceplugin-sample-z5dcq-llwlw 1/1 Running 0 28m +``` +> **Note**: For quick experiments, [kind](https://kind.sigs.k8s.io/docs/user/configuration/) can be used to deploy the cluster. With `kind`, host path `/var/run/aesmd/` must be mounted to the nodes manually using [Extra Mounts](https://kind.sigs.k8s.io/docs/user/configuration/#extra-mounts`). \ +> Example: +>``` +>$ cat kind_config.yaml +>kind: Cluster +>apiVersion: kind.x-k8s.io/v1alpha4 +>name: +>nodes: +>- role: control-plane +> extraMounts: +> - hostPath: /var/run/aesmd +> containerPath: /var/run/aesmd +> propagation: Bidirectional +>``` +> And bootstrap kind with it \ +> `$ kind create cluster --config kind_config.yaml` + +The sample application runs SGX DCAP Quote Generation sample: + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_enclave_apps/overlays/sgx_ecdsa_aesmd_quote?ref=' +$ kubectl get pods + NAME READY STATUS RESTARTS AGE + intel-sgx-aesmd-mrnm8 1/1 Running 0 3h55m + ecdsa-quote-intelsgx-demo-job-vtq84 0/1 Completed 0 4s + sgxdeviceplugin-sample-z5dcq-llwlw 1/1 Running 0 35m +$ kubectl logs ecdsa-quote-intelsgx-demo-job-vtq84 + + Step1: Call sgx_qe_get_target_info:succeed! + Step2: Call create_app_report:succeed! + Step3: Call sgx_qe_get_quote_size:succeed! + Step4: Call sgx_qe_get_quote:succeed!cert_key_type = 0x5 +``` + +> **Note**: The deployment example above uses [kustomize](https://github.com/kubernetes-sigs/kustomize) +> that is available in kubectl since Kubernetes v1.14 release. diff --git a/0.26/_sources/cmd/vpu_plugin/README.md.txt b/0.26/_sources/cmd/vpu_plugin/README.md.txt new file mode 100644 index 000000000..c27c490cb --- /dev/null +++ b/0.26/_sources/cmd/vpu_plugin/README.md.txt @@ -0,0 +1,199 @@ +# Intel VPU device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Installation](#installation) + * [Pre-built Images](#pre-built-images) + * [Verify Plugin Registration](#verify-plugin-registration) +* [Testing and Demos](#testing-and-demos) + +## Introduction + +The VPU device plugin supports below cards: + +[Intel VCAC-A](https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/media-analytics-vcac-a-accelerator-card-by-celestica-datasheet.pdf). +This card has: +- 1 Intel Core i3-7100U processor +- 12 MyriadX VPUs +- 8GB DDR4 memory +- PCIe interface to Xeon E3/E5 server + +[Intel Mustang V100](https://software.intel.com/en-us/articles/introducing-the-iei-tank-aiot-developer-kit-and-mustang-v100-mx8-pcie-accelerator-card). +This card has: +- 8 MyriadX VPUs +- PCIe interface to 6th+ Generation Core PC or Xeon E3/E5 server + +[Gen 3 Intel® Movidius™ VPU HDDL VE3](https://www.intel.com/content/www/us/en/products/details/processors/movidius-vpu.html) +This card has: +- 3 Intel® Movidius Gen 3 Intel® Movidius™ VPU SoCs + +[Intel® Movidius™ S VPU](https://www.intel.com/content/www/us/en/products/details/processors/movidius-vpu.html) +This card has: +- 6 Intel® Movidius Gen 3 Intel® Movidius™ VPU SoCs + +> **Note:** This device plugin need HDDL daemon service to be running either natively or from a container. +> To get VCAC-A or Mustang card running hddl, please refer to: +> https://github.com/OpenVisualCloud/Dockerfiles/blob/master/VCAC-A/script/setup_hddl.sh + +## Installation + +The following sections detail how to use the VPU device plugin. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-vpu-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/vpu_plugin?ref=' +daemonset.apps/intel-vpu-plugin created +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +For xlink device, deploy DaemonSet as +```bash +$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/vpu_plugin/overlays/xlink +daemonset.apps/intel-vpu-plugin created +``` + +Nothing else is needed. See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin. + +> **Note**: It is also possible to run the VPU device plugin using a non-root user. To do this, +the nodes' DAC rules must be configured to device plugin socket creation and kubelet registration. +Furthermore, the deployments `securityContext` must be configured with appropriate `runAsUser/runAsGroup`. + +### Verify Plugin Registration + +You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes: + +```bash +$ kubectl get nodes -o=jsonpath="{range .items[*]}{.metadata.name}{'\n'}{' hddl: '}{.status.allocatable.vpu\.intel\.com/hddl}{'\n'}" +vcaanode00 + hddl: 12 +``` + +## Testing and Demos + +We can test the plugin is working by deploying the provided example OpenVINO image with HDDL plugin enabled. + +### Build a Docker image with an classification example + +```bash +$ cd $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes +$ make ubuntu-demo-openvino +... +Successfully tagged intel/ubuntu-demo-openvino:devel +``` + +### Create a job running unit tests off the local Docker image + +```bash +$ cd $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes +$ kubectl apply -f demo/intelvpu-job.yaml +job.batch/intelvpu-demo-job created +``` + +### Review the job logs + +```bash +$ kubectl get pods | fgrep intelvpu +# substitute the 'xxxxx' below for the pod name listed in the above +$ kubectl logs intelvpu-demo-job-xxxxx ++ export HDDL_INSTALL_DIR=/root/hddl ++ HDDL_INSTALL_DIR=/root/hddl ++ export LD_LIBRARY_PATH=/root/inference_engine_samples_build/intel64/Release/lib/ ++ LD_LIBRARY_PATH=/root/inference_engine_samples_build/intel64/Release/lib/ ++ /root/inference_engine_samples_build/intel64/Release/classification_sample_async -m /root/openvino_models/ir/FP16/classification/squeezenet/1.1/caffe/squeezenet1.1.xml -i /root/car.png -d HDDL +[ INFO ] InferenceEngine: + API version ............ 2.0 + Build .................. custom_releases/2019/R2_f5827d4773ebbe727c9acac5f007f7d94dd4be4e + Description ....... API +[ INFO ] Parsing input parameters +[ INFO ] Parsing input parameters +[ INFO ] Files were added: 1 +[ INFO ] /root/car.png +[ INFO ] Creating Inference Engine + HDDL + HDDLPlugin version ......... 2.0 + Build ........... 27579 + +[ INFO ] Loading network files +[ INFO ] Preparing input blobs +[ WARNING ] Image is resized from (787, 259) to (227, 227) +[ INFO ] Batch size is 1 +[ INFO ] Loading model to the device +[07:49:01.0427][6]I[ServiceStarter.cpp:40] Info: Waiting for HDDL Service getting ready ... +[07:49:01.0428][6]I[ServiceStarter.cpp:45] Info: Found HDDL Service is running. +[HDDLPlugin] [07:49:01.0429][6]I[HddlClient.cpp:256] Hddl api version: 2.2 +[HDDLPlugin] [07:49:01.0429][6]I[HddlClient.cpp:259] Info: Create Dispatcher2. +[HDDLPlugin] [07:49:01.0432][10]I[Dispatcher2.cpp:148] Info: SenderRoutine starts. +[HDDLPlugin] [07:49:01.0432][6]I[HddlClient.cpp:270] Info: RegisterClient HDDLPlugin. +[HDDLPlugin] [07:49:01.0435][6]I[HddlClient.cpp:275] Client Id: 3 +[ INFO ] Create infer request +[HDDLPlugin] [07:49:01.7235][6]I[HddlBlob.cpp:166] Info: HddlBlob initialize ion ... +[HDDLPlugin] [07:49:01.7237][6]I[HddlBlob.cpp:176] Info: HddlBlob initialize ion successfully. +[ INFO ] Start inference (10 asynchronous executions) +[ INFO ] Completed 1 async request execution +[ INFO ] Completed 2 async request execution +[ INFO ] Completed 3 async request execution +[ INFO ] Completed 4 async request execution +[ INFO ] Completed 5 async request execution +[ INFO ] Completed 6 async request execution +[ INFO ] Completed 7 async request execution +[ INFO ] Completed 8 async request execution +[ INFO ] Completed 9 async request execution +[ INFO ] Completed 10 async request execution +[ INFO ] Processing output blobs + +Top 10 results: + +Image /root/car.png + +classid probability label +------- ----------- ----- +817 0.8295898 sports car, sport car +511 0.0961304 convertible +479 0.0439453 car wheel +751 0.0101318 racer, race car, racing car +436 0.0074234 beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon +656 0.0042267 minivan +586 0.0029869 half track +717 0.0018148 pickup, pickup truck +864 0.0013924 tow truck, tow car, wrecker +581 0.0006595 grille, radiator grille + +[HDDLPlugin] [07:49:01.9231][11]I[Dispatcher2.cpp:212] Info: Listen Thread wake up and to exit. +[HDDLPlugin] [07:49:01.9232][6]I[Dispatcher2.cpp:81] Info: Client dispatcher exit. +[HDDLPlugin] [07:49:01.9235][6]I[HddlClient.cpp:203] Info: Hddl client unregistered. +[ INFO ] Execution successful + +[ INFO ] This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool +``` + +If the pod did not successfully launch, possibly because it could not obtain the vpu HDDL +resource, it will be stuck in the `Pending` status: + +```bash +$ kubectl get pods +NAME READY STATUS RESTARTS AGE +intelvpu-demo-job-xxxxx 0/1 Pending 0 8s +``` + +This can be verified by checking the Events of the pod: + +```bash +$ kubectl describe pod intelvpu-demo-job-xxxxx +... +Events: +Type Reason Age From Message +---- ------ ---- ---- ------- +Warning FailedScheduling default-scheduler 0/1 nodes are available: 1 Insufficient vpu.intel.com/hddl. +``` diff --git a/0.26/_sources/cmd/xpumanager_sidecar/README.md.txt b/0.26/_sources/cmd/xpumanager_sidecar/README.md.txt new file mode 100644 index 000000000..dff89d89c --- /dev/null +++ b/0.26/_sources/cmd/xpumanager_sidecar/README.md.txt @@ -0,0 +1,72 @@ +# XeLink sidecar for Intel XPU Manager + +Table of Contents + +* [Introduction](#introduction) +* [Modes and Configuration Options](#modes-and-configuration-options) +* [Installation](#installation) + * [Install XPU-Manager with the Sidecar](#install-xpu-manager-with-the-sidecar) + * [Install Sidecar to an Existing XPU-Manager](#install-sidecar-to-an-existing-xpu-manager) +* [Verify Sidecar Functionality](#verify-sidecar-functionality) + +## Introduction + +Intel GPUs can be interconnected via an XeLink. In some workloads it is beneficial to use GPUs that are XeLinked together for optimal performance. XeLink information is provided by [Intel XPU Manager](https://www.github.com/intel/xpumanager) via its metrics API. Xelink sidecar retrieves the information from XPU Manager and stores it on the node under ```/etc/kubernetes/node-feature-discovery/features.d/``` as a feature label file. [NFD](https://github.com/kubernetes-sigs/node-feature-discovery) reads this file and converts it to Kubernetes node labels. These labels are then used by [GAS](https://github.com/intel/platform-aware-scheduling/tree/master/gpu-aware-scheduling) to make [scheduling decisions](https://github.com/intel/platform-aware-scheduling/blob/master/gpu-aware-scheduling/docs/usage.md#multi-gpu-allocation-with-xe-link-connections) for Pods. + +## Modes and Configuration Options + +| Flag | Argument | Default | Meaning | +|:---- |:-------- |:------- |:------- | +| -lane-count | int | 4 | Minimum lane count for an XeLink interconnect to be accepted | +| -interval | int | 10 | Interval for XeLink topology fetching and label writing (seconds, >= 1) | +| -startup-delay | int | 10 | Startup delay before the first topology fetching (seconds, >= 0) | +| -label-namespace | string | gpu.intel.com | Namespace or prefix for the labels. i.e. **gpu.intel.com**/xe-links | + +The sidecar also accepts a number of other arguments. Please use the -h option to see the complete list of options. + +## Installation + +The following sections detail how to obtain, deploy and test the XPU-Manager XeLink sidecar. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-xpumanager-sidecar) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. + +Note: Replace `` with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin. + +#### Install XPU-Manager with the Sidecar + +Install XPU-Manager daemonset with the XeLink sidecar + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/xpumanager_sidecar?ref=' +``` + +Please see XPU-Manager Kubernetes files for additional info on [installation](https://github.com/intel/xpumanager/tree/master/deployment/kubernetes). + +#### Install Sidecar to an Existing XPU-Manager + +Use patch to add sidecar into the XPU-Manager daemonset. + +```bash +$ kubectl patch daemonsets.apps intel-xpumanager --patch-file 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/xpumanager_sidecar/kustom/kustom_xpumanager.yaml?ref=' +``` + +NOTE: The sidecar patch will remove other resources from the XPU-Manager container. If your XPU-Manager daemonset is using, for example, the smarter device manager resources, those will be removed. + +#### Verify Sidecar Functionality + +You can verify the sidecar's functionality by checking node's xe-links labels: + +```bash +$ kubectl get nodes -A -o=jsonpath="{range .items[*]}{.metadata.name},{.metadata.labels.gpu\.intel\.com\/xe-links}{'\n'}{end}" +master,0.0-1.0_0.1-1.1 +``` diff --git a/0.26/_sources/demo/readme.md.txt b/0.26/_sources/demo/readme.md.txt new file mode 100644 index 000000000..7e582d2dc --- /dev/null +++ b/0.26/_sources/demo/readme.md.txt @@ -0,0 +1,192 @@ +# Demo + +Table of Contents + +- [Demo overview](#demo-overview) +- [Intel® GPU Device Plugin demo video](#intel-gpu-device-plugin-demo-video) +- [Intel® FPGA Device Plugin demo videos](#intel-fpga-device-plugin-demo-videos) +- [Intel® SGX Device Plugin demo videos](#intel-sgx-device-plugin-demo-videos) +- [Intel® QuickAssist Technology Device Plugin OpenSSL demo video](#intel-quickassist-technology-device-plugin-openssl-demo-video) +- [Intel® QuickAssist Technology Device Plugin with DPDK demo video](#intel-quickassist-technology-device-plugin-with-dpdk-demo-video) + +## Demo overview + +Acceleration of compute and data processing of workloads like video +transcoding, compression, and encryption is enabled in Kubernetes with +the [Device Plugin Framework](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/). This repository +contains a set of Kubernetes plugins and instructions to enable Intel devices +for the acceleration of your workloads orchestrated by Kubernetes. + +The current list of supported Intel Device Plugins includes: + +- [GPU Device Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/gpu_plugin/README.md) with support for [Intel® Graphics Technology](https://www.intel.com/content/www/us/en/architecture-and-technology/visual-technology/graphics-overview.html) +- [Intel® FPGA Device Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/fpga_plugin/README.md) +- [Intel® QuickAssist Technology (QAT) Device Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/qat_plugin/README.md) + +We have included an example demo and configuration information for the Intel +Device Plugins for Kubernetes below. Please join us on the sig-node-rtk channel +on [kubernetes.slack.com](https://kubernetes.slack.com/) to ask questions, +contribute to the community, and learn about the work we are doing with +Kubernetes and the Device Plugin Framework. + +## Intel® GPU Device Plugin demo video + +The screencast demonstrates the deployment of the Intel® GPU Device Plugin for +Kubernetes including Kubeless Function as a Service (FaaS) media transcoder +JavaScript function. The media transcoding workload is scheduled on two different worker nodes. +Only one worker node has a GPU. The time difference in transcoding speed is captured. + +### Demo platform configuration + +- Hardware 2-nodes + - 1x Virtual Machine on Intel® Xeon® E5-2687 CPU @ 3.0 GHz + - 1x Intel® NUC KIT NUC6i7KYK (Skull Canyon) with Intel integrated GPU +- Software + - Ubuntu* 18.04 (Kernel: 4.15.0-36-generic) + - Kubernetes* 1.11 + - Docker* 18.3.1 + - Intel® GPU Device Plugin built from main branch + +### Screencast + +[![Intel® GPU Device Plugin demo](https://img.youtube.com/vi/sg31rV1FdQk/0.jpg)](https://youtu.be/sg31rV1FdQk) + +## Intel® FPGA Device Plugin demo videos + +The screencasts demonstrate the deployment of the Intel® FPGA Device Plugin for +Kubernetes. + +- Demo 1 executes a native loopback 3 (NLB3) workload in preprogrammed mode +- Demo 2 executes NLB3 workload in orchestrated mode +- Demo 3 runs an [OpenCL workload](https://www.intel.com/content/www/us/en/programmable/support/support-resources/design-examples/design-software/opencl/matrix-multiplication.html) to do English letters recognition, and compares time used with and without FPGA to show the acceleration. + +The demos begin with a fully [configured Kubernetes cluster](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/). + +### Demo 1 and 2 (NLB3) platform configuration + +- Hardware + - 1-node, Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz + - Total memory 62 GB + - 2 x Intel® Arria® 10 GX FPGA Programmable Acceleration Card(PAC) +- Software + - openSUSE Leap 15.1 (Kernel: 5.4.5-1.g47eef04-default) + - Kubernetes* 1.17 + - CRI-O 1.13.1 (for orchestrated mode) + - Docker 19.03.1 (for preprogrammed mode) + - runc 1.0.0-rc8 + - Intel® FPGA Device Plugin built from main branch + +### Demo 3 (OpenCL) platform configuration + +- Hardware + - Multi-node, FPGA host has 24 cores Intel(R) Xeon(R) Gold 6252N CPU @ 2.30GHz + - Total memory 195 GB + - Intel® FPGA Programmable Acceleration Card D5005 +- Software + - Centos 7 (Kernel: 3.10.0-1062.12.1.el7.x86_64) on worker node host + - Kubernetes* 1.17 + - CRI-O 1.17.3 + - Intel® FPGA Device Plugin built from main branch + +### Demo steps + +1. Validate the status of the [Kubernetes cluster](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/). +2. Provision the [Intel® FPGA Device Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/fpga_plugin/README.md). +3. Create bitstream storage (for orchestrated mode only) +4. Run the NLB3 or OpenCL workload. + +### Screencasts + +- Intel® FPGA Device Plugin deployment in preprogrammed mode and NLB workload: + +[](https://asciinema.org/a/oIwOtM8hflsWTDu6UhNVS8401) + +- Intel® FPGA Device Plugin deployment with orchestrated/region mode and NLB workload: + +[](https://asciinema.org/a/sUnLNwpazbUXKdpC66g09W5w1) + +- Intel® FPGA Device Plugin deployment with orchestrated/region mode and OpenCL workload: + +[](https://asciinema.org/a/344184) + +## Intel® SGX Device Plugin demo videos + +This video demonstrates the Intel® Software Guard Extensions ECDSA Quote Generation in Kubernetes* + +- Hardware + - 1-node, 3rd Generation Intel® Xeon® Scalable Platform, code-named “Ice Lake” +- Software + - 18.04.5 LTS + - Kubernetes* 1.19 + - containerd 1.3.3 + - Intel® SGX Device Plugin built from main branch + - Intel® SGX SDK 2.12 and DCAP 1.9 + - node-feature-discovery 0.6.0 + +### Demo steps + +1. Validate the status of the [Kubernetes cluster](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/). +2. Provision [node-feature-discovery](https://github.com/kubernetes-sigs/node-feature-discovery). +3. Provision the [Intel® SGX Device Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/sgx_plugin/README.md) using [Intel® Device Plugin Operator](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/operator/README.md). +4. Check the SGX resources and labels are correctly registered. +5. Run Intel® SGX DCAP ECDSA Quote Generation in both "out-of-proc" and "in-proc" modes. + +### Screencasts + +Intel® SGX Device Plugin and SGX DCAP ECDSA Quote Generation demo +[](https://asciinema.org/a/0xa888OjktVyz6xf0RSQ6Pi0w) + +## Intel® QuickAssist Technology Device Plugin OpenSSL demo video + +The screencast demonstrates the deployment of the [Intel® QAT Device Plugin](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/cmd/qat_plugin/README.md) for +Kubernetes and executes a sample QAT accelerated OpenSSL workload with the OCI +compatible Kata Containers runtime, a lightweight Virtual Machine (VM) that feels +and performs like traditional containers, but provides the workload isolation +and security advantages of VMs. The demo begins with a fully [configured Kubernetes cluster](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) +and [Kata Containers](https://github.com/kata-containers/documentation/tree/master/install) runtime for workloads. + +### Demo platform configuration + +- Hardware + - 1-node, 2x Intel® Xeon® CPU E5-2687W v4 @ 3.00GHz + - Total memory 251 GB DDR4 + - Intel® QAT C62x chipset +- Software + - OpenSUSE* 15 (Kernel:4.12.14-lp150.12.22-default) + - Kubernetes* 1.12 + - Containerd 1.2 + - Kata Containers* 1.3.0 + - Intel® QAT Device Plugin built from main + - QAT 1.7 L.4.3.0-00033 + +### Demo steps + +1. Load the host drivers and prepare the virtual function (VF) devices. +2. Check the Kubernetes cluster is in good shape. +3. Deploy the Intel® QAT device plugin for Kubernetes. +4. Deploy an Intel® QAT Accelerated OpenSSL workload. +5. Testing! + +### Screencast +Intel® QAT Device Plugin deployment + +[](https://asciinema.org/a/2N7wF3c9oeCuB9sFqTNm2gmOr) + +## Intel® QuickAssist Technology Device Plugin with DPDK demo video + +### Demo steps + +1. Check health of Kubernetes nodes. +2. Check for allocatable resources. +3. List QAT Virtual Functions. +4. Deploy QAT Device Plugin as a Daemonset. +5. Check again for allocatable resources. +6. List QAT Virtual Functions again, ensuring they are now bound to DPDK driver. +7. View pod specification file for pod requesting QAT VFs. +8. Create pod requesting QAT VFs. +9. Get a shell to the running container and run a DPDK application. + +### Screencast +Intel® QAT Device Plugin with DPDK: + +[](https://asciinema.org/a/PoWOz4q2lX4AF4K9A2AV1RtSA) \ No newline at end of file diff --git a/0.26/_sources/docs/extensions.rst.txt b/0.26/_sources/docs/extensions.rst.txt new file mode 100644 index 000000000..19a65ed46 --- /dev/null +++ b/0.26/_sources/docs/extensions.rst.txt @@ -0,0 +1,19 @@ +Extensions +########## + +.. toctree:: + + ../cmd/dlb_plugin/README.md + ../cmd/dsa_plugin/README.md + ../cmd/fpga_admissionwebhook/README.md + ../cmd/fpga_crihook/README.md + ../cmd/fpga_plugin/README.md + ../cmd/fpga_tool/README.md + ../cmd/gpu_nfdhook/README.md + ../cmd/gpu_plugin/README.md + ../cmd/iaa_plugin/README.md + ../cmd/operator/README.md + ../cmd/qat_plugin/README.md + ../cmd/sgx_plugin/README.md + ../cmd/sgx_admissionwebhook/README.md + ../cmd/vpu_plugin/README.md diff --git a/0.26/_sources/index.rst.txt b/0.26/_sources/index.rst.txt new file mode 100644 index 000000000..31b020242 --- /dev/null +++ b/0.26/_sources/index.rst.txt @@ -0,0 +1,10 @@ +Intel® Device Plugins for Kubernetes +#################################### + +.. toctree:: + + README.md + DEVEL.md + docs/extensions.rst + demo/readme.md + Project GitHub repository \ No newline at end of file diff --git a/0.26/_static/_sphinx_javascript_frameworks_compat.js b/0.26/_static/_sphinx_javascript_frameworks_compat.js new file mode 100644 index 000000000..8549469dc --- /dev/null +++ b/0.26/_static/_sphinx_javascript_frameworks_compat.js @@ -0,0 +1,134 @@ +/* + * _sphinx_javascript_frameworks_compat.js + * ~~~~~~~~~~ + * + * Compatability shim for jQuery and underscores.js. + * + * WILL BE REMOVED IN Sphinx 6.0 + * xref RemovedInSphinx60Warning + * + */ + +/** + * select a different prefix for underscore + */ +$u = _.noConflict(); + + +/** + * small helper function to urldecode strings + * + * See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent#Decoding_query_parameters_from_a_URL + */ +jQuery.urldecode = function(x) { + if (!x) { + return x + } + return decodeURIComponent(x.replace(/\+/g, ' ')); +}; + +/** + * small helper function to urlencode strings + */ +jQuery.urlencode = encodeURIComponent; + +/** + * This function returns the parsed url parameters of the + * current request. Multiple values per key are supported, + * it will always return arrays of strings for the value parts. + */ +jQuery.getQueryParameters = function(s) { + if (typeof s === 'undefined') + s = document.location.search; + var parts = s.substr(s.indexOf('?') + 1).split('&'); + var result = {}; + for (var i = 0; i < parts.length; i++) { + var tmp = parts[i].split('=', 2); + var key = jQuery.urldecode(tmp[0]); + var value = jQuery.urldecode(tmp[1]); + if (key in result) + result[key].push(value); + else + result[key] = [value]; + } + return result; +}; + +/** + * highlight a given string on a jquery object by wrapping it in + * span elements with the given class name. + */ +jQuery.fn.highlightText = function(text, className) { + function highlight(node, addItems) { + if (node.nodeType === 3) { + var val = node.nodeValue; + var pos = val.toLowerCase().indexOf(text); + if (pos >= 0 && + !jQuery(node.parentNode).hasClass(className) && + !jQuery(node.parentNode).hasClass("nohighlight")) { + var span; + var isInSVG = jQuery(node).closest("body, svg, foreignObject").is("svg"); + if (isInSVG) { + span = document.createElementNS("http://www.w3.org/2000/svg", "tspan"); + } else { + span = document.createElement("span"); + span.className = className; + } + span.appendChild(document.createTextNode(val.substr(pos, text.length))); + node.parentNode.insertBefore(span, node.parentNode.insertBefore( + document.createTextNode(val.substr(pos + text.length)), + node.nextSibling)); + node.nodeValue = val.substr(0, pos); + if (isInSVG) { + var rect = document.createElementNS("http://www.w3.org/2000/svg", "rect"); + var bbox = node.parentElement.getBBox(); + rect.x.baseVal.value = bbox.x; + rect.y.baseVal.value = bbox.y; + rect.width.baseVal.value = bbox.width; + rect.height.baseVal.value = bbox.height; + rect.setAttribute('class', className); + addItems.push({ + "parent": node.parentNode, + "target": rect}); + } + } + } + else if (!jQuery(node).is("button, select, textarea")) { + jQuery.each(node.childNodes, function() { + highlight(this, addItems); + }); + } + } + var addItems = []; + var result = this.each(function() { + highlight(this, addItems); + }); + for (var i = 0; i < addItems.length; ++i) { + jQuery(addItems[i].parent).before(addItems[i].target); + } + return result; +}; + +/* + * backward compatibility for jQuery.browser + * This will be supported until firefox bug is fixed. + */ +if (!jQuery.browser) { + jQuery.uaMatch = function(ua) { + ua = ua.toLowerCase(); + + var match = /(chrome)[ \/]([\w.]+)/.exec(ua) || + /(webkit)[ \/]([\w.]+)/.exec(ua) || + /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) || + /(msie) ([\w.]+)/.exec(ua) || + ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) || + []; + + return { + browser: match[ 1 ] || "", + version: match[ 2 ] || "0" + }; + }; + jQuery.browser = {}; + jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true; +} diff --git a/0.26/_static/basic.css b/0.26/_static/basic.css new file mode 100644 index 000000000..eeb0519a6 --- /dev/null +++ b/0.26/_static/basic.css @@ -0,0 +1,899 @@ +/* + * basic.css + * ~~~~~~~~~ + * + * Sphinx stylesheet -- basic theme. + * + * :copyright: Copyright 2007-2022 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ + +/* -- main layout ----------------------------------------------------------- */ + +div.clearer { + clear: both; +} + +div.section::after { + display: block; + content: ''; + clear: left; +} + +/* -- relbar ---------------------------------------------------------------- */ + +div.related { + width: 100%; + font-size: 90%; +} + +div.related h3 { + display: none; +} + +div.related ul { + margin: 0; + padding: 0 0 0 10px; + list-style: none; +} + +div.related li { + display: inline; +} + +div.related li.right { + float: right; + margin-right: 5px; +} + +/* -- sidebar --------------------------------------------------------------- */ + +div.sphinxsidebarwrapper { + padding: 10px 5px 0 10px; +} + +div.sphinxsidebar { + float: left; + width: 230px; + margin-left: -100%; + font-size: 90%; + word-wrap: break-word; + overflow-wrap : break-word; +} + +div.sphinxsidebar ul { + list-style: none; +} + +div.sphinxsidebar ul ul, +div.sphinxsidebar ul.want-points { + margin-left: 20px; + list-style: square; +} + +div.sphinxsidebar ul ul { + margin-top: 0; + margin-bottom: 0; +} + +div.sphinxsidebar form { + margin-top: 10px; +} + +div.sphinxsidebar input { + border: 1px solid #98dbcc; + font-family: sans-serif; + font-size: 1em; +} + +div.sphinxsidebar #searchbox form.search { + overflow: hidden; +} + +div.sphinxsidebar #searchbox input[type="text"] { + float: left; + width: 80%; + padding: 0.25em; + box-sizing: border-box; +} + +div.sphinxsidebar #searchbox input[type="submit"] { + float: left; + width: 20%; + border-left: none; + padding: 0.25em; + box-sizing: border-box; +} + + +img { + border: 0; + max-width: 100%; +} + +/* -- search page ----------------------------------------------------------- */ + +ul.search { + margin: 10px 0 0 20px; + padding: 0; +} + +ul.search li { + padding: 5px 0 5px 20px; + background-image: url(file.png); + background-repeat: no-repeat; + background-position: 0 7px; +} + +ul.search li a { + font-weight: bold; +} + +ul.search li p.context { + color: #888; + margin: 2px 0 0 30px; + text-align: left; +} + +ul.keywordmatches li.goodmatch a { + font-weight: bold; +} + +/* -- index page ------------------------------------------------------------ */ + +table.contentstable { + width: 90%; + margin-left: auto; + margin-right: auto; +} + +table.contentstable p.biglink { + line-height: 150%; +} + +a.biglink { + font-size: 1.3em; +} + +span.linkdescr { + font-style: italic; + padding-top: 5px; + font-size: 90%; +} + +/* -- general index --------------------------------------------------------- */ + +table.indextable { + width: 100%; +} + +table.indextable td { + text-align: left; + vertical-align: top; +} + +table.indextable ul { + margin-top: 0; + margin-bottom: 0; + list-style-type: none; +} + +table.indextable > tbody > tr > td > ul { + padding-left: 0em; +} + +table.indextable tr.pcap { + height: 10px; +} + +table.indextable tr.cap { + margin-top: 10px; + background-color: #f2f2f2; +} + +img.toggler { + margin-right: 3px; + margin-top: 3px; + cursor: pointer; +} + +div.modindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +div.genindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +/* -- domain module index --------------------------------------------------- */ + +table.modindextable td { + padding: 2px; + border-collapse: collapse; +} + +/* -- general body styles --------------------------------------------------- */ + +div.body { + min-width: 360px; + max-width: 800px; +} + +div.body p, div.body dd, div.body li, div.body blockquote { + -moz-hyphens: auto; + -ms-hyphens: auto; + -webkit-hyphens: auto; + hyphens: auto; +} + +a.headerlink { + visibility: hidden; +} +a.brackets:before, +span.brackets > a:before{ + content: "["; +} + +a.brackets:after, +span.brackets > a:after { + content: "]"; +} + + +h1:hover > a.headerlink, +h2:hover > a.headerlink, +h3:hover > a.headerlink, +h4:hover > a.headerlink, +h5:hover > a.headerlink, +h6:hover > a.headerlink, +dt:hover > a.headerlink, +caption:hover > a.headerlink, +p.caption:hover > a.headerlink, +div.code-block-caption:hover > a.headerlink { + visibility: visible; +} + +div.body p.caption { + text-align: inherit; +} + +div.body td { + text-align: left; +} + +.first { + margin-top: 0 !important; +} + +p.rubric { + margin-top: 30px; + font-weight: bold; +} + +img.align-left, figure.align-left, .figure.align-left, object.align-left { + clear: left; + float: left; + margin-right: 1em; +} + +img.align-right, figure.align-right, .figure.align-right, object.align-right { + clear: right; + float: right; + margin-left: 1em; +} + +img.align-center, figure.align-center, .figure.align-center, object.align-center { + display: block; + margin-left: auto; + margin-right: auto; +} + +img.align-default, figure.align-default, .figure.align-default { + display: block; + margin-left: auto; + margin-right: auto; +} + +.align-left { + text-align: left; +} + +.align-center { + text-align: center; +} + +.align-default { + text-align: center; +} + +.align-right { + text-align: right; +} + +/* -- sidebars -------------------------------------------------------------- */ + +div.sidebar, +aside.sidebar { + margin: 0 0 0.5em 1em; + border: 1px solid #ddb; + padding: 7px; + background-color: #ffe; + width: 40%; + float: right; + clear: right; + overflow-x: auto; +} + +p.sidebar-title { + font-weight: bold; +} +div.admonition, div.topic, blockquote { + clear: left; +} + +/* -- topics ---------------------------------------------------------------- */ +div.topic { + border: 1px solid #ccc; + padding: 7px; + margin: 10px 0 10px 0; +} + +p.topic-title { + font-size: 1.1em; + font-weight: bold; + margin-top: 10px; +} + +/* -- admonitions ----------------------------------------------------------- */ + +div.admonition { + margin-top: 10px; + margin-bottom: 10px; + padding: 7px; +} + +div.admonition dt { + font-weight: bold; +} + +p.admonition-title { + margin: 0px 10px 5px 0px; + font-weight: bold; +} + +div.body p.centered { + text-align: center; + margin-top: 25px; +} + +/* -- content of sidebars/topics/admonitions -------------------------------- */ + +div.sidebar > :last-child, +aside.sidebar > :last-child, +div.topic > :last-child, +div.admonition > :last-child { + margin-bottom: 0; +} + +div.sidebar::after, +aside.sidebar::after, +div.topic::after, +div.admonition::after, +blockquote::after { + display: block; + content: ''; + clear: both; +} + +/* -- tables ---------------------------------------------------------------- */ + +table.docutils { + margin-top: 10px; + margin-bottom: 10px; + border: 0; + border-collapse: collapse; +} + +table.align-center { + margin-left: auto; + margin-right: auto; +} + +table.align-default { + margin-left: auto; + margin-right: auto; +} + +table caption span.caption-number { + font-style: italic; +} + +table caption span.caption-text { +} + +table.docutils td, table.docutils th { + padding: 1px 8px 1px 5px; + border-top: 0; + border-left: 0; + border-right: 0; + border-bottom: 1px solid #aaa; +} + +th { + text-align: left; + padding-right: 5px; +} + +table.citation { + border-left: solid 1px gray; + margin-left: 1px; +} + +table.citation td { + border-bottom: none; +} + +th > :first-child, +td > :first-child { + margin-top: 0px; +} + +th > :last-child, +td > :last-child { + margin-bottom: 0px; +} + +/* -- figures --------------------------------------------------------------- */ + +div.figure, figure { + margin: 0.5em; + padding: 0.5em; +} + +div.figure p.caption, figcaption { + padding: 0.3em; +} + +div.figure p.caption span.caption-number, +figcaption span.caption-number { + font-style: italic; +} + +div.figure p.caption span.caption-text, +figcaption span.caption-text { +} + +/* -- field list styles ----------------------------------------------------- */ + +table.field-list td, table.field-list th { + border: 0 !important; +} + +.field-list ul { + margin: 0; + padding-left: 1em; +} + +.field-list p { + margin: 0; +} + +.field-name { + -moz-hyphens: manual; + -ms-hyphens: manual; + -webkit-hyphens: manual; + hyphens: manual; +} + +/* -- hlist styles ---------------------------------------------------------- */ + +table.hlist { + margin: 1em 0; +} + +table.hlist td { + vertical-align: top; +} + +/* -- object description styles --------------------------------------------- */ + +.sig { + font-family: 'Consolas', 'Menlo', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace; +} + +.sig-name, code.descname { + background-color: transparent; + font-weight: bold; +} + +.sig-name { + font-size: 1.1em; +} + +code.descname { + font-size: 1.2em; +} + +.sig-prename, code.descclassname { + background-color: transparent; +} + +.optional { + font-size: 1.3em; +} + +.sig-paren { + font-size: larger; +} + +.sig-param.n { + font-style: italic; +} + +/* C++ specific styling */ + +.sig-inline.c-texpr, +.sig-inline.cpp-texpr { + font-family: unset; +} + +.sig.c .k, .sig.c .kt, +.sig.cpp .k, .sig.cpp .kt { + color: #0033B3; +} + +.sig.c .m, +.sig.cpp .m { + color: #1750EB; +} + +.sig.c .s, .sig.c .sc, +.sig.cpp .s, .sig.cpp .sc { + color: #067D17; +} + + +/* -- other body styles ----------------------------------------------------- */ + +ol.arabic { + list-style: decimal; +} + +ol.loweralpha { + list-style: lower-alpha; +} + +ol.upperalpha { + list-style: upper-alpha; +} + +ol.lowerroman { + list-style: lower-roman; +} + +ol.upperroman { + list-style: upper-roman; +} + +:not(li) > ol > li:first-child > :first-child, +:not(li) > ul > li:first-child > :first-child { + margin-top: 0px; +} + +:not(li) > ol > li:last-child > :last-child, +:not(li) > ul > li:last-child > :last-child { + margin-bottom: 0px; +} + +ol.simple ol p, +ol.simple ul p, +ul.simple ol p, +ul.simple ul p { + margin-top: 0; +} + +ol.simple > li:not(:first-child) > p, +ul.simple > li:not(:first-child) > p { + margin-top: 0; +} + +ol.simple p, +ul.simple p { + margin-bottom: 0; +} +dl.footnote > dt, +dl.citation > dt { + float: left; + margin-right: 0.5em; +} + +dl.footnote > dd, +dl.citation > dd { + margin-bottom: 0em; +} + +dl.footnote > dd:after, +dl.citation > dd:after { + content: ""; + clear: both; +} + +dl.field-list { + display: grid; + grid-template-columns: fit-content(30%) auto; +} + +dl.field-list > dt { + font-weight: bold; + word-break: break-word; + padding-left: 0.5em; + padding-right: 5px; +} +dl.field-list > dt:after { + content: ":"; +} + + +dl.field-list > dd { + padding-left: 0.5em; + margin-top: 0em; + margin-left: 0em; + margin-bottom: 0em; +} + +dl { + margin-bottom: 15px; +} + +dd > :first-child { + margin-top: 0px; +} + +dd ul, dd table { + margin-bottom: 10px; +} + +dd { + margin-top: 3px; + margin-bottom: 10px; + margin-left: 30px; +} + +dl > dd:last-child, +dl > dd:last-child > :last-child { + margin-bottom: 0; +} + +dt:target, span.highlighted { + background-color: #fbe54e; +} + +rect.highlighted { + fill: #fbe54e; +} + +dl.glossary dt { + font-weight: bold; + font-size: 1.1em; +} + +.versionmodified { + font-style: italic; +} + +.system-message { + background-color: #fda; + padding: 5px; + border: 3px solid red; +} + +.footnote:target { + background-color: #ffa; +} + +.line-block { + display: block; + margin-top: 1em; + margin-bottom: 1em; +} + +.line-block .line-block { + margin-top: 0; + margin-bottom: 0; + margin-left: 1.5em; +} + +.guilabel, .menuselection { + font-family: sans-serif; +} + +.accelerator { + text-decoration: underline; +} + +.classifier { + font-style: oblique; +} + +.classifier:before { + font-style: normal; + margin: 0 0.5em; + content: ":"; + display: inline-block; +} + +abbr, acronym { + border-bottom: dotted 1px; + cursor: help; +} + +/* -- code displays --------------------------------------------------------- */ + +pre { + overflow: auto; + overflow-y: hidden; /* fixes display issues on Chrome browsers */ +} + +pre, div[class*="highlight-"] { + clear: both; +} + +span.pre { + -moz-hyphens: none; + -ms-hyphens: none; + -webkit-hyphens: none; + hyphens: none; + white-space: nowrap; +} + +div[class*="highlight-"] { + margin: 1em 0; +} + +td.linenos pre { + border: 0; + background-color: transparent; + color: #aaa; +} + +table.highlighttable { + display: block; +} + +table.highlighttable tbody { + display: block; +} + +table.highlighttable tr { + display: flex; +} + +table.highlighttable td { + margin: 0; + padding: 0; +} + +table.highlighttable td.linenos { + padding-right: 0.5em; +} + +table.highlighttable td.code { + flex: 1; + overflow: hidden; +} + +.highlight .hll { + display: block; +} + +div.highlight pre, +table.highlighttable pre { + margin: 0; +} + +div.code-block-caption + div { + margin-top: 0; +} + +div.code-block-caption { + margin-top: 1em; + padding: 2px 5px; + font-size: small; +} + +div.code-block-caption code { + background-color: transparent; +} + +table.highlighttable td.linenos, +span.linenos, +div.highlight span.gp { /* gp: Generic.Prompt */ + user-select: none; + -webkit-user-select: text; /* Safari fallback only */ + -webkit-user-select: none; /* Chrome/Safari */ + -moz-user-select: none; /* Firefox */ + -ms-user-select: none; /* IE10+ */ +} + +div.code-block-caption span.caption-number { + padding: 0.1em 0.3em; + font-style: italic; +} + +div.code-block-caption span.caption-text { +} + +div.literal-block-wrapper { + margin: 1em 0; +} + +code.xref, a code { + background-color: transparent; + font-weight: bold; +} + +h1 code, h2 code, h3 code, h4 code, h5 code, h6 code { + background-color: transparent; +} + +.viewcode-link { + float: right; +} + +.viewcode-back { + float: right; + font-family: sans-serif; +} + +div.viewcode-block:target { + margin: -1px -10px; + padding: 0 10px; +} + +/* -- math display ---------------------------------------------------------- */ + +img.math { + vertical-align: middle; +} + +div.body div.math p { + text-align: center; +} + +span.eqno { + float: right; +} + +span.eqno a.headerlink { + position: absolute; + z-index: 1; +} + +div.math:hover a.headerlink { + visibility: visible; +} + +/* -- printout stylesheet --------------------------------------------------- */ + +@media print { + div.document, + div.documentwrapper, + div.bodywrapper { + margin: 0 !important; + width: 100%; + } + + div.sphinxsidebar, + div.related, + div.footer, + #top-link { + display: none; + } +} \ No newline at end of file diff --git a/0.26/_static/css/badge_only.css b/0.26/_static/css/badge_only.css new file mode 100644 index 000000000..c718cee44 --- /dev/null +++ b/0.26/_static/css/badge_only.css @@ -0,0 +1 @@ +.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}@font-face{font-family:FontAwesome;font-style:normal;font-weight:400;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#FontAwesome) format("svg")}.fa:before{font-family:FontAwesome;font-style:normal;font-weight:400;line-height:1}.fa:before,a .fa{text-decoration:inherit}.fa:before,a .fa,li .fa{display:inline-block}li .fa-large:before{width:1.875em}ul.fas{list-style-type:none;margin-left:2em;text-indent:-.8em}ul.fas li .fa{width:.8em}ul.fas li .fa-large:before{vertical-align:baseline}.fa-book:before,.icon-book:before{content:"\f02d"}.fa-caret-down:before,.icon-caret-down:before{content:"\f0d7"}.fa-caret-up:before,.icon-caret-up:before{content:"\f0d8"}.fa-caret-left:before,.icon-caret-left:before{content:"\f0d9"}.fa-caret-right:before,.icon-caret-right:before{content:"\f0da"}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60}.rst-versions .rst-current-version:after{clear:both;content:"";display:block}.rst-versions .rst-current-version .fa{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}} \ No newline at end of file diff --git a/0.26/_static/css/fonts/Roboto-Slab-Bold.woff b/0.26/_static/css/fonts/Roboto-Slab-Bold.woff new file mode 100644 index 000000000..6cb600001 Binary files /dev/null and b/0.26/_static/css/fonts/Roboto-Slab-Bold.woff differ diff --git a/0.26/_static/css/fonts/Roboto-Slab-Bold.woff2 b/0.26/_static/css/fonts/Roboto-Slab-Bold.woff2 new file mode 100644 index 000000000..7059e2314 Binary files /dev/null and b/0.26/_static/css/fonts/Roboto-Slab-Bold.woff2 differ diff --git a/0.26/_static/css/fonts/Roboto-Slab-Regular.woff b/0.26/_static/css/fonts/Roboto-Slab-Regular.woff new file mode 100644 index 000000000..f815f63f9 Binary files /dev/null and b/0.26/_static/css/fonts/Roboto-Slab-Regular.woff differ diff --git a/0.26/_static/css/fonts/Roboto-Slab-Regular.woff2 b/0.26/_static/css/fonts/Roboto-Slab-Regular.woff2 new file mode 100644 index 000000000..f2c76e5bd Binary files /dev/null and b/0.26/_static/css/fonts/Roboto-Slab-Regular.woff2 differ diff --git a/0.26/_static/css/fonts/fontawesome-webfont.eot b/0.26/_static/css/fonts/fontawesome-webfont.eot new file mode 100644 index 000000000..e9f60ca95 Binary files /dev/null and b/0.26/_static/css/fonts/fontawesome-webfont.eot differ diff --git a/0.26/_static/css/fonts/fontawesome-webfont.svg b/0.26/_static/css/fonts/fontawesome-webfont.svg new file mode 100644 index 000000000..855c845e5 --- /dev/null +++ b/0.26/_static/css/fonts/fontawesome-webfont.svg @@ -0,0 +1,2671 @@ + + + + +Created by FontForge 20120731 at Mon Oct 24 17:37:40 2016 + By ,,, +Copyright Dave Gandy 2016. All rights reserved. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/0.26/_static/css/fonts/fontawesome-webfont.ttf b/0.26/_static/css/fonts/fontawesome-webfont.ttf new file mode 100644 index 000000000..35acda2fa Binary files /dev/null and b/0.26/_static/css/fonts/fontawesome-webfont.ttf differ diff --git a/0.26/_static/css/fonts/fontawesome-webfont.woff b/0.26/_static/css/fonts/fontawesome-webfont.woff new file mode 100644 index 000000000..400014a4b Binary files /dev/null and b/0.26/_static/css/fonts/fontawesome-webfont.woff differ diff --git a/0.26/_static/css/fonts/fontawesome-webfont.woff2 b/0.26/_static/css/fonts/fontawesome-webfont.woff2 new file mode 100644 index 000000000..4d13fc604 Binary files /dev/null and b/0.26/_static/css/fonts/fontawesome-webfont.woff2 differ diff --git a/0.26/_static/css/fonts/lato-bold-italic.woff b/0.26/_static/css/fonts/lato-bold-italic.woff new file mode 100644 index 000000000..88ad05b9f Binary files /dev/null and b/0.26/_static/css/fonts/lato-bold-italic.woff differ diff --git a/0.26/_static/css/fonts/lato-bold-italic.woff2 b/0.26/_static/css/fonts/lato-bold-italic.woff2 new file mode 100644 index 000000000..c4e3d804b Binary files /dev/null and b/0.26/_static/css/fonts/lato-bold-italic.woff2 differ diff --git a/0.26/_static/css/fonts/lato-bold.woff b/0.26/_static/css/fonts/lato-bold.woff new file mode 100644 index 000000000..c6dff51f0 Binary files /dev/null and b/0.26/_static/css/fonts/lato-bold.woff differ diff --git a/0.26/_static/css/fonts/lato-bold.woff2 b/0.26/_static/css/fonts/lato-bold.woff2 new file mode 100644 index 000000000..bb195043c Binary files /dev/null and b/0.26/_static/css/fonts/lato-bold.woff2 differ diff --git a/0.26/_static/css/fonts/lato-normal-italic.woff b/0.26/_static/css/fonts/lato-normal-italic.woff new file mode 100644 index 000000000..76114bc03 Binary files /dev/null and b/0.26/_static/css/fonts/lato-normal-italic.woff differ diff --git a/0.26/_static/css/fonts/lato-normal-italic.woff2 b/0.26/_static/css/fonts/lato-normal-italic.woff2 new file mode 100644 index 000000000..3404f37e2 Binary files /dev/null and b/0.26/_static/css/fonts/lato-normal-italic.woff2 differ diff --git a/0.26/_static/css/fonts/lato-normal.woff b/0.26/_static/css/fonts/lato-normal.woff new file mode 100644 index 000000000..ae1307ff5 Binary files /dev/null and b/0.26/_static/css/fonts/lato-normal.woff differ diff --git a/0.26/_static/css/fonts/lato-normal.woff2 b/0.26/_static/css/fonts/lato-normal.woff2 new file mode 100644 index 000000000..3bf984332 Binary files /dev/null and b/0.26/_static/css/fonts/lato-normal.woff2 differ diff --git a/0.26/_static/css/theme.css b/0.26/_static/css/theme.css new file mode 100644 index 000000000..19a446a0e --- /dev/null +++ b/0.26/_static/css/theme.css @@ -0,0 +1,4 @@ +html{box-sizing:border-box}*,:after,:before{box-sizing:inherit}article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{display:block}audio,canvas,video{display:inline-block;*display:inline;*zoom:1}[hidden],audio:not([controls]){display:none}*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}html{font-size:100%;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}a:active,a:hover{outline:0}abbr[title]{border-bottom:1px dotted}b,strong{font-weight:700}blockquote{margin:0}dfn{font-style:italic}ins{background:#ff9;text-decoration:none}ins,mark{color:#000}mark{background:#ff0;font-style:italic;font-weight:700}.rst-content code,.rst-content tt,code,kbd,pre,samp{font-family:monospace,serif;_font-family:courier new,monospace;font-size:1em}pre{white-space:pre}q{quotes:none}q:after,q:before{content:"";content:none}small{font-size:85%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sup{top:-.5em}sub{bottom:-.25em}dl,ol,ul{margin:0;padding:0;list-style:none;list-style-image:none}li{list-style:none}dd{margin:0}img{border:0;-ms-interpolation-mode:bicubic;vertical-align:middle;max-width:100%}svg:not(:root){overflow:hidden}figure,form{margin:0}label{cursor:pointer}button,input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}button,input{line-height:normal}button,input[type=button],input[type=reset],input[type=submit]{cursor:pointer;-webkit-appearance:button;*overflow:visible}button[disabled],input[disabled]{cursor:default}input[type=search]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}textarea{resize:vertical}table{border-collapse:collapse;border-spacing:0}td{vertical-align:top}.chromeframe{margin:.2em 0;background:#ccc;color:#000;padding:.2em 0}.ir{display:block;border:0;text-indent:-999em;overflow:hidden;background-color:transparent;background-repeat:no-repeat;text-align:left;direction:ltr;*line-height:0}.ir br{display:none}.hidden{display:none!important;visibility:hidden}.visuallyhidden{border:0;clip:rect(0 0 0 0);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px}.visuallyhidden.focusable:active,.visuallyhidden.focusable:focus{clip:auto;height:auto;margin:0;overflow:visible;position:static;width:auto}.invisible{visibility:hidden}.relative{position:relative}big,small{font-size:100%}@media print{body,html,section{background:none!important}*{box-shadow:none!important;text-shadow:none!important;filter:none!important;-ms-filter:none!important}a,a:visited{text-decoration:underline}.ir a:after,a[href^="#"]:after,a[href^="javascript:"]:after{content:""}blockquote,pre{page-break-inside:avoid}thead{display:table-header-group}img,tr{page-break-inside:avoid}img{max-width:100%!important}@page{margin:.5cm}.rst-content .toctree-wrapper>p.caption,h2,h3,p{orphans:3;widows:3}.rst-content .toctree-wrapper>p.caption,h2,h3{page-break-after:avoid}}.btn,.fa:before,.icon:before,.rst-content .admonition,.rst-content .admonition-title:before,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .code-block-caption .headerlink:before,.rst-content .danger,.rst-content .eqno .headerlink:before,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content p .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-alert,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a button.toctree-expand:before,.wy-menu-vertical li.on a button.toctree-expand:before,.wy-menu-vertical li button.toctree-expand:before,input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week],select,textarea{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}/*! + * Font Awesome 4.7.0 by @davegandy - http://fontawesome.io - @fontawesome + * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License) + */@font-face{font-family:FontAwesome;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713);src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix&v=4.7.0) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#fontawesomeregular) format("svg");font-weight:400;font-style:normal}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content .eqno .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content p .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-menu-vertical li.current>a button.toctree-expand,.wy-menu-vertical li.on a button.toctree-expand,.wy-menu-vertical li button.toctree-expand{display:inline-block;font:normal normal normal 14px/1 FontAwesome;font-size:inherit;text-rendering:auto;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.fa-lg{font-size:1.33333em;line-height:.75em;vertical-align:-15%}.fa-2x{font-size:2em}.fa-3x{font-size:3em}.fa-4x{font-size:4em}.fa-5x{font-size:5em}.fa-fw{width:1.28571em;text-align:center}.fa-ul{padding-left:0;margin-left:2.14286em;list-style-type:none}.fa-ul>li{position:relative}.fa-li{position:absolute;left:-2.14286em;width:2.14286em;top:.14286em;text-align:center}.fa-li.fa-lg{left:-1.85714em}.fa-border{padding:.2em .25em .15em;border:.08em solid #eee;border-radius:.1em}.fa-pull-left{float:left}.fa-pull-right{float:right}.fa-pull-left.icon,.fa.fa-pull-left,.rst-content .code-block-caption .fa-pull-left.headerlink,.rst-content .eqno .fa-pull-left.headerlink,.rst-content .fa-pull-left.admonition-title,.rst-content code.download span.fa-pull-left:first-child,.rst-content dl dt .fa-pull-left.headerlink,.rst-content h1 .fa-pull-left.headerlink,.rst-content h2 .fa-pull-left.headerlink,.rst-content h3 .fa-pull-left.headerlink,.rst-content h4 .fa-pull-left.headerlink,.rst-content h5 .fa-pull-left.headerlink,.rst-content h6 .fa-pull-left.headerlink,.rst-content p .fa-pull-left.headerlink,.rst-content table>caption .fa-pull-left.headerlink,.rst-content tt.download span.fa-pull-left:first-child,.wy-menu-vertical li.current>a button.fa-pull-left.toctree-expand,.wy-menu-vertical li.on a button.fa-pull-left.toctree-expand,.wy-menu-vertical li button.fa-pull-left.toctree-expand{margin-right:.3em}.fa-pull-right.icon,.fa.fa-pull-right,.rst-content .code-block-caption .fa-pull-right.headerlink,.rst-content .eqno .fa-pull-right.headerlink,.rst-content .fa-pull-right.admonition-title,.rst-content code.download span.fa-pull-right:first-child,.rst-content dl dt .fa-pull-right.headerlink,.rst-content h1 .fa-pull-right.headerlink,.rst-content h2 .fa-pull-right.headerlink,.rst-content h3 .fa-pull-right.headerlink,.rst-content h4 .fa-pull-right.headerlink,.rst-content h5 .fa-pull-right.headerlink,.rst-content h6 .fa-pull-right.headerlink,.rst-content p .fa-pull-right.headerlink,.rst-content table>caption .fa-pull-right.headerlink,.rst-content tt.download span.fa-pull-right:first-child,.wy-menu-vertical li.current>a button.fa-pull-right.toctree-expand,.wy-menu-vertical li.on a button.fa-pull-right.toctree-expand,.wy-menu-vertical li button.fa-pull-right.toctree-expand{margin-left:.3em}.pull-right{float:right}.pull-left{float:left}.fa.pull-left,.pull-left.icon,.rst-content .code-block-caption .pull-left.headerlink,.rst-content .eqno .pull-left.headerlink,.rst-content .pull-left.admonition-title,.rst-content code.download span.pull-left:first-child,.rst-content dl dt .pull-left.headerlink,.rst-content h1 .pull-left.headerlink,.rst-content h2 .pull-left.headerlink,.rst-content h3 .pull-left.headerlink,.rst-content h4 .pull-left.headerlink,.rst-content h5 .pull-left.headerlink,.rst-content h6 .pull-left.headerlink,.rst-content p .pull-left.headerlink,.rst-content table>caption .pull-left.headerlink,.rst-content tt.download span.pull-left:first-child,.wy-menu-vertical li.current>a button.pull-left.toctree-expand,.wy-menu-vertical li.on a button.pull-left.toctree-expand,.wy-menu-vertical li button.pull-left.toctree-expand{margin-right:.3em}.fa.pull-right,.pull-right.icon,.rst-content .code-block-caption .pull-right.headerlink,.rst-content .eqno .pull-right.headerlink,.rst-content .pull-right.admonition-title,.rst-content code.download span.pull-right:first-child,.rst-content dl dt .pull-right.headerlink,.rst-content h1 .pull-right.headerlink,.rst-content h2 .pull-right.headerlink,.rst-content h3 .pull-right.headerlink,.rst-content h4 .pull-right.headerlink,.rst-content h5 .pull-right.headerlink,.rst-content h6 .pull-right.headerlink,.rst-content p .pull-right.headerlink,.rst-content table>caption .pull-right.headerlink,.rst-content tt.download span.pull-right:first-child,.wy-menu-vertical li.current>a button.pull-right.toctree-expand,.wy-menu-vertical li.on a button.pull-right.toctree-expand,.wy-menu-vertical li button.pull-right.toctree-expand{margin-left:.3em}.fa-spin{-webkit-animation:fa-spin 2s linear infinite;animation:fa-spin 2s linear infinite}.fa-pulse{-webkit-animation:fa-spin 1s steps(8) infinite;animation:fa-spin 1s steps(8) infinite}@-webkit-keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}@keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}.fa-rotate-90{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=1)";-webkit-transform:rotate(90deg);-ms-transform:rotate(90deg);transform:rotate(90deg)}.fa-rotate-180{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2)";-webkit-transform:rotate(180deg);-ms-transform:rotate(180deg);transform:rotate(180deg)}.fa-rotate-270{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=3)";-webkit-transform:rotate(270deg);-ms-transform:rotate(270deg);transform:rotate(270deg)}.fa-flip-horizontal{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1)";-webkit-transform:scaleX(-1);-ms-transform:scaleX(-1);transform:scaleX(-1)}.fa-flip-vertical{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1)";-webkit-transform:scaleY(-1);-ms-transform:scaleY(-1);transform:scaleY(-1)}:root .fa-flip-horizontal,:root .fa-flip-vertical,:root .fa-rotate-90,:root .fa-rotate-180,:root .fa-rotate-270{filter:none}.fa-stack{position:relative;display:inline-block;width:2em;height:2em;line-height:2em;vertical-align:middle}.fa-stack-1x,.fa-stack-2x{position:absolute;left:0;width:100%;text-align:center}.fa-stack-1x{line-height:inherit}.fa-stack-2x{font-size:2em}.fa-inverse{color:#fff}.fa-glass:before{content:""}.fa-music:before{content:""}.fa-search:before,.icon-search:before{content:""}.fa-envelope-o:before{content:""}.fa-heart:before{content:""}.fa-star:before{content:""}.fa-star-o:before{content:""}.fa-user:before{content:""}.fa-film:before{content:""}.fa-th-large:before{content:""}.fa-th:before{content:""}.fa-th-list:before{content:""}.fa-check:before{content:""}.fa-close:before,.fa-remove:before,.fa-times:before{content:""}.fa-search-plus:before{content:""}.fa-search-minus:before{content:""}.fa-power-off:before{content:""}.fa-signal:before{content:""}.fa-cog:before,.fa-gear:before{content:""}.fa-trash-o:before{content:""}.fa-home:before,.icon-home:before{content:""}.fa-file-o:before{content:""}.fa-clock-o:before{content:""}.fa-road:before{content:""}.fa-download:before,.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{content:""}.fa-arrow-circle-o-down:before{content:""}.fa-arrow-circle-o-up:before{content:""}.fa-inbox:before{content:""}.fa-play-circle-o:before{content:""}.fa-repeat:before,.fa-rotate-right:before{content:""}.fa-refresh:before{content:""}.fa-list-alt:before{content:""}.fa-lock:before{content:""}.fa-flag:before{content:""}.fa-headphones:before{content:""}.fa-volume-off:before{content:""}.fa-volume-down:before{content:""}.fa-volume-up:before{content:""}.fa-qrcode:before{content:""}.fa-barcode:before{content:""}.fa-tag:before{content:""}.fa-tags:before{content:""}.fa-book:before,.icon-book:before{content:""}.fa-bookmark:before{content:""}.fa-print:before{content:""}.fa-camera:before{content:""}.fa-font:before{content:""}.fa-bold:before{content:""}.fa-italic:before{content:""}.fa-text-height:before{content:""}.fa-text-width:before{content:""}.fa-align-left:before{content:""}.fa-align-center:before{content:""}.fa-align-right:before{content:""}.fa-align-justify:before{content:""}.fa-list:before{content:""}.fa-dedent:before,.fa-outdent:before{content:""}.fa-indent:before{content:""}.fa-video-camera:before{content:""}.fa-image:before,.fa-photo:before,.fa-picture-o:before{content:""}.fa-pencil:before{content:""}.fa-map-marker:before{content:""}.fa-adjust:before{content:""}.fa-tint:before{content:""}.fa-edit:before,.fa-pencil-square-o:before{content:""}.fa-share-square-o:before{content:""}.fa-check-square-o:before{content:""}.fa-arrows:before{content:""}.fa-step-backward:before{content:""}.fa-fast-backward:before{content:""}.fa-backward:before{content:""}.fa-play:before{content:""}.fa-pause:before{content:""}.fa-stop:before{content:""}.fa-forward:before{content:""}.fa-fast-forward:before{content:""}.fa-step-forward:before{content:""}.fa-eject:before{content:""}.fa-chevron-left:before{content:""}.fa-chevron-right:before{content:""}.fa-plus-circle:before{content:""}.fa-minus-circle:before{content:""}.fa-times-circle:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before{content:""}.fa-check-circle:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before{content:""}.fa-question-circle:before{content:""}.fa-info-circle:before{content:""}.fa-crosshairs:before{content:""}.fa-times-circle-o:before{content:""}.fa-check-circle-o:before{content:""}.fa-ban:before{content:""}.fa-arrow-left:before{content:""}.fa-arrow-right:before{content:""}.fa-arrow-up:before{content:""}.fa-arrow-down:before{content:""}.fa-mail-forward:before,.fa-share:before{content:""}.fa-expand:before{content:""}.fa-compress:before{content:""}.fa-plus:before{content:""}.fa-minus:before{content:""}.fa-asterisk:before{content:""}.fa-exclamation-circle:before,.rst-content .admonition-title:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before{content:""}.fa-gift:before{content:""}.fa-leaf:before{content:""}.fa-fire:before,.icon-fire:before{content:""}.fa-eye:before{content:""}.fa-eye-slash:before{content:""}.fa-exclamation-triangle:before,.fa-warning:before{content:""}.fa-plane:before{content:""}.fa-calendar:before{content:""}.fa-random:before{content:""}.fa-comment:before{content:""}.fa-magnet:before{content:""}.fa-chevron-up:before{content:""}.fa-chevron-down:before{content:""}.fa-retweet:before{content:""}.fa-shopping-cart:before{content:""}.fa-folder:before{content:""}.fa-folder-open:before{content:""}.fa-arrows-v:before{content:""}.fa-arrows-h:before{content:""}.fa-bar-chart-o:before,.fa-bar-chart:before{content:""}.fa-twitter-square:before{content:""}.fa-facebook-square:before{content:""}.fa-camera-retro:before{content:""}.fa-key:before{content:""}.fa-cogs:before,.fa-gears:before{content:""}.fa-comments:before{content:""}.fa-thumbs-o-up:before{content:""}.fa-thumbs-o-down:before{content:""}.fa-star-half:before{content:""}.fa-heart-o:before{content:""}.fa-sign-out:before{content:""}.fa-linkedin-square:before{content:""}.fa-thumb-tack:before{content:""}.fa-external-link:before{content:""}.fa-sign-in:before{content:""}.fa-trophy:before{content:""}.fa-github-square:before{content:""}.fa-upload:before{content:""}.fa-lemon-o:before{content:""}.fa-phone:before{content:""}.fa-square-o:before{content:""}.fa-bookmark-o:before{content:""}.fa-phone-square:before{content:""}.fa-twitter:before{content:""}.fa-facebook-f:before,.fa-facebook:before{content:""}.fa-github:before,.icon-github:before{content:""}.fa-unlock:before{content:""}.fa-credit-card:before{content:""}.fa-feed:before,.fa-rss:before{content:""}.fa-hdd-o:before{content:""}.fa-bullhorn:before{content:""}.fa-bell:before{content:""}.fa-certificate:before{content:""}.fa-hand-o-right:before{content:""}.fa-hand-o-left:before{content:""}.fa-hand-o-up:before{content:""}.fa-hand-o-down:before{content:""}.fa-arrow-circle-left:before,.icon-circle-arrow-left:before{content:""}.fa-arrow-circle-right:before,.icon-circle-arrow-right:before{content:""}.fa-arrow-circle-up:before{content:""}.fa-arrow-circle-down:before{content:""}.fa-globe:before{content:""}.fa-wrench:before{content:""}.fa-tasks:before{content:""}.fa-filter:before{content:""}.fa-briefcase:before{content:""}.fa-arrows-alt:before{content:""}.fa-group:before,.fa-users:before{content:""}.fa-chain:before,.fa-link:before,.icon-link:before{content:""}.fa-cloud:before{content:""}.fa-flask:before{content:""}.fa-cut:before,.fa-scissors:before{content:""}.fa-copy:before,.fa-files-o:before{content:""}.fa-paperclip:before{content:""}.fa-floppy-o:before,.fa-save:before{content:""}.fa-square:before{content:""}.fa-bars:before,.fa-navicon:before,.fa-reorder:before{content:""}.fa-list-ul:before{content:""}.fa-list-ol:before{content:""}.fa-strikethrough:before{content:""}.fa-underline:before{content:""}.fa-table:before{content:""}.fa-magic:before{content:""}.fa-truck:before{content:""}.fa-pinterest:before{content:""}.fa-pinterest-square:before{content:""}.fa-google-plus-square:before{content:""}.fa-google-plus:before{content:""}.fa-money:before{content:""}.fa-caret-down:before,.icon-caret-down:before,.wy-dropdown .caret:before{content:""}.fa-caret-up:before{content:""}.fa-caret-left:before{content:""}.fa-caret-right:before{content:""}.fa-columns:before{content:""}.fa-sort:before,.fa-unsorted:before{content:""}.fa-sort-desc:before,.fa-sort-down:before{content:""}.fa-sort-asc:before,.fa-sort-up:before{content:""}.fa-envelope:before{content:""}.fa-linkedin:before{content:""}.fa-rotate-left:before,.fa-undo:before{content:""}.fa-gavel:before,.fa-legal:before{content:""}.fa-dashboard:before,.fa-tachometer:before{content:""}.fa-comment-o:before{content:""}.fa-comments-o:before{content:""}.fa-bolt:before,.fa-flash:before{content:""}.fa-sitemap:before{content:""}.fa-umbrella:before{content:""}.fa-clipboard:before,.fa-paste:before{content:""}.fa-lightbulb-o:before{content:""}.fa-exchange:before{content:""}.fa-cloud-download:before{content:""}.fa-cloud-upload:before{content:""}.fa-user-md:before{content:""}.fa-stethoscope:before{content:""}.fa-suitcase:before{content:""}.fa-bell-o:before{content:""}.fa-coffee:before{content:""}.fa-cutlery:before{content:""}.fa-file-text-o:before{content:""}.fa-building-o:before{content:""}.fa-hospital-o:before{content:""}.fa-ambulance:before{content:""}.fa-medkit:before{content:""}.fa-fighter-jet:before{content:""}.fa-beer:before{content:""}.fa-h-square:before{content:""}.fa-plus-square:before{content:""}.fa-angle-double-left:before{content:""}.fa-angle-double-right:before{content:""}.fa-angle-double-up:before{content:""}.fa-angle-double-down:before{content:""}.fa-angle-left:before{content:""}.fa-angle-right:before{content:""}.fa-angle-up:before{content:""}.fa-angle-down:before{content:""}.fa-desktop:before{content:""}.fa-laptop:before{content:""}.fa-tablet:before{content:""}.fa-mobile-phone:before,.fa-mobile:before{content:""}.fa-circle-o:before{content:""}.fa-quote-left:before{content:""}.fa-quote-right:before{content:""}.fa-spinner:before{content:""}.fa-circle:before{content:""}.fa-mail-reply:before,.fa-reply:before{content:""}.fa-github-alt:before{content:""}.fa-folder-o:before{content:""}.fa-folder-open-o:before{content:""}.fa-smile-o:before{content:""}.fa-frown-o:before{content:""}.fa-meh-o:before{content:""}.fa-gamepad:before{content:""}.fa-keyboard-o:before{content:""}.fa-flag-o:before{content:""}.fa-flag-checkered:before{content:""}.fa-terminal:before{content:""}.fa-code:before{content:""}.fa-mail-reply-all:before,.fa-reply-all:before{content:""}.fa-star-half-empty:before,.fa-star-half-full:before,.fa-star-half-o:before{content:""}.fa-location-arrow:before{content:""}.fa-crop:before{content:""}.fa-code-fork:before{content:""}.fa-chain-broken:before,.fa-unlink:before{content:""}.fa-question:before{content:""}.fa-info:before{content:""}.fa-exclamation:before{content:""}.fa-superscript:before{content:""}.fa-subscript:before{content:""}.fa-eraser:before{content:""}.fa-puzzle-piece:before{content:""}.fa-microphone:before{content:""}.fa-microphone-slash:before{content:""}.fa-shield:before{content:""}.fa-calendar-o:before{content:""}.fa-fire-extinguisher:before{content:""}.fa-rocket:before{content:""}.fa-maxcdn:before{content:""}.fa-chevron-circle-left:before{content:""}.fa-chevron-circle-right:before{content:""}.fa-chevron-circle-up:before{content:""}.fa-chevron-circle-down:before{content:""}.fa-html5:before{content:""}.fa-css3:before{content:""}.fa-anchor:before{content:""}.fa-unlock-alt:before{content:""}.fa-bullseye:before{content:""}.fa-ellipsis-h:before{content:""}.fa-ellipsis-v:before{content:""}.fa-rss-square:before{content:""}.fa-play-circle:before{content:""}.fa-ticket:before{content:""}.fa-minus-square:before{content:""}.fa-minus-square-o:before,.wy-menu-vertical li.current>a button.toctree-expand:before,.wy-menu-vertical li.on a button.toctree-expand:before{content:""}.fa-level-up:before{content:""}.fa-level-down:before{content:""}.fa-check-square:before{content:""}.fa-pencil-square:before{content:""}.fa-external-link-square:before{content:""}.fa-share-square:before{content:""}.fa-compass:before{content:""}.fa-caret-square-o-down:before,.fa-toggle-down:before{content:""}.fa-caret-square-o-up:before,.fa-toggle-up:before{content:""}.fa-caret-square-o-right:before,.fa-toggle-right:before{content:""}.fa-eur:before,.fa-euro:before{content:""}.fa-gbp:before{content:""}.fa-dollar:before,.fa-usd:before{content:""}.fa-inr:before,.fa-rupee:before{content:""}.fa-cny:before,.fa-jpy:before,.fa-rmb:before,.fa-yen:before{content:""}.fa-rouble:before,.fa-rub:before,.fa-ruble:before{content:""}.fa-krw:before,.fa-won:before{content:""}.fa-bitcoin:before,.fa-btc:before{content:""}.fa-file:before{content:""}.fa-file-text:before{content:""}.fa-sort-alpha-asc:before{content:""}.fa-sort-alpha-desc:before{content:""}.fa-sort-amount-asc:before{content:""}.fa-sort-amount-desc:before{content:""}.fa-sort-numeric-asc:before{content:""}.fa-sort-numeric-desc:before{content:""}.fa-thumbs-up:before{content:""}.fa-thumbs-down:before{content:""}.fa-youtube-square:before{content:""}.fa-youtube:before{content:""}.fa-xing:before{content:""}.fa-xing-square:before{content:""}.fa-youtube-play:before{content:""}.fa-dropbox:before{content:""}.fa-stack-overflow:before{content:""}.fa-instagram:before{content:""}.fa-flickr:before{content:""}.fa-adn:before{content:""}.fa-bitbucket:before,.icon-bitbucket:before{content:""}.fa-bitbucket-square:before{content:""}.fa-tumblr:before{content:""}.fa-tumblr-square:before{content:""}.fa-long-arrow-down:before{content:""}.fa-long-arrow-up:before{content:""}.fa-long-arrow-left:before{content:""}.fa-long-arrow-right:before{content:""}.fa-apple:before{content:""}.fa-windows:before{content:""}.fa-android:before{content:""}.fa-linux:before{content:""}.fa-dribbble:before{content:""}.fa-skype:before{content:""}.fa-foursquare:before{content:""}.fa-trello:before{content:""}.fa-female:before{content:""}.fa-male:before{content:""}.fa-gittip:before,.fa-gratipay:before{content:""}.fa-sun-o:before{content:""}.fa-moon-o:before{content:""}.fa-archive:before{content:""}.fa-bug:before{content:""}.fa-vk:before{content:""}.fa-weibo:before{content:""}.fa-renren:before{content:""}.fa-pagelines:before{content:""}.fa-stack-exchange:before{content:""}.fa-arrow-circle-o-right:before{content:""}.fa-arrow-circle-o-left:before{content:""}.fa-caret-square-o-left:before,.fa-toggle-left:before{content:""}.fa-dot-circle-o:before{content:""}.fa-wheelchair:before{content:""}.fa-vimeo-square:before{content:""}.fa-try:before,.fa-turkish-lira:before{content:""}.fa-plus-square-o:before,.wy-menu-vertical li button.toctree-expand:before{content:""}.fa-space-shuttle:before{content:""}.fa-slack:before{content:""}.fa-envelope-square:before{content:""}.fa-wordpress:before{content:""}.fa-openid:before{content:""}.fa-bank:before,.fa-institution:before,.fa-university:before{content:""}.fa-graduation-cap:before,.fa-mortar-board:before{content:""}.fa-yahoo:before{content:""}.fa-google:before{content:""}.fa-reddit:before{content:""}.fa-reddit-square:before{content:""}.fa-stumbleupon-circle:before{content:""}.fa-stumbleupon:before{content:""}.fa-delicious:before{content:""}.fa-digg:before{content:""}.fa-pied-piper-pp:before{content:""}.fa-pied-piper-alt:before{content:""}.fa-drupal:before{content:""}.fa-joomla:before{content:""}.fa-language:before{content:""}.fa-fax:before{content:""}.fa-building:before{content:""}.fa-child:before{content:""}.fa-paw:before{content:""}.fa-spoon:before{content:""}.fa-cube:before{content:""}.fa-cubes:before{content:""}.fa-behance:before{content:""}.fa-behance-square:before{content:""}.fa-steam:before{content:""}.fa-steam-square:before{content:""}.fa-recycle:before{content:""}.fa-automobile:before,.fa-car:before{content:""}.fa-cab:before,.fa-taxi:before{content:""}.fa-tree:before{content:""}.fa-spotify:before{content:""}.fa-deviantart:before{content:""}.fa-soundcloud:before{content:""}.fa-database:before{content:""}.fa-file-pdf-o:before{content:""}.fa-file-word-o:before{content:""}.fa-file-excel-o:before{content:""}.fa-file-powerpoint-o:before{content:""}.fa-file-image-o:before,.fa-file-photo-o:before,.fa-file-picture-o:before{content:""}.fa-file-archive-o:before,.fa-file-zip-o:before{content:""}.fa-file-audio-o:before,.fa-file-sound-o:before{content:""}.fa-file-movie-o:before,.fa-file-video-o:before{content:""}.fa-file-code-o:before{content:""}.fa-vine:before{content:""}.fa-codepen:before{content:""}.fa-jsfiddle:before{content:""}.fa-life-bouy:before,.fa-life-buoy:before,.fa-life-ring:before,.fa-life-saver:before,.fa-support:before{content:""}.fa-circle-o-notch:before{content:""}.fa-ra:before,.fa-rebel:before,.fa-resistance:before{content:""}.fa-empire:before,.fa-ge:before{content:""}.fa-git-square:before{content:""}.fa-git:before{content:""}.fa-hacker-news:before,.fa-y-combinator-square:before,.fa-yc-square:before{content:""}.fa-tencent-weibo:before{content:""}.fa-qq:before{content:""}.fa-wechat:before,.fa-weixin:before{content:""}.fa-paper-plane:before,.fa-send:before{content:""}.fa-paper-plane-o:before,.fa-send-o:before{content:""}.fa-history:before{content:""}.fa-circle-thin:before{content:""}.fa-header:before{content:""}.fa-paragraph:before{content:""}.fa-sliders:before{content:""}.fa-share-alt:before{content:""}.fa-share-alt-square:before{content:""}.fa-bomb:before{content:""}.fa-futbol-o:before,.fa-soccer-ball-o:before{content:""}.fa-tty:before{content:""}.fa-binoculars:before{content:""}.fa-plug:before{content:""}.fa-slideshare:before{content:""}.fa-twitch:before{content:""}.fa-yelp:before{content:""}.fa-newspaper-o:before{content:""}.fa-wifi:before{content:""}.fa-calculator:before{content:""}.fa-paypal:before{content:""}.fa-google-wallet:before{content:""}.fa-cc-visa:before{content:""}.fa-cc-mastercard:before{content:""}.fa-cc-discover:before{content:""}.fa-cc-amex:before{content:""}.fa-cc-paypal:before{content:""}.fa-cc-stripe:before{content:""}.fa-bell-slash:before{content:""}.fa-bell-slash-o:before{content:""}.fa-trash:before{content:""}.fa-copyright:before{content:""}.fa-at:before{content:""}.fa-eyedropper:before{content:""}.fa-paint-brush:before{content:""}.fa-birthday-cake:before{content:""}.fa-area-chart:before{content:""}.fa-pie-chart:before{content:""}.fa-line-chart:before{content:""}.fa-lastfm:before{content:""}.fa-lastfm-square:before{content:""}.fa-toggle-off:before{content:""}.fa-toggle-on:before{content:""}.fa-bicycle:before{content:""}.fa-bus:before{content:""}.fa-ioxhost:before{content:""}.fa-angellist:before{content:""}.fa-cc:before{content:""}.fa-ils:before,.fa-shekel:before,.fa-sheqel:before{content:""}.fa-meanpath:before{content:""}.fa-buysellads:before{content:""}.fa-connectdevelop:before{content:""}.fa-dashcube:before{content:""}.fa-forumbee:before{content:""}.fa-leanpub:before{content:""}.fa-sellsy:before{content:""}.fa-shirtsinbulk:before{content:""}.fa-simplybuilt:before{content:""}.fa-skyatlas:before{content:""}.fa-cart-plus:before{content:""}.fa-cart-arrow-down:before{content:""}.fa-diamond:before{content:""}.fa-ship:before{content:""}.fa-user-secret:before{content:""}.fa-motorcycle:before{content:""}.fa-street-view:before{content:""}.fa-heartbeat:before{content:""}.fa-venus:before{content:""}.fa-mars:before{content:""}.fa-mercury:before{content:""}.fa-intersex:before,.fa-transgender:before{content:""}.fa-transgender-alt:before{content:""}.fa-venus-double:before{content:""}.fa-mars-double:before{content:""}.fa-venus-mars:before{content:""}.fa-mars-stroke:before{content:""}.fa-mars-stroke-v:before{content:""}.fa-mars-stroke-h:before{content:""}.fa-neuter:before{content:""}.fa-genderless:before{content:""}.fa-facebook-official:before{content:""}.fa-pinterest-p:before{content:""}.fa-whatsapp:before{content:""}.fa-server:before{content:""}.fa-user-plus:before{content:""}.fa-user-times:before{content:""}.fa-bed:before,.fa-hotel:before{content:""}.fa-viacoin:before{content:""}.fa-train:before{content:""}.fa-subway:before{content:""}.fa-medium:before{content:""}.fa-y-combinator:before,.fa-yc:before{content:""}.fa-optin-monster:before{content:""}.fa-opencart:before{content:""}.fa-expeditedssl:before{content:""}.fa-battery-4:before,.fa-battery-full:before,.fa-battery:before{content:""}.fa-battery-3:before,.fa-battery-three-quarters:before{content:""}.fa-battery-2:before,.fa-battery-half:before{content:""}.fa-battery-1:before,.fa-battery-quarter:before{content:""}.fa-battery-0:before,.fa-battery-empty:before{content:""}.fa-mouse-pointer:before{content:""}.fa-i-cursor:before{content:""}.fa-object-group:before{content:""}.fa-object-ungroup:before{content:""}.fa-sticky-note:before{content:""}.fa-sticky-note-o:before{content:""}.fa-cc-jcb:before{content:""}.fa-cc-diners-club:before{content:""}.fa-clone:before{content:""}.fa-balance-scale:before{content:""}.fa-hourglass-o:before{content:""}.fa-hourglass-1:before,.fa-hourglass-start:before{content:""}.fa-hourglass-2:before,.fa-hourglass-half:before{content:""}.fa-hourglass-3:before,.fa-hourglass-end:before{content:""}.fa-hourglass:before{content:""}.fa-hand-grab-o:before,.fa-hand-rock-o:before{content:""}.fa-hand-paper-o:before,.fa-hand-stop-o:before{content:""}.fa-hand-scissors-o:before{content:""}.fa-hand-lizard-o:before{content:""}.fa-hand-spock-o:before{content:""}.fa-hand-pointer-o:before{content:""}.fa-hand-peace-o:before{content:""}.fa-trademark:before{content:""}.fa-registered:before{content:""}.fa-creative-commons:before{content:""}.fa-gg:before{content:""}.fa-gg-circle:before{content:""}.fa-tripadvisor:before{content:""}.fa-odnoklassniki:before{content:""}.fa-odnoklassniki-square:before{content:""}.fa-get-pocket:before{content:""}.fa-wikipedia-w:before{content:""}.fa-safari:before{content:""}.fa-chrome:before{content:""}.fa-firefox:before{content:""}.fa-opera:before{content:""}.fa-internet-explorer:before{content:""}.fa-television:before,.fa-tv:before{content:""}.fa-contao:before{content:""}.fa-500px:before{content:""}.fa-amazon:before{content:""}.fa-calendar-plus-o:before{content:""}.fa-calendar-minus-o:before{content:""}.fa-calendar-times-o:before{content:""}.fa-calendar-check-o:before{content:""}.fa-industry:before{content:""}.fa-map-pin:before{content:""}.fa-map-signs:before{content:""}.fa-map-o:before{content:""}.fa-map:before{content:""}.fa-commenting:before{content:""}.fa-commenting-o:before{content:""}.fa-houzz:before{content:""}.fa-vimeo:before{content:""}.fa-black-tie:before{content:""}.fa-fonticons:before{content:""}.fa-reddit-alien:before{content:""}.fa-edge:before{content:""}.fa-credit-card-alt:before{content:""}.fa-codiepie:before{content:""}.fa-modx:before{content:""}.fa-fort-awesome:before{content:""}.fa-usb:before{content:""}.fa-product-hunt:before{content:""}.fa-mixcloud:before{content:""}.fa-scribd:before{content:""}.fa-pause-circle:before{content:""}.fa-pause-circle-o:before{content:""}.fa-stop-circle:before{content:""}.fa-stop-circle-o:before{content:""}.fa-shopping-bag:before{content:""}.fa-shopping-basket:before{content:""}.fa-hashtag:before{content:""}.fa-bluetooth:before{content:""}.fa-bluetooth-b:before{content:""}.fa-percent:before{content:""}.fa-gitlab:before,.icon-gitlab:before{content:""}.fa-wpbeginner:before{content:""}.fa-wpforms:before{content:""}.fa-envira:before{content:""}.fa-universal-access:before{content:""}.fa-wheelchair-alt:before{content:""}.fa-question-circle-o:before{content:""}.fa-blind:before{content:""}.fa-audio-description:before{content:""}.fa-volume-control-phone:before{content:""}.fa-braille:before{content:""}.fa-assistive-listening-systems:before{content:""}.fa-american-sign-language-interpreting:before,.fa-asl-interpreting:before{content:""}.fa-deaf:before,.fa-deafness:before,.fa-hard-of-hearing:before{content:""}.fa-glide:before{content:""}.fa-glide-g:before{content:""}.fa-sign-language:before,.fa-signing:before{content:""}.fa-low-vision:before{content:""}.fa-viadeo:before{content:""}.fa-viadeo-square:before{content:""}.fa-snapchat:before{content:""}.fa-snapchat-ghost:before{content:""}.fa-snapchat-square:before{content:""}.fa-pied-piper:before{content:""}.fa-first-order:before{content:""}.fa-yoast:before{content:""}.fa-themeisle:before{content:""}.fa-google-plus-circle:before,.fa-google-plus-official:before{content:""}.fa-fa:before,.fa-font-awesome:before{content:""}.fa-handshake-o:before{content:""}.fa-envelope-open:before{content:""}.fa-envelope-open-o:before{content:""}.fa-linode:before{content:""}.fa-address-book:before{content:""}.fa-address-book-o:before{content:""}.fa-address-card:before,.fa-vcard:before{content:""}.fa-address-card-o:before,.fa-vcard-o:before{content:""}.fa-user-circle:before{content:""}.fa-user-circle-o:before{content:""}.fa-user-o:before{content:""}.fa-id-badge:before{content:""}.fa-drivers-license:before,.fa-id-card:before{content:""}.fa-drivers-license-o:before,.fa-id-card-o:before{content:""}.fa-quora:before{content:""}.fa-free-code-camp:before{content:""}.fa-telegram:before{content:""}.fa-thermometer-4:before,.fa-thermometer-full:before,.fa-thermometer:before{content:""}.fa-thermometer-3:before,.fa-thermometer-three-quarters:before{content:""}.fa-thermometer-2:before,.fa-thermometer-half:before{content:""}.fa-thermometer-1:before,.fa-thermometer-quarter:before{content:""}.fa-thermometer-0:before,.fa-thermometer-empty:before{content:""}.fa-shower:before{content:""}.fa-bath:before,.fa-bathtub:before,.fa-s15:before{content:""}.fa-podcast:before{content:""}.fa-window-maximize:before{content:""}.fa-window-minimize:before{content:""}.fa-window-restore:before{content:""}.fa-times-rectangle:before,.fa-window-close:before{content:""}.fa-times-rectangle-o:before,.fa-window-close-o:before{content:""}.fa-bandcamp:before{content:""}.fa-grav:before{content:""}.fa-etsy:before{content:""}.fa-imdb:before{content:""}.fa-ravelry:before{content:""}.fa-eercast:before{content:""}.fa-microchip:before{content:""}.fa-snowflake-o:before{content:""}.fa-superpowers:before{content:""}.fa-wpexplorer:before{content:""}.fa-meetup:before{content:""}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;margin:0;overflow:visible;clip:auto}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content .eqno .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content p .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-dropdown .caret,.wy-inline-validate.wy-inline-validate-danger .wy-input-context,.wy-inline-validate.wy-inline-validate-info .wy-input-context,.wy-inline-validate.wy-inline-validate-success .wy-input-context,.wy-inline-validate.wy-inline-validate-warning .wy-input-context,.wy-menu-vertical li.current>a button.toctree-expand,.wy-menu-vertical li.on a button.toctree-expand,.wy-menu-vertical li button.toctree-expand{font-family:inherit}.fa:before,.icon:before,.rst-content .admonition-title:before,.rst-content .code-block-caption .headerlink:before,.rst-content .eqno .headerlink:before,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content p .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a button.toctree-expand:before,.wy-menu-vertical li.on a button.toctree-expand:before,.wy-menu-vertical li button.toctree-expand:before{font-family:FontAwesome;display:inline-block;font-style:normal;font-weight:400;line-height:1;text-decoration:inherit}.rst-content .code-block-caption a .headerlink,.rst-content .eqno a .headerlink,.rst-content a .admonition-title,.rst-content code.download a span:first-child,.rst-content dl dt a .headerlink,.rst-content h1 a .headerlink,.rst-content h2 a .headerlink,.rst-content h3 a .headerlink,.rst-content h4 a .headerlink,.rst-content h5 a .headerlink,.rst-content h6 a .headerlink,.rst-content p.caption a .headerlink,.rst-content p a .headerlink,.rst-content table>caption a .headerlink,.rst-content tt.download a span:first-child,.wy-menu-vertical li.current>a button.toctree-expand,.wy-menu-vertical li.on a button.toctree-expand,.wy-menu-vertical li a button.toctree-expand,a .fa,a .icon,a .rst-content .admonition-title,a .rst-content .code-block-caption .headerlink,a .rst-content .eqno .headerlink,a .rst-content code.download span:first-child,a .rst-content dl dt .headerlink,a .rst-content h1 .headerlink,a .rst-content h2 .headerlink,a .rst-content h3 .headerlink,a .rst-content h4 .headerlink,a .rst-content h5 .headerlink,a .rst-content h6 .headerlink,a .rst-content p.caption .headerlink,a .rst-content p .headerlink,a .rst-content table>caption .headerlink,a .rst-content tt.download span:first-child,a .wy-menu-vertical li button.toctree-expand{display:inline-block;text-decoration:inherit}.btn .fa,.btn .icon,.btn .rst-content .admonition-title,.btn .rst-content .code-block-caption .headerlink,.btn .rst-content .eqno .headerlink,.btn .rst-content code.download span:first-child,.btn .rst-content dl dt .headerlink,.btn .rst-content h1 .headerlink,.btn .rst-content h2 .headerlink,.btn .rst-content h3 .headerlink,.btn .rst-content h4 .headerlink,.btn .rst-content h5 .headerlink,.btn .rst-content h6 .headerlink,.btn .rst-content p .headerlink,.btn .rst-content table>caption .headerlink,.btn .rst-content tt.download span:first-child,.btn .wy-menu-vertical li.current>a button.toctree-expand,.btn .wy-menu-vertical li.on a button.toctree-expand,.btn .wy-menu-vertical li button.toctree-expand,.nav .fa,.nav .icon,.nav .rst-content .admonition-title,.nav .rst-content .code-block-caption .headerlink,.nav .rst-content .eqno .headerlink,.nav .rst-content code.download span:first-child,.nav .rst-content dl dt .headerlink,.nav .rst-content h1 .headerlink,.nav .rst-content h2 .headerlink,.nav .rst-content h3 .headerlink,.nav .rst-content h4 .headerlink,.nav .rst-content h5 .headerlink,.nav .rst-content h6 .headerlink,.nav .rst-content p .headerlink,.nav .rst-content table>caption .headerlink,.nav .rst-content tt.download span:first-child,.nav .wy-menu-vertical li.current>a button.toctree-expand,.nav .wy-menu-vertical li.on a button.toctree-expand,.nav .wy-menu-vertical li button.toctree-expand,.rst-content .btn .admonition-title,.rst-content .code-block-caption .btn .headerlink,.rst-content .code-block-caption .nav .headerlink,.rst-content .eqno .btn .headerlink,.rst-content .eqno .nav .headerlink,.rst-content .nav .admonition-title,.rst-content code.download .btn span:first-child,.rst-content code.download .nav span:first-child,.rst-content dl dt .btn .headerlink,.rst-content dl dt .nav .headerlink,.rst-content h1 .btn .headerlink,.rst-content h1 .nav .headerlink,.rst-content h2 .btn .headerlink,.rst-content h2 .nav .headerlink,.rst-content h3 .btn .headerlink,.rst-content h3 .nav .headerlink,.rst-content h4 .btn .headerlink,.rst-content h4 .nav .headerlink,.rst-content h5 .btn .headerlink,.rst-content h5 .nav .headerlink,.rst-content h6 .btn .headerlink,.rst-content h6 .nav .headerlink,.rst-content p .btn .headerlink,.rst-content p .nav .headerlink,.rst-content table>caption .btn .headerlink,.rst-content table>caption .nav .headerlink,.rst-content tt.download .btn span:first-child,.rst-content tt.download .nav span:first-child,.wy-menu-vertical li .btn button.toctree-expand,.wy-menu-vertical li.current>a .btn button.toctree-expand,.wy-menu-vertical li.current>a .nav button.toctree-expand,.wy-menu-vertical li .nav button.toctree-expand,.wy-menu-vertical li.on a .btn button.toctree-expand,.wy-menu-vertical li.on a .nav button.toctree-expand{display:inline}.btn .fa-large.icon,.btn .fa.fa-large,.btn .rst-content .code-block-caption .fa-large.headerlink,.btn .rst-content .eqno .fa-large.headerlink,.btn .rst-content .fa-large.admonition-title,.btn .rst-content code.download span.fa-large:first-child,.btn .rst-content dl dt .fa-large.headerlink,.btn .rst-content h1 .fa-large.headerlink,.btn .rst-content h2 .fa-large.headerlink,.btn .rst-content h3 .fa-large.headerlink,.btn .rst-content h4 .fa-large.headerlink,.btn .rst-content h5 .fa-large.headerlink,.btn .rst-content h6 .fa-large.headerlink,.btn .rst-content p .fa-large.headerlink,.btn .rst-content table>caption .fa-large.headerlink,.btn .rst-content tt.download span.fa-large:first-child,.btn .wy-menu-vertical li button.fa-large.toctree-expand,.nav .fa-large.icon,.nav .fa.fa-large,.nav .rst-content .code-block-caption .fa-large.headerlink,.nav .rst-content .eqno .fa-large.headerlink,.nav .rst-content .fa-large.admonition-title,.nav .rst-content code.download span.fa-large:first-child,.nav .rst-content dl dt .fa-large.headerlink,.nav .rst-content h1 .fa-large.headerlink,.nav .rst-content h2 .fa-large.headerlink,.nav .rst-content h3 .fa-large.headerlink,.nav .rst-content h4 .fa-large.headerlink,.nav .rst-content h5 .fa-large.headerlink,.nav .rst-content h6 .fa-large.headerlink,.nav .rst-content p .fa-large.headerlink,.nav .rst-content table>caption .fa-large.headerlink,.nav .rst-content tt.download span.fa-large:first-child,.nav .wy-menu-vertical li button.fa-large.toctree-expand,.rst-content .btn .fa-large.admonition-title,.rst-content .code-block-caption .btn .fa-large.headerlink,.rst-content .code-block-caption .nav .fa-large.headerlink,.rst-content .eqno .btn .fa-large.headerlink,.rst-content .eqno .nav .fa-large.headerlink,.rst-content .nav .fa-large.admonition-title,.rst-content code.download .btn span.fa-large:first-child,.rst-content code.download .nav span.fa-large:first-child,.rst-content dl dt .btn .fa-large.headerlink,.rst-content dl dt .nav .fa-large.headerlink,.rst-content h1 .btn .fa-large.headerlink,.rst-content h1 .nav .fa-large.headerlink,.rst-content h2 .btn .fa-large.headerlink,.rst-content h2 .nav .fa-large.headerlink,.rst-content h3 .btn .fa-large.headerlink,.rst-content h3 .nav .fa-large.headerlink,.rst-content h4 .btn .fa-large.headerlink,.rst-content h4 .nav .fa-large.headerlink,.rst-content h5 .btn .fa-large.headerlink,.rst-content h5 .nav .fa-large.headerlink,.rst-content h6 .btn .fa-large.headerlink,.rst-content h6 .nav .fa-large.headerlink,.rst-content p .btn .fa-large.headerlink,.rst-content p .nav .fa-large.headerlink,.rst-content table>caption .btn .fa-large.headerlink,.rst-content table>caption .nav .fa-large.headerlink,.rst-content tt.download .btn span.fa-large:first-child,.rst-content tt.download .nav span.fa-large:first-child,.wy-menu-vertical li .btn button.fa-large.toctree-expand,.wy-menu-vertical li .nav button.fa-large.toctree-expand{line-height:.9em}.btn .fa-spin.icon,.btn .fa.fa-spin,.btn .rst-content .code-block-caption .fa-spin.headerlink,.btn .rst-content .eqno .fa-spin.headerlink,.btn .rst-content .fa-spin.admonition-title,.btn .rst-content code.download span.fa-spin:first-child,.btn .rst-content dl dt .fa-spin.headerlink,.btn .rst-content h1 .fa-spin.headerlink,.btn .rst-content h2 .fa-spin.headerlink,.btn .rst-content h3 .fa-spin.headerlink,.btn .rst-content h4 .fa-spin.headerlink,.btn .rst-content h5 .fa-spin.headerlink,.btn .rst-content h6 .fa-spin.headerlink,.btn .rst-content p .fa-spin.headerlink,.btn .rst-content table>caption .fa-spin.headerlink,.btn .rst-content tt.download span.fa-spin:first-child,.btn .wy-menu-vertical li button.fa-spin.toctree-expand,.nav .fa-spin.icon,.nav .fa.fa-spin,.nav .rst-content .code-block-caption .fa-spin.headerlink,.nav .rst-content .eqno .fa-spin.headerlink,.nav .rst-content .fa-spin.admonition-title,.nav .rst-content code.download span.fa-spin:first-child,.nav .rst-content dl dt .fa-spin.headerlink,.nav .rst-content h1 .fa-spin.headerlink,.nav .rst-content h2 .fa-spin.headerlink,.nav .rst-content h3 .fa-spin.headerlink,.nav .rst-content h4 .fa-spin.headerlink,.nav .rst-content h5 .fa-spin.headerlink,.nav .rst-content h6 .fa-spin.headerlink,.nav .rst-content p .fa-spin.headerlink,.nav .rst-content table>caption .fa-spin.headerlink,.nav .rst-content tt.download span.fa-spin:first-child,.nav .wy-menu-vertical li button.fa-spin.toctree-expand,.rst-content .btn .fa-spin.admonition-title,.rst-content .code-block-caption .btn .fa-spin.headerlink,.rst-content .code-block-caption .nav .fa-spin.headerlink,.rst-content .eqno .btn .fa-spin.headerlink,.rst-content .eqno .nav .fa-spin.headerlink,.rst-content .nav .fa-spin.admonition-title,.rst-content code.download .btn span.fa-spin:first-child,.rst-content code.download .nav span.fa-spin:first-child,.rst-content dl dt .btn .fa-spin.headerlink,.rst-content dl dt .nav .fa-spin.headerlink,.rst-content h1 .btn .fa-spin.headerlink,.rst-content h1 .nav .fa-spin.headerlink,.rst-content h2 .btn .fa-spin.headerlink,.rst-content h2 .nav .fa-spin.headerlink,.rst-content h3 .btn .fa-spin.headerlink,.rst-content h3 .nav .fa-spin.headerlink,.rst-content h4 .btn .fa-spin.headerlink,.rst-content h4 .nav .fa-spin.headerlink,.rst-content h5 .btn .fa-spin.headerlink,.rst-content h5 .nav .fa-spin.headerlink,.rst-content h6 .btn .fa-spin.headerlink,.rst-content h6 .nav .fa-spin.headerlink,.rst-content p .btn .fa-spin.headerlink,.rst-content p .nav .fa-spin.headerlink,.rst-content table>caption .btn .fa-spin.headerlink,.rst-content table>caption .nav .fa-spin.headerlink,.rst-content tt.download .btn span.fa-spin:first-child,.rst-content tt.download .nav span.fa-spin:first-child,.wy-menu-vertical li .btn button.fa-spin.toctree-expand,.wy-menu-vertical li .nav button.fa-spin.toctree-expand{display:inline-block}.btn.fa:before,.btn.icon:before,.rst-content .btn.admonition-title:before,.rst-content .code-block-caption .btn.headerlink:before,.rst-content .eqno .btn.headerlink:before,.rst-content code.download span.btn:first-child:before,.rst-content dl dt .btn.headerlink:before,.rst-content h1 .btn.headerlink:before,.rst-content h2 .btn.headerlink:before,.rst-content h3 .btn.headerlink:before,.rst-content h4 .btn.headerlink:before,.rst-content h5 .btn.headerlink:before,.rst-content h6 .btn.headerlink:before,.rst-content p .btn.headerlink:before,.rst-content table>caption .btn.headerlink:before,.rst-content tt.download span.btn:first-child:before,.wy-menu-vertical li button.btn.toctree-expand:before{opacity:.5;-webkit-transition:opacity .05s ease-in;-moz-transition:opacity .05s ease-in;transition:opacity .05s ease-in}.btn.fa:hover:before,.btn.icon:hover:before,.rst-content .btn.admonition-title:hover:before,.rst-content .code-block-caption .btn.headerlink:hover:before,.rst-content .eqno .btn.headerlink:hover:before,.rst-content code.download span.btn:first-child:hover:before,.rst-content dl dt .btn.headerlink:hover:before,.rst-content h1 .btn.headerlink:hover:before,.rst-content h2 .btn.headerlink:hover:before,.rst-content h3 .btn.headerlink:hover:before,.rst-content h4 .btn.headerlink:hover:before,.rst-content h5 .btn.headerlink:hover:before,.rst-content h6 .btn.headerlink:hover:before,.rst-content p .btn.headerlink:hover:before,.rst-content table>caption .btn.headerlink:hover:before,.rst-content tt.download span.btn:first-child:hover:before,.wy-menu-vertical li button.btn.toctree-expand:hover:before{opacity:1}.btn-mini .fa:before,.btn-mini .icon:before,.btn-mini .rst-content .admonition-title:before,.btn-mini .rst-content .code-block-caption .headerlink:before,.btn-mini .rst-content .eqno .headerlink:before,.btn-mini .rst-content code.download span:first-child:before,.btn-mini .rst-content dl dt .headerlink:before,.btn-mini .rst-content h1 .headerlink:before,.btn-mini .rst-content h2 .headerlink:before,.btn-mini .rst-content h3 .headerlink:before,.btn-mini .rst-content h4 .headerlink:before,.btn-mini .rst-content h5 .headerlink:before,.btn-mini .rst-content h6 .headerlink:before,.btn-mini .rst-content p .headerlink:before,.btn-mini .rst-content table>caption .headerlink:before,.btn-mini .rst-content tt.download span:first-child:before,.btn-mini .wy-menu-vertical li button.toctree-expand:before,.rst-content .btn-mini .admonition-title:before,.rst-content .code-block-caption .btn-mini .headerlink:before,.rst-content .eqno .btn-mini .headerlink:before,.rst-content code.download .btn-mini span:first-child:before,.rst-content dl dt .btn-mini .headerlink:before,.rst-content h1 .btn-mini .headerlink:before,.rst-content h2 .btn-mini .headerlink:before,.rst-content h3 .btn-mini .headerlink:before,.rst-content h4 .btn-mini .headerlink:before,.rst-content h5 .btn-mini .headerlink:before,.rst-content h6 .btn-mini .headerlink:before,.rst-content p .btn-mini .headerlink:before,.rst-content table>caption .btn-mini .headerlink:before,.rst-content tt.download .btn-mini span:first-child:before,.wy-menu-vertical li .btn-mini button.toctree-expand:before{font-size:14px;vertical-align:-15%}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.wy-alert{padding:12px;line-height:24px;margin-bottom:24px;background:#e7f2fa}.rst-content .admonition-title,.wy-alert-title{font-weight:700;display:block;color:#fff;background:#6ab0de;padding:6px 12px;margin:-12px -12px 12px}.rst-content .danger,.rst-content .error,.rst-content .wy-alert-danger.admonition,.rst-content .wy-alert-danger.admonition-todo,.rst-content .wy-alert-danger.attention,.rst-content .wy-alert-danger.caution,.rst-content .wy-alert-danger.hint,.rst-content .wy-alert-danger.important,.rst-content .wy-alert-danger.note,.rst-content .wy-alert-danger.seealso,.rst-content .wy-alert-danger.tip,.rst-content .wy-alert-danger.warning,.wy-alert.wy-alert-danger{background:#fdf3f2}.rst-content .danger .admonition-title,.rst-content .danger .wy-alert-title,.rst-content .error .admonition-title,.rst-content .error .wy-alert-title,.rst-content .wy-alert-danger.admonition-todo .admonition-title,.rst-content .wy-alert-danger.admonition-todo .wy-alert-title,.rst-content .wy-alert-danger.admonition .admonition-title,.rst-content .wy-alert-danger.admonition .wy-alert-title,.rst-content .wy-alert-danger.attention .admonition-title,.rst-content .wy-alert-danger.attention .wy-alert-title,.rst-content .wy-alert-danger.caution .admonition-title,.rst-content .wy-alert-danger.caution .wy-alert-title,.rst-content .wy-alert-danger.hint .admonition-title,.rst-content .wy-alert-danger.hint .wy-alert-title,.rst-content .wy-alert-danger.important .admonition-title,.rst-content .wy-alert-danger.important .wy-alert-title,.rst-content .wy-alert-danger.note .admonition-title,.rst-content .wy-alert-danger.note .wy-alert-title,.rst-content .wy-alert-danger.seealso .admonition-title,.rst-content .wy-alert-danger.seealso .wy-alert-title,.rst-content .wy-alert-danger.tip .admonition-title,.rst-content .wy-alert-danger.tip .wy-alert-title,.rst-content .wy-alert-danger.warning .admonition-title,.rst-content .wy-alert-danger.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-danger .admonition-title,.wy-alert.wy-alert-danger .rst-content .admonition-title,.wy-alert.wy-alert-danger .wy-alert-title{background:#f29f97}.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .warning,.rst-content .wy-alert-warning.admonition,.rst-content .wy-alert-warning.danger,.rst-content .wy-alert-warning.error,.rst-content .wy-alert-warning.hint,.rst-content .wy-alert-warning.important,.rst-content .wy-alert-warning.note,.rst-content .wy-alert-warning.seealso,.rst-content .wy-alert-warning.tip,.wy-alert.wy-alert-warning{background:#ffedcc}.rst-content .admonition-todo .admonition-title,.rst-content .admonition-todo .wy-alert-title,.rst-content .attention .admonition-title,.rst-content .attention .wy-alert-title,.rst-content .caution .admonition-title,.rst-content .caution .wy-alert-title,.rst-content .warning .admonition-title,.rst-content .warning .wy-alert-title,.rst-content .wy-alert-warning.admonition .admonition-title,.rst-content .wy-alert-warning.admonition .wy-alert-title,.rst-content .wy-alert-warning.danger .admonition-title,.rst-content .wy-alert-warning.danger .wy-alert-title,.rst-content .wy-alert-warning.error .admonition-title,.rst-content .wy-alert-warning.error .wy-alert-title,.rst-content .wy-alert-warning.hint .admonition-title,.rst-content .wy-alert-warning.hint .wy-alert-title,.rst-content .wy-alert-warning.important .admonition-title,.rst-content .wy-alert-warning.important .wy-alert-title,.rst-content .wy-alert-warning.note .admonition-title,.rst-content .wy-alert-warning.note .wy-alert-title,.rst-content .wy-alert-warning.seealso .admonition-title,.rst-content .wy-alert-warning.seealso .wy-alert-title,.rst-content .wy-alert-warning.tip .admonition-title,.rst-content .wy-alert-warning.tip .wy-alert-title,.rst-content .wy-alert.wy-alert-warning .admonition-title,.wy-alert.wy-alert-warning .rst-content .admonition-title,.wy-alert.wy-alert-warning .wy-alert-title{background:#f0b37e}.rst-content .note,.rst-content .seealso,.rst-content .wy-alert-info.admonition,.rst-content .wy-alert-info.admonition-todo,.rst-content .wy-alert-info.attention,.rst-content .wy-alert-info.caution,.rst-content .wy-alert-info.danger,.rst-content .wy-alert-info.error,.rst-content .wy-alert-info.hint,.rst-content .wy-alert-info.important,.rst-content .wy-alert-info.tip,.rst-content .wy-alert-info.warning,.wy-alert.wy-alert-info{background:#e7f2fa}.rst-content .note .admonition-title,.rst-content .note .wy-alert-title,.rst-content .seealso .admonition-title,.rst-content .seealso .wy-alert-title,.rst-content .wy-alert-info.admonition-todo .admonition-title,.rst-content .wy-alert-info.admonition-todo .wy-alert-title,.rst-content .wy-alert-info.admonition .admonition-title,.rst-content .wy-alert-info.admonition .wy-alert-title,.rst-content .wy-alert-info.attention .admonition-title,.rst-content .wy-alert-info.attention .wy-alert-title,.rst-content .wy-alert-info.caution .admonition-title,.rst-content .wy-alert-info.caution .wy-alert-title,.rst-content .wy-alert-info.danger .admonition-title,.rst-content .wy-alert-info.danger .wy-alert-title,.rst-content .wy-alert-info.error .admonition-title,.rst-content .wy-alert-info.error .wy-alert-title,.rst-content .wy-alert-info.hint .admonition-title,.rst-content .wy-alert-info.hint .wy-alert-title,.rst-content .wy-alert-info.important .admonition-title,.rst-content .wy-alert-info.important .wy-alert-title,.rst-content .wy-alert-info.tip .admonition-title,.rst-content .wy-alert-info.tip .wy-alert-title,.rst-content .wy-alert-info.warning .admonition-title,.rst-content .wy-alert-info.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-info .admonition-title,.wy-alert.wy-alert-info .rst-content .admonition-title,.wy-alert.wy-alert-info .wy-alert-title{background:#6ab0de}.rst-content .hint,.rst-content .important,.rst-content .tip,.rst-content .wy-alert-success.admonition,.rst-content .wy-alert-success.admonition-todo,.rst-content .wy-alert-success.attention,.rst-content .wy-alert-success.caution,.rst-content .wy-alert-success.danger,.rst-content .wy-alert-success.error,.rst-content .wy-alert-success.note,.rst-content .wy-alert-success.seealso,.rst-content .wy-alert-success.warning,.wy-alert.wy-alert-success{background:#dbfaf4}.rst-content .hint .admonition-title,.rst-content .hint .wy-alert-title,.rst-content .important .admonition-title,.rst-content .important .wy-alert-title,.rst-content .tip .admonition-title,.rst-content .tip .wy-alert-title,.rst-content .wy-alert-success.admonition-todo .admonition-title,.rst-content .wy-alert-success.admonition-todo .wy-alert-title,.rst-content .wy-alert-success.admonition .admonition-title,.rst-content .wy-alert-success.admonition .wy-alert-title,.rst-content .wy-alert-success.attention .admonition-title,.rst-content .wy-alert-success.attention .wy-alert-title,.rst-content .wy-alert-success.caution .admonition-title,.rst-content .wy-alert-success.caution .wy-alert-title,.rst-content .wy-alert-success.danger .admonition-title,.rst-content .wy-alert-success.danger .wy-alert-title,.rst-content .wy-alert-success.error .admonition-title,.rst-content .wy-alert-success.error .wy-alert-title,.rst-content .wy-alert-success.note .admonition-title,.rst-content .wy-alert-success.note .wy-alert-title,.rst-content .wy-alert-success.seealso .admonition-title,.rst-content .wy-alert-success.seealso .wy-alert-title,.rst-content .wy-alert-success.warning .admonition-title,.rst-content .wy-alert-success.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-success .admonition-title,.wy-alert.wy-alert-success .rst-content .admonition-title,.wy-alert.wy-alert-success .wy-alert-title{background:#1abc9c}.rst-content .wy-alert-neutral.admonition,.rst-content .wy-alert-neutral.admonition-todo,.rst-content .wy-alert-neutral.attention,.rst-content .wy-alert-neutral.caution,.rst-content .wy-alert-neutral.danger,.rst-content .wy-alert-neutral.error,.rst-content .wy-alert-neutral.hint,.rst-content .wy-alert-neutral.important,.rst-content .wy-alert-neutral.note,.rst-content .wy-alert-neutral.seealso,.rst-content .wy-alert-neutral.tip,.rst-content .wy-alert-neutral.warning,.wy-alert.wy-alert-neutral{background:#f3f6f6}.rst-content .wy-alert-neutral.admonition-todo .admonition-title,.rst-content .wy-alert-neutral.admonition-todo .wy-alert-title,.rst-content .wy-alert-neutral.admonition .admonition-title,.rst-content .wy-alert-neutral.admonition .wy-alert-title,.rst-content .wy-alert-neutral.attention .admonition-title,.rst-content .wy-alert-neutral.attention .wy-alert-title,.rst-content .wy-alert-neutral.caution .admonition-title,.rst-content .wy-alert-neutral.caution .wy-alert-title,.rst-content .wy-alert-neutral.danger .admonition-title,.rst-content .wy-alert-neutral.danger .wy-alert-title,.rst-content .wy-alert-neutral.error .admonition-title,.rst-content .wy-alert-neutral.error .wy-alert-title,.rst-content .wy-alert-neutral.hint .admonition-title,.rst-content .wy-alert-neutral.hint .wy-alert-title,.rst-content .wy-alert-neutral.important .admonition-title,.rst-content .wy-alert-neutral.important .wy-alert-title,.rst-content .wy-alert-neutral.note .admonition-title,.rst-content .wy-alert-neutral.note .wy-alert-title,.rst-content .wy-alert-neutral.seealso .admonition-title,.rst-content .wy-alert-neutral.seealso .wy-alert-title,.rst-content .wy-alert-neutral.tip .admonition-title,.rst-content .wy-alert-neutral.tip .wy-alert-title,.rst-content .wy-alert-neutral.warning .admonition-title,.rst-content .wy-alert-neutral.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-neutral .admonition-title,.wy-alert.wy-alert-neutral .rst-content .admonition-title,.wy-alert.wy-alert-neutral .wy-alert-title{color:#404040;background:#e1e4e5}.rst-content .wy-alert-neutral.admonition-todo a,.rst-content .wy-alert-neutral.admonition a,.rst-content .wy-alert-neutral.attention a,.rst-content .wy-alert-neutral.caution a,.rst-content .wy-alert-neutral.danger a,.rst-content .wy-alert-neutral.error a,.rst-content .wy-alert-neutral.hint a,.rst-content .wy-alert-neutral.important a,.rst-content .wy-alert-neutral.note a,.rst-content .wy-alert-neutral.seealso a,.rst-content .wy-alert-neutral.tip a,.rst-content .wy-alert-neutral.warning a,.wy-alert.wy-alert-neutral a{color:#2980b9}.rst-content .admonition-todo p:last-child,.rst-content .admonition p:last-child,.rst-content .attention p:last-child,.rst-content .caution p:last-child,.rst-content .danger p:last-child,.rst-content .error p:last-child,.rst-content .hint p:last-child,.rst-content .important p:last-child,.rst-content .note p:last-child,.rst-content .seealso p:last-child,.rst-content .tip p:last-child,.rst-content .warning p:last-child,.wy-alert p:last-child{margin-bottom:0}.wy-tray-container{position:fixed;bottom:0;left:0;z-index:600}.wy-tray-container li{display:block;width:300px;background:transparent;color:#fff;text-align:center;box-shadow:0 5px 5px 0 rgba(0,0,0,.1);padding:0 24px;min-width:20%;opacity:0;height:0;line-height:56px;overflow:hidden;-webkit-transition:all .3s ease-in;-moz-transition:all .3s ease-in;transition:all .3s ease-in}.wy-tray-container li.wy-tray-item-success{background:#27ae60}.wy-tray-container li.wy-tray-item-info{background:#2980b9}.wy-tray-container li.wy-tray-item-warning{background:#e67e22}.wy-tray-container li.wy-tray-item-danger{background:#e74c3c}.wy-tray-container li.on{opacity:1;height:56px}@media screen and (max-width:768px){.wy-tray-container{bottom:auto;top:0;width:100%}.wy-tray-container li{width:100%}}button{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle;cursor:pointer;line-height:normal;-webkit-appearance:button;*overflow:visible}button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}button[disabled]{cursor:default}.btn{display:inline-block;border-radius:2px;line-height:normal;white-space:nowrap;text-align:center;cursor:pointer;font-size:100%;padding:6px 12px 8px;color:#fff;border:1px solid rgba(0,0,0,.1);background-color:#27ae60;text-decoration:none;font-weight:400;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 2px -1px hsla(0,0%,100%,.5),inset 0 -2px 0 0 rgba(0,0,0,.1);outline-none:false;vertical-align:middle;*display:inline;zoom:1;-webkit-user-drag:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transition:all .1s linear;-moz-transition:all .1s linear;transition:all .1s linear}.btn-hover{background:#2e8ece;color:#fff}.btn:hover{background:#2cc36b;color:#fff}.btn:focus{background:#2cc36b;outline:0}.btn:active{box-shadow:inset 0 -1px 0 0 rgba(0,0,0,.05),inset 0 2px 0 0 rgba(0,0,0,.1);padding:8px 12px 6px}.btn:visited{color:#fff}.btn-disabled,.btn-disabled:active,.btn-disabled:focus,.btn-disabled:hover,.btn:disabled{background-image:none;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);filter:alpha(opacity=40);opacity:.4;cursor:not-allowed;box-shadow:none}.btn::-moz-focus-inner{padding:0;border:0}.btn-small{font-size:80%}.btn-info{background-color:#2980b9!important}.btn-info:hover{background-color:#2e8ece!important}.btn-neutral{background-color:#f3f6f6!important;color:#404040!important}.btn-neutral:hover{background-color:#e5ebeb!important;color:#404040}.btn-neutral:visited{color:#404040!important}.btn-success{background-color:#27ae60!important}.btn-success:hover{background-color:#295!important}.btn-danger{background-color:#e74c3c!important}.btn-danger:hover{background-color:#ea6153!important}.btn-warning{background-color:#e67e22!important}.btn-warning:hover{background-color:#e98b39!important}.btn-invert{background-color:#222}.btn-invert:hover{background-color:#2f2f2f!important}.btn-link{background-color:transparent!important;color:#2980b9;box-shadow:none;border-color:transparent!important}.btn-link:active,.btn-link:hover{background-color:transparent!important;color:#409ad5!important;box-shadow:none}.btn-link:visited{color:#9b59b6}.wy-btn-group .btn,.wy-control .btn{vertical-align:middle}.wy-btn-group{margin-bottom:24px;*zoom:1}.wy-btn-group:after,.wy-btn-group:before{display:table;content:""}.wy-btn-group:after{clear:both}.wy-dropdown{position:relative;display:inline-block}.wy-dropdown-active .wy-dropdown-menu{display:block}.wy-dropdown-menu{position:absolute;left:0;display:none;float:left;top:100%;min-width:100%;background:#fcfcfc;z-index:100;border:1px solid #cfd7dd;box-shadow:0 2px 2px 0 rgba(0,0,0,.1);padding:12px}.wy-dropdown-menu>dd>a{display:block;clear:both;color:#404040;white-space:nowrap;font-size:90%;padding:0 12px;cursor:pointer}.wy-dropdown-menu>dd>a:hover{background:#2980b9;color:#fff}.wy-dropdown-menu>dd.divider{border-top:1px solid #cfd7dd;margin:6px 0}.wy-dropdown-menu>dd.search{padding-bottom:12px}.wy-dropdown-menu>dd.search input[type=search]{width:100%}.wy-dropdown-menu>dd.call-to-action{background:#e3e3e3;text-transform:uppercase;font-weight:500;font-size:80%}.wy-dropdown-menu>dd.call-to-action:hover{background:#e3e3e3}.wy-dropdown-menu>dd.call-to-action .btn{color:#fff}.wy-dropdown.wy-dropdown-up .wy-dropdown-menu{bottom:100%;top:auto;left:auto;right:0}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu{background:#fcfcfc;margin-top:2px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a{padding:6px 12px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a:hover{background:#2980b9;color:#fff}.wy-dropdown.wy-dropdown-left .wy-dropdown-menu{right:0;left:auto;text-align:right}.wy-dropdown-arrow:before{content:" ";border-bottom:5px solid #f5f5f5;border-left:5px solid transparent;border-right:5px solid transparent;position:absolute;display:block;top:-4px;left:50%;margin-left:-3px}.wy-dropdown-arrow.wy-dropdown-arrow-left:before{left:11px}.wy-form-stacked select{display:block}.wy-form-aligned .wy-help-inline,.wy-form-aligned input,.wy-form-aligned label,.wy-form-aligned select,.wy-form-aligned textarea{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-form-aligned .wy-control-group>label{display:inline-block;vertical-align:middle;width:10em;margin:6px 12px 0 0;float:left}.wy-form-aligned .wy-control{float:left}.wy-form-aligned .wy-control label{display:block}.wy-form-aligned .wy-control select{margin-top:6px}fieldset{margin:0}fieldset,legend{border:0;padding:0}legend{width:100%;white-space:normal;margin-bottom:24px;font-size:150%;*margin-left:-7px}label,legend{display:block}label{margin:0 0 .3125em;color:#333;font-size:90%}input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}.wy-control-group{margin-bottom:24px;max-width:1200px;margin-left:auto;margin-right:auto;*zoom:1}.wy-control-group:after,.wy-control-group:before{display:table;content:""}.wy-control-group:after{clear:both}.wy-control-group.wy-control-group-required>label:after{content:" *";color:#e74c3c}.wy-control-group .wy-form-full,.wy-control-group .wy-form-halves,.wy-control-group .wy-form-thirds{padding-bottom:12px}.wy-control-group .wy-form-full input[type=color],.wy-control-group .wy-form-full input[type=date],.wy-control-group .wy-form-full input[type=datetime-local],.wy-control-group .wy-form-full input[type=datetime],.wy-control-group .wy-form-full input[type=email],.wy-control-group .wy-form-full input[type=month],.wy-control-group .wy-form-full input[type=number],.wy-control-group .wy-form-full input[type=password],.wy-control-group .wy-form-full input[type=search],.wy-control-group .wy-form-full input[type=tel],.wy-control-group .wy-form-full input[type=text],.wy-control-group .wy-form-full input[type=time],.wy-control-group .wy-form-full input[type=url],.wy-control-group .wy-form-full input[type=week],.wy-control-group .wy-form-full select,.wy-control-group .wy-form-halves input[type=color],.wy-control-group .wy-form-halves input[type=date],.wy-control-group .wy-form-halves input[type=datetime-local],.wy-control-group .wy-form-halves input[type=datetime],.wy-control-group .wy-form-halves input[type=email],.wy-control-group .wy-form-halves input[type=month],.wy-control-group .wy-form-halves input[type=number],.wy-control-group .wy-form-halves input[type=password],.wy-control-group .wy-form-halves input[type=search],.wy-control-group .wy-form-halves input[type=tel],.wy-control-group .wy-form-halves input[type=text],.wy-control-group .wy-form-halves input[type=time],.wy-control-group .wy-form-halves input[type=url],.wy-control-group .wy-form-halves input[type=week],.wy-control-group .wy-form-halves select,.wy-control-group .wy-form-thirds input[type=color],.wy-control-group .wy-form-thirds input[type=date],.wy-control-group .wy-form-thirds input[type=datetime-local],.wy-control-group .wy-form-thirds input[type=datetime],.wy-control-group .wy-form-thirds input[type=email],.wy-control-group .wy-form-thirds input[type=month],.wy-control-group .wy-form-thirds input[type=number],.wy-control-group .wy-form-thirds input[type=password],.wy-control-group .wy-form-thirds input[type=search],.wy-control-group .wy-form-thirds input[type=tel],.wy-control-group .wy-form-thirds input[type=text],.wy-control-group .wy-form-thirds input[type=time],.wy-control-group .wy-form-thirds input[type=url],.wy-control-group .wy-form-thirds input[type=week],.wy-control-group .wy-form-thirds select{width:100%}.wy-control-group .wy-form-full{float:left;display:block;width:100%;margin-right:0}.wy-control-group .wy-form-full:last-child{margin-right:0}.wy-control-group .wy-form-halves{float:left;display:block;margin-right:2.35765%;width:48.82117%}.wy-control-group .wy-form-halves:last-child,.wy-control-group .wy-form-halves:nth-of-type(2n){margin-right:0}.wy-control-group .wy-form-halves:nth-of-type(odd){clear:left}.wy-control-group .wy-form-thirds{float:left;display:block;margin-right:2.35765%;width:31.76157%}.wy-control-group .wy-form-thirds:last-child,.wy-control-group .wy-form-thirds:nth-of-type(3n){margin-right:0}.wy-control-group .wy-form-thirds:nth-of-type(3n+1){clear:left}.wy-control-group.wy-control-group-no-input .wy-control,.wy-control-no-input{margin:6px 0 0;font-size:90%}.wy-control-no-input{display:inline-block}.wy-control-group.fluid-input input[type=color],.wy-control-group.fluid-input input[type=date],.wy-control-group.fluid-input input[type=datetime-local],.wy-control-group.fluid-input input[type=datetime],.wy-control-group.fluid-input input[type=email],.wy-control-group.fluid-input input[type=month],.wy-control-group.fluid-input input[type=number],.wy-control-group.fluid-input input[type=password],.wy-control-group.fluid-input input[type=search],.wy-control-group.fluid-input input[type=tel],.wy-control-group.fluid-input input[type=text],.wy-control-group.fluid-input input[type=time],.wy-control-group.fluid-input input[type=url],.wy-control-group.fluid-input input[type=week]{width:100%}.wy-form-message-inline{padding-left:.3em;color:#666;font-size:90%}.wy-form-message{display:block;color:#999;font-size:70%;margin-top:.3125em;font-style:italic}.wy-form-message p{font-size:inherit;font-style:italic;margin-bottom:6px}.wy-form-message p:last-child{margin-bottom:0}input{line-height:normal}input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;*overflow:visible}input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week]{-webkit-appearance:none;padding:6px;display:inline-block;border:1px solid #ccc;font-size:80%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 3px #ddd;border-radius:0;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}input[type=datetime-local]{padding:.34375em .625em}input[disabled]{cursor:default}input[type=checkbox],input[type=radio]{padding:0;margin-right:.3125em;*height:13px;*width:13px}input[type=checkbox],input[type=radio],input[type=search]{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}input[type=search]::-webkit-search-cancel-button,input[type=search]::-webkit-search-decoration{-webkit-appearance:none}input[type=color]:focus,input[type=date]:focus,input[type=datetime-local]:focus,input[type=datetime]:focus,input[type=email]:focus,input[type=month]:focus,input[type=number]:focus,input[type=password]:focus,input[type=search]:focus,input[type=tel]:focus,input[type=text]:focus,input[type=time]:focus,input[type=url]:focus,input[type=week]:focus{outline:0;outline:thin dotted\9;border-color:#333}input.no-focus:focus{border-color:#ccc!important}input[type=checkbox]:focus,input[type=file]:focus,input[type=radio]:focus{outline:thin dotted #333;outline:1px auto #129fea}input[type=color][disabled],input[type=date][disabled],input[type=datetime-local][disabled],input[type=datetime][disabled],input[type=email][disabled],input[type=month][disabled],input[type=number][disabled],input[type=password][disabled],input[type=search][disabled],input[type=tel][disabled],input[type=text][disabled],input[type=time][disabled],input[type=url][disabled],input[type=week][disabled]{cursor:not-allowed;background-color:#fafafa}input:focus:invalid,select:focus:invalid,textarea:focus:invalid{color:#e74c3c;border:1px solid #e74c3c}input:focus:invalid:focus,select:focus:invalid:focus,textarea:focus:invalid:focus{border-color:#e74c3c}input[type=checkbox]:focus:invalid:focus,input[type=file]:focus:invalid:focus,input[type=radio]:focus:invalid:focus{outline-color:#e74c3c}input.wy-input-large{padding:12px;font-size:100%}textarea{overflow:auto;vertical-align:top;width:100%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif}select,textarea{padding:.5em .625em;display:inline-block;border:1px solid #ccc;font-size:80%;box-shadow:inset 0 1px 3px #ddd;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}select{border:1px solid #ccc;background-color:#fff}select[multiple]{height:auto}select:focus,textarea:focus{outline:0}input[readonly],select[disabled],select[readonly],textarea[disabled],textarea[readonly]{cursor:not-allowed;background-color:#fafafa}input[type=checkbox][disabled],input[type=radio][disabled]{cursor:not-allowed}.wy-checkbox,.wy-radio{margin:6px 0;color:#404040;display:block}.wy-checkbox input,.wy-radio input{vertical-align:baseline}.wy-form-message-inline{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-input-prefix,.wy-input-suffix{white-space:nowrap;padding:6px}.wy-input-prefix .wy-input-context,.wy-input-suffix .wy-input-context{line-height:27px;padding:0 8px;display:inline-block;font-size:80%;background-color:#f3f6f6;border:1px solid #ccc;color:#999}.wy-input-suffix .wy-input-context{border-left:0}.wy-input-prefix .wy-input-context{border-right:0}.wy-switch{position:relative;display:block;height:24px;margin-top:12px;cursor:pointer}.wy-switch:before{left:0;top:0;width:36px;height:12px;background:#ccc}.wy-switch:after,.wy-switch:before{position:absolute;content:"";display:block;border-radius:4px;-webkit-transition:all .2s ease-in-out;-moz-transition:all .2s ease-in-out;transition:all .2s ease-in-out}.wy-switch:after{width:18px;height:18px;background:#999;left:-3px;top:-3px}.wy-switch span{position:absolute;left:48px;display:block;font-size:12px;color:#ccc;line-height:1}.wy-switch.active:before{background:#1e8449}.wy-switch.active:after{left:24px;background:#27ae60}.wy-switch.disabled{cursor:not-allowed;opacity:.8}.wy-control-group.wy-control-group-error .wy-form-message,.wy-control-group.wy-control-group-error>label{color:#e74c3c}.wy-control-group.wy-control-group-error input[type=color],.wy-control-group.wy-control-group-error input[type=date],.wy-control-group.wy-control-group-error input[type=datetime-local],.wy-control-group.wy-control-group-error input[type=datetime],.wy-control-group.wy-control-group-error input[type=email],.wy-control-group.wy-control-group-error input[type=month],.wy-control-group.wy-control-group-error input[type=number],.wy-control-group.wy-control-group-error input[type=password],.wy-control-group.wy-control-group-error input[type=search],.wy-control-group.wy-control-group-error input[type=tel],.wy-control-group.wy-control-group-error input[type=text],.wy-control-group.wy-control-group-error input[type=time],.wy-control-group.wy-control-group-error input[type=url],.wy-control-group.wy-control-group-error input[type=week],.wy-control-group.wy-control-group-error textarea{border:1px solid #e74c3c}.wy-inline-validate{white-space:nowrap}.wy-inline-validate .wy-input-context{padding:.5em .625em;display:inline-block;font-size:80%}.wy-inline-validate.wy-inline-validate-success .wy-input-context{color:#27ae60}.wy-inline-validate.wy-inline-validate-danger .wy-input-context{color:#e74c3c}.wy-inline-validate.wy-inline-validate-warning .wy-input-context{color:#e67e22}.wy-inline-validate.wy-inline-validate-info .wy-input-context{color:#2980b9}.rotate-90{-webkit-transform:rotate(90deg);-moz-transform:rotate(90deg);-ms-transform:rotate(90deg);-o-transform:rotate(90deg);transform:rotate(90deg)}.rotate-180{-webkit-transform:rotate(180deg);-moz-transform:rotate(180deg);-ms-transform:rotate(180deg);-o-transform:rotate(180deg);transform:rotate(180deg)}.rotate-270{-webkit-transform:rotate(270deg);-moz-transform:rotate(270deg);-ms-transform:rotate(270deg);-o-transform:rotate(270deg);transform:rotate(270deg)}.mirror{-webkit-transform:scaleX(-1);-moz-transform:scaleX(-1);-ms-transform:scaleX(-1);-o-transform:scaleX(-1);transform:scaleX(-1)}.mirror.rotate-90{-webkit-transform:scaleX(-1) rotate(90deg);-moz-transform:scaleX(-1) rotate(90deg);-ms-transform:scaleX(-1) rotate(90deg);-o-transform:scaleX(-1) rotate(90deg);transform:scaleX(-1) rotate(90deg)}.mirror.rotate-180{-webkit-transform:scaleX(-1) rotate(180deg);-moz-transform:scaleX(-1) rotate(180deg);-ms-transform:scaleX(-1) rotate(180deg);-o-transform:scaleX(-1) rotate(180deg);transform:scaleX(-1) rotate(180deg)}.mirror.rotate-270{-webkit-transform:scaleX(-1) rotate(270deg);-moz-transform:scaleX(-1) rotate(270deg);-ms-transform:scaleX(-1) rotate(270deg);-o-transform:scaleX(-1) rotate(270deg);transform:scaleX(-1) rotate(270deg)}@media only screen and (max-width:480px){.wy-form button[type=submit]{margin:.7em 0 0}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=text],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week],.wy-form label{margin-bottom:.3em;display:block}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week]{margin-bottom:0}.wy-form-aligned .wy-control-group label{margin-bottom:.3em;text-align:left;display:block;width:100%}.wy-form-aligned .wy-control{margin:1.5em 0 0}.wy-form-message,.wy-form-message-inline,.wy-form .wy-help-inline{display:block;font-size:80%;padding:6px 0}}@media screen and (max-width:768px){.tablet-hide{display:none}}@media screen and (max-width:480px){.mobile-hide{display:none}}.float-left{float:left}.float-right{float:right}.full-width{width:100%}.rst-content table.docutils,.rst-content table.field-list,.wy-table{border-collapse:collapse;border-spacing:0;empty-cells:show;margin-bottom:24px}.rst-content table.docutils caption,.rst-content table.field-list caption,.wy-table caption{color:#000;font:italic 85%/1 arial,sans-serif;padding:1em 0;text-align:center}.rst-content table.docutils td,.rst-content table.docutils th,.rst-content table.field-list td,.rst-content table.field-list th,.wy-table td,.wy-table th{font-size:90%;margin:0;overflow:visible;padding:8px 16px}.rst-content table.docutils td:first-child,.rst-content table.docutils th:first-child,.rst-content table.field-list td:first-child,.rst-content table.field-list th:first-child,.wy-table td:first-child,.wy-table th:first-child{border-left-width:0}.rst-content table.docutils thead,.rst-content table.field-list thead,.wy-table thead{color:#000;text-align:left;vertical-align:bottom;white-space:nowrap}.rst-content table.docutils thead th,.rst-content table.field-list thead th,.wy-table thead th{font-weight:700;border-bottom:2px solid #e1e4e5}.rst-content table.docutils td,.rst-content table.field-list td,.wy-table td{background-color:transparent;vertical-align:middle}.rst-content table.docutils td p,.rst-content table.field-list td p,.wy-table td p{line-height:18px}.rst-content table.docutils td p:last-child,.rst-content table.field-list td p:last-child,.wy-table td p:last-child{margin-bottom:0}.rst-content table.docutils .wy-table-cell-min,.rst-content table.field-list .wy-table-cell-min,.wy-table .wy-table-cell-min{width:1%;padding-right:0}.rst-content table.docutils .wy-table-cell-min input[type=checkbox],.rst-content table.field-list .wy-table-cell-min input[type=checkbox],.wy-table .wy-table-cell-min input[type=checkbox]{margin:0}.wy-table-secondary{color:grey;font-size:90%}.wy-table-tertiary{color:grey;font-size:80%}.rst-content table.docutils:not(.field-list) tr:nth-child(2n-1) td,.wy-table-backed,.wy-table-odd td,.wy-table-striped tr:nth-child(2n-1) td{background-color:#f3f6f6}.rst-content table.docutils,.wy-table-bordered-all{border:1px solid #e1e4e5}.rst-content table.docutils td,.wy-table-bordered-all td{border-bottom:1px solid #e1e4e5;border-left:1px solid #e1e4e5}.rst-content table.docutils tbody>tr:last-child td,.wy-table-bordered-all tbody>tr:last-child td{border-bottom-width:0}.wy-table-bordered{border:1px solid #e1e4e5}.wy-table-bordered-rows td{border-bottom:1px solid #e1e4e5}.wy-table-bordered-rows tbody>tr:last-child td{border-bottom-width:0}.wy-table-horizontal td,.wy-table-horizontal th{border-width:0 0 1px;border-bottom:1px solid #e1e4e5}.wy-table-horizontal tbody>tr:last-child td{border-bottom-width:0}.wy-table-responsive{margin-bottom:24px;max-width:100%;overflow:auto}.wy-table-responsive table{margin-bottom:0!important}.wy-table-responsive table td,.wy-table-responsive table th{white-space:nowrap}a{color:#2980b9;text-decoration:none;cursor:pointer}a:hover{color:#3091d1}a:visited{color:#9b59b6}html{height:100%}body,html{overflow-x:hidden}body{font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;font-weight:400;color:#404040;min-height:100%;background:#edf0f2}.wy-text-left{text-align:left}.wy-text-center{text-align:center}.wy-text-right{text-align:right}.wy-text-large{font-size:120%}.wy-text-normal{font-size:100%}.wy-text-small,small{font-size:80%}.wy-text-strike{text-decoration:line-through}.wy-text-warning{color:#e67e22!important}a.wy-text-warning:hover{color:#eb9950!important}.wy-text-info{color:#2980b9!important}a.wy-text-info:hover{color:#409ad5!important}.wy-text-success{color:#27ae60!important}a.wy-text-success:hover{color:#36d278!important}.wy-text-danger{color:#e74c3c!important}a.wy-text-danger:hover{color:#ed7669!important}.wy-text-neutral{color:#404040!important}a.wy-text-neutral:hover{color:#595959!important}.rst-content .toctree-wrapper>p.caption,h1,h2,h3,h4,h5,h6,legend{margin-top:0;font-weight:700;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif}p{line-height:24px;font-size:16px;margin:0 0 24px}h1{font-size:175%}.rst-content .toctree-wrapper>p.caption,h2{font-size:150%}h3{font-size:125%}h4{font-size:115%}h5{font-size:110%}h6{font-size:100%}hr{display:block;height:1px;border:0;border-top:1px solid #e1e4e5;margin:24px 0;padding:0}.rst-content code,.rst-content tt,code{white-space:nowrap;max-width:100%;background:#fff;border:1px solid #e1e4e5;font-size:75%;padding:0 5px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;color:#e74c3c;overflow-x:auto}.rst-content tt.code-large,code.code-large{font-size:90%}.rst-content .section ul,.rst-content .toctree-wrapper ul,.rst-content section ul,.wy-plain-list-disc,article ul{list-style:disc;line-height:24px;margin-bottom:24px}.rst-content .section ul li,.rst-content .toctree-wrapper ul li,.rst-content section ul li,.wy-plain-list-disc li,article ul li{list-style:disc;margin-left:24px}.rst-content .section ul li p:last-child,.rst-content .section ul li ul,.rst-content .toctree-wrapper ul li p:last-child,.rst-content .toctree-wrapper ul li ul,.rst-content section ul li p:last-child,.rst-content section ul li ul,.wy-plain-list-disc li p:last-child,.wy-plain-list-disc li ul,article ul li p:last-child,article ul li ul{margin-bottom:0}.rst-content .section ul li li,.rst-content .toctree-wrapper ul li li,.rst-content section ul li li,.wy-plain-list-disc li li,article ul li li{list-style:circle}.rst-content .section ul li li li,.rst-content .toctree-wrapper ul li li li,.rst-content section ul li li li,.wy-plain-list-disc li li li,article ul li li li{list-style:square}.rst-content .section ul li ol li,.rst-content .toctree-wrapper ul li ol li,.rst-content section ul li ol li,.wy-plain-list-disc li ol li,article ul li ol li{list-style:decimal}.rst-content .section ol,.rst-content .section ol.arabic,.rst-content .toctree-wrapper ol,.rst-content .toctree-wrapper ol.arabic,.rst-content section ol,.rst-content section ol.arabic,.wy-plain-list-decimal,article ol{list-style:decimal;line-height:24px;margin-bottom:24px}.rst-content .section ol.arabic li,.rst-content .section ol li,.rst-content .toctree-wrapper ol.arabic li,.rst-content .toctree-wrapper ol li,.rst-content section ol.arabic li,.rst-content section ol li,.wy-plain-list-decimal li,article ol li{list-style:decimal;margin-left:24px}.rst-content .section ol.arabic li ul,.rst-content .section ol li p:last-child,.rst-content .section ol li ul,.rst-content .toctree-wrapper ol.arabic li ul,.rst-content .toctree-wrapper ol li p:last-child,.rst-content .toctree-wrapper ol li ul,.rst-content section ol.arabic li ul,.rst-content section ol li p:last-child,.rst-content section ol li ul,.wy-plain-list-decimal li p:last-child,.wy-plain-list-decimal li ul,article ol li p:last-child,article ol li ul{margin-bottom:0}.rst-content .section ol.arabic li ul li,.rst-content .section ol li ul li,.rst-content .toctree-wrapper ol.arabic li ul li,.rst-content .toctree-wrapper ol li ul li,.rst-content section ol.arabic li ul li,.rst-content section ol li ul li,.wy-plain-list-decimal li ul li,article ol li ul li{list-style:disc}.wy-breadcrumbs{*zoom:1}.wy-breadcrumbs:after,.wy-breadcrumbs:before{display:table;content:""}.wy-breadcrumbs:after{clear:both}.wy-breadcrumbs>li{display:inline-block;padding-top:5px}.wy-breadcrumbs>li.wy-breadcrumbs-aside{float:right}.rst-content .wy-breadcrumbs>li code,.rst-content .wy-breadcrumbs>li tt,.wy-breadcrumbs>li .rst-content tt,.wy-breadcrumbs>li code{all:inherit;color:inherit}.breadcrumb-item:before{content:"/";color:#bbb;font-size:13px;padding:0 6px 0 3px}.wy-breadcrumbs-extra{margin-bottom:0;color:#b3b3b3;font-size:80%;display:inline-block}@media screen and (max-width:480px){.wy-breadcrumbs-extra,.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}@media print{.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}html{font-size:16px}.wy-affix{position:fixed;top:1.618em}.wy-menu a:hover{text-decoration:none}.wy-menu-horiz{*zoom:1}.wy-menu-horiz:after,.wy-menu-horiz:before{display:table;content:""}.wy-menu-horiz:after{clear:both}.wy-menu-horiz li,.wy-menu-horiz ul{display:inline-block}.wy-menu-horiz li:hover{background:hsla(0,0%,100%,.1)}.wy-menu-horiz li.divide-left{border-left:1px solid #404040}.wy-menu-horiz li.divide-right{border-right:1px solid #404040}.wy-menu-horiz a{height:32px;display:inline-block;line-height:32px;padding:0 16px}.wy-menu-vertical{width:300px}.wy-menu-vertical header,.wy-menu-vertical p.caption{color:#55a5d9;height:32px;line-height:32px;padding:0 1.618em;margin:12px 0 0;display:block;font-weight:700;text-transform:uppercase;font-size:85%;white-space:nowrap}.wy-menu-vertical ul{margin-bottom:0}.wy-menu-vertical li.divide-top{border-top:1px solid #404040}.wy-menu-vertical li.divide-bottom{border-bottom:1px solid #404040}.wy-menu-vertical li.current{background:#e3e3e3}.wy-menu-vertical li.current a{color:grey;border-right:1px solid #c9c9c9;padding:.4045em 2.427em}.wy-menu-vertical li.current a:hover{background:#d6d6d6}.rst-content .wy-menu-vertical li tt,.wy-menu-vertical li .rst-content tt,.wy-menu-vertical li code{border:none;background:inherit;color:inherit;padding-left:0;padding-right:0}.wy-menu-vertical li button.toctree-expand{display:block;float:left;margin-left:-1.2em;line-height:18px;color:#4d4d4d;border:none;background:none;padding:0}.wy-menu-vertical li.current>a,.wy-menu-vertical li.on a{color:#404040;font-weight:700;position:relative;background:#fcfcfc;border:none;padding:.4045em 1.618em}.wy-menu-vertical li.current>a:hover,.wy-menu-vertical li.on a:hover{background:#fcfcfc}.wy-menu-vertical li.current>a:hover button.toctree-expand,.wy-menu-vertical li.on a:hover button.toctree-expand{color:grey}.wy-menu-vertical li.current>a button.toctree-expand,.wy-menu-vertical li.on a button.toctree-expand{display:block;line-height:18px;color:#333}.wy-menu-vertical li.toctree-l1.current>a{border-bottom:1px solid #c9c9c9;border-top:1px solid #c9c9c9}.wy-menu-vertical .toctree-l1.current .toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .toctree-l11>ul{display:none}.wy-menu-vertical .toctree-l1.current .current.toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .current.toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .current.toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .current.toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .current.toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .current.toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .current.toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .current.toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .current.toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .current.toctree-l11>ul{display:block}.wy-menu-vertical li.toctree-l3,.wy-menu-vertical li.toctree-l4{font-size:.9em}.wy-menu-vertical li.toctree-l2 a,.wy-menu-vertical li.toctree-l3 a,.wy-menu-vertical li.toctree-l4 a,.wy-menu-vertical li.toctree-l5 a,.wy-menu-vertical li.toctree-l6 a,.wy-menu-vertical li.toctree-l7 a,.wy-menu-vertical li.toctree-l8 a,.wy-menu-vertical li.toctree-l9 a,.wy-menu-vertical li.toctree-l10 a{color:#404040}.wy-menu-vertical li.toctree-l2 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l3 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l4 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l5 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l6 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l7 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l8 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l9 a:hover button.toctree-expand,.wy-menu-vertical li.toctree-l10 a:hover button.toctree-expand{color:grey}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a,.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a,.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a,.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a,.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a,.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a,.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a,.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{display:block}.wy-menu-vertical li.toctree-l2.current>a{padding:.4045em 2.427em}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a{padding:.4045em 1.618em .4045em 4.045em}.wy-menu-vertical li.toctree-l3.current>a{padding:.4045em 4.045em}.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a{padding:.4045em 1.618em .4045em 5.663em}.wy-menu-vertical li.toctree-l4.current>a{padding:.4045em 5.663em}.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a{padding:.4045em 1.618em .4045em 7.281em}.wy-menu-vertical li.toctree-l5.current>a{padding:.4045em 7.281em}.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a{padding:.4045em 1.618em .4045em 8.899em}.wy-menu-vertical li.toctree-l6.current>a{padding:.4045em 8.899em}.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a{padding:.4045em 1.618em .4045em 10.517em}.wy-menu-vertical li.toctree-l7.current>a{padding:.4045em 10.517em}.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a{padding:.4045em 1.618em .4045em 12.135em}.wy-menu-vertical li.toctree-l8.current>a{padding:.4045em 12.135em}.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a{padding:.4045em 1.618em .4045em 13.753em}.wy-menu-vertical li.toctree-l9.current>a{padding:.4045em 13.753em}.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a{padding:.4045em 1.618em .4045em 15.371em}.wy-menu-vertical li.toctree-l10.current>a{padding:.4045em 15.371em}.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{padding:.4045em 1.618em .4045em 16.989em}.wy-menu-vertical li.toctree-l2.current>a,.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a{background:#c9c9c9}.wy-menu-vertical li.toctree-l2 button.toctree-expand{color:#a3a3a3}.wy-menu-vertical li.toctree-l3.current>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a{background:#bdbdbd}.wy-menu-vertical li.toctree-l3 button.toctree-expand{color:#969696}.wy-menu-vertical li.current ul{display:block}.wy-menu-vertical li ul{margin-bottom:0;display:none}.wy-menu-vertical li ul li a{margin-bottom:0;color:#d9d9d9;font-weight:400}.wy-menu-vertical a{line-height:18px;padding:.4045em 1.618em;display:block;position:relative;font-size:90%;color:#d9d9d9}.wy-menu-vertical a:hover{background-color:#4e4a4a;cursor:pointer}.wy-menu-vertical a:hover button.toctree-expand{color:#d9d9d9}.wy-menu-vertical a:active{background-color:#2980b9;cursor:pointer;color:#fff}.wy-menu-vertical a:active button.toctree-expand{color:#fff}.wy-side-nav-search{display:block;width:300px;padding:.809em;margin-bottom:.809em;z-index:200;background-color:#2980b9;text-align:center;color:#fcfcfc}.wy-side-nav-search input[type=text]{width:100%;border-radius:50px;padding:6px 12px;border-color:#2472a4}.wy-side-nav-search img{display:block;margin:auto auto .809em;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-side-nav-search .wy-dropdown>a,.wy-side-nav-search>a{color:#fcfcfc;font-size:100%;font-weight:700;display:inline-block;padding:4px 6px;margin-bottom:.809em;max-width:100%}.wy-side-nav-search .wy-dropdown>a:hover,.wy-side-nav-search>a:hover{background:hsla(0,0%,100%,.1)}.wy-side-nav-search .wy-dropdown>a img.logo,.wy-side-nav-search>a img.logo{display:block;margin:0 auto;height:auto;width:auto;border-radius:0;max-width:100%;background:transparent}.wy-side-nav-search .wy-dropdown>a.icon img.logo,.wy-side-nav-search>a.icon img.logo{margin-top:.85em}.wy-side-nav-search>div.version{margin-top:-.4045em;margin-bottom:.809em;font-weight:400;color:hsla(0,0%,100%,.3)}.wy-nav .wy-menu-vertical header{color:#2980b9}.wy-nav .wy-menu-vertical a{color:#b3b3b3}.wy-nav .wy-menu-vertical a:hover{background-color:#2980b9;color:#fff}[data-menu-wrap]{-webkit-transition:all .2s ease-in;-moz-transition:all .2s ease-in;transition:all .2s ease-in;position:absolute;opacity:1;width:100%;opacity:0}[data-menu-wrap].move-center{left:0;right:auto;opacity:1}[data-menu-wrap].move-left{right:auto;left:-100%;opacity:0}[data-menu-wrap].move-right{right:-100%;left:auto;opacity:0}.wy-body-for-nav{background:#fcfcfc}.wy-grid-for-nav{position:absolute;width:100%;height:100%}.wy-nav-side{position:fixed;top:0;bottom:0;left:0;padding-bottom:2em;width:300px;overflow-x:hidden;overflow-y:hidden;min-height:100%;color:#9b9b9b;background:#343131;z-index:200}.wy-side-scroll{width:320px;position:relative;overflow-x:hidden;overflow-y:scroll;height:100%}.wy-nav-top{display:none;background:#2980b9;color:#fff;padding:.4045em .809em;position:relative;line-height:50px;text-align:center;font-size:100%;*zoom:1}.wy-nav-top:after,.wy-nav-top:before{display:table;content:""}.wy-nav-top:after{clear:both}.wy-nav-top a{color:#fff;font-weight:700}.wy-nav-top img{margin-right:12px;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-nav-top i{font-size:30px;float:left;cursor:pointer;padding-top:inherit}.wy-nav-content-wrap{margin-left:300px;background:#fcfcfc;min-height:100%}.wy-nav-content{padding:1.618em 3.236em;height:100%;max-width:800px;margin:auto}.wy-body-mask{position:fixed;width:100%;height:100%;background:rgba(0,0,0,.2);display:none;z-index:499}.wy-body-mask.on{display:block}footer{color:grey}footer p{margin-bottom:12px}.rst-content footer span.commit tt,footer span.commit .rst-content tt,footer span.commit code{padding:0;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:1em;background:none;border:none;color:grey}.rst-footer-buttons{*zoom:1}.rst-footer-buttons:after,.rst-footer-buttons:before{width:100%;display:table;content:""}.rst-footer-buttons:after{clear:both}.rst-breadcrumbs-buttons{margin-top:12px;*zoom:1}.rst-breadcrumbs-buttons:after,.rst-breadcrumbs-buttons:before{display:table;content:""}.rst-breadcrumbs-buttons:after{clear:both}#search-results .search li{margin-bottom:24px;border-bottom:1px solid #e1e4e5;padding-bottom:24px}#search-results .search li:first-child{border-top:1px solid #e1e4e5;padding-top:24px}#search-results .search li a{font-size:120%;margin-bottom:12px;display:inline-block}#search-results .context{color:grey;font-size:90%}.genindextable li>ul{margin-left:24px}@media screen and (max-width:768px){.wy-body-for-nav{background:#fcfcfc}.wy-nav-top{display:block}.wy-nav-side{left:-300px}.wy-nav-side.shift{width:85%;left:0}.wy-menu.wy-menu-vertical,.wy-side-nav-search,.wy-side-scroll{width:auto}.wy-nav-content-wrap{margin-left:0}.wy-nav-content-wrap .wy-nav-content{padding:1.618em}.wy-nav-content-wrap.shift{position:fixed;min-width:100%;left:85%;top:0;height:100%;overflow:hidden}}@media screen and (min-width:1100px){.wy-nav-content-wrap{background:rgba(0,0,0,.05)}.wy-nav-content{margin:0;background:#fcfcfc}}@media print{.rst-versions,.wy-nav-side,footer{display:none}.wy-nav-content-wrap{margin-left:0}}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60;*zoom:1}.rst-versions .rst-current-version:after,.rst-versions .rst-current-version:before{display:table;content:""}.rst-versions .rst-current-version:after{clear:both}.rst-content .code-block-caption .rst-versions .rst-current-version .headerlink,.rst-content .eqno .rst-versions .rst-current-version .headerlink,.rst-content .rst-versions .rst-current-version .admonition-title,.rst-content code.download .rst-versions .rst-current-version span:first-child,.rst-content dl dt .rst-versions .rst-current-version .headerlink,.rst-content h1 .rst-versions .rst-current-version .headerlink,.rst-content h2 .rst-versions .rst-current-version .headerlink,.rst-content h3 .rst-versions .rst-current-version .headerlink,.rst-content h4 .rst-versions .rst-current-version .headerlink,.rst-content h5 .rst-versions .rst-current-version .headerlink,.rst-content h6 .rst-versions .rst-current-version .headerlink,.rst-content p .rst-versions .rst-current-version .headerlink,.rst-content table>caption .rst-versions .rst-current-version .headerlink,.rst-content tt.download .rst-versions .rst-current-version span:first-child,.rst-versions .rst-current-version .fa,.rst-versions .rst-current-version .icon,.rst-versions .rst-current-version .rst-content .admonition-title,.rst-versions .rst-current-version .rst-content .code-block-caption .headerlink,.rst-versions .rst-current-version .rst-content .eqno .headerlink,.rst-versions .rst-current-version .rst-content code.download span:first-child,.rst-versions .rst-current-version .rst-content dl dt .headerlink,.rst-versions .rst-current-version .rst-content h1 .headerlink,.rst-versions .rst-current-version .rst-content h2 .headerlink,.rst-versions .rst-current-version .rst-content h3 .headerlink,.rst-versions .rst-current-version .rst-content h4 .headerlink,.rst-versions .rst-current-version .rst-content h5 .headerlink,.rst-versions .rst-current-version .rst-content h6 .headerlink,.rst-versions .rst-current-version .rst-content p .headerlink,.rst-versions .rst-current-version .rst-content table>caption .headerlink,.rst-versions .rst-current-version .rst-content tt.download span:first-child,.rst-versions .rst-current-version .wy-menu-vertical li button.toctree-expand,.wy-menu-vertical li .rst-versions .rst-current-version button.toctree-expand{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}}.rst-content .toctree-wrapper>p.caption,.rst-content h1,.rst-content h2,.rst-content h3,.rst-content h4,.rst-content h5,.rst-content h6{margin-bottom:24px}.rst-content img{max-width:100%;height:auto}.rst-content div.figure,.rst-content figure{margin-bottom:24px}.rst-content div.figure .caption-text,.rst-content figure .caption-text{font-style:italic}.rst-content div.figure p:last-child.caption,.rst-content figure p:last-child.caption{margin-bottom:0}.rst-content div.figure.align-center,.rst-content figure.align-center{text-align:center}.rst-content .section>a>img,.rst-content .section>img,.rst-content section>a>img,.rst-content section>img{margin-bottom:24px}.rst-content abbr[title]{text-decoration:none}.rst-content.style-external-links a.reference.external:after{font-family:FontAwesome;content:"\f08e";color:#b3b3b3;vertical-align:super;font-size:60%;margin:0 .2em}.rst-content blockquote{margin-left:24px;line-height:24px;margin-bottom:24px}.rst-content pre.literal-block{white-space:pre;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;display:block;overflow:auto}.rst-content div[class^=highlight],.rst-content pre.literal-block{border:1px solid #e1e4e5;overflow-x:auto;margin:1px 0 24px}.rst-content div[class^=highlight] div[class^=highlight],.rst-content pre.literal-block div[class^=highlight]{padding:0;border:none;margin:0}.rst-content div[class^=highlight] td.code{width:100%}.rst-content .linenodiv pre{border-right:1px solid #e6e9ea;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;user-select:none;pointer-events:none}.rst-content div[class^=highlight] pre{white-space:pre;margin:0;padding:12px;display:block;overflow:auto}.rst-content div[class^=highlight] pre .hll{display:block;margin:0 -12px;padding:0 12px}.rst-content .linenodiv pre,.rst-content div[class^=highlight] pre,.rst-content pre.literal-block{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:12px;line-height:1.4}.rst-content div.highlight .gp,.rst-content div.highlight span.linenos{user-select:none;pointer-events:none}.rst-content div.highlight span.linenos{display:inline-block;padding-left:0;padding-right:12px;margin-right:12px;border-right:1px solid #e6e9ea}.rst-content .code-block-caption{font-style:italic;font-size:85%;line-height:1;padding:1em 0;text-align:center}@media print{.rst-content .codeblock,.rst-content div[class^=highlight],.rst-content div[class^=highlight] pre{white-space:pre-wrap}}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning{clear:both}.rst-content .admonition-todo .last,.rst-content .admonition-todo>:last-child,.rst-content .admonition .last,.rst-content .admonition>:last-child,.rst-content .attention .last,.rst-content .attention>:last-child,.rst-content .caution .last,.rst-content .caution>:last-child,.rst-content .danger .last,.rst-content .danger>:last-child,.rst-content .error .last,.rst-content .error>:last-child,.rst-content .hint .last,.rst-content .hint>:last-child,.rst-content .important .last,.rst-content .important>:last-child,.rst-content .note .last,.rst-content .note>:last-child,.rst-content .seealso .last,.rst-content .seealso>:last-child,.rst-content .tip .last,.rst-content .tip>:last-child,.rst-content .warning .last,.rst-content .warning>:last-child{margin-bottom:0}.rst-content .admonition-title:before{margin-right:4px}.rst-content .admonition table{border-color:rgba(0,0,0,.1)}.rst-content .admonition table td,.rst-content .admonition table th{background:transparent!important;border-color:rgba(0,0,0,.1)!important}.rst-content .section ol.loweralpha,.rst-content .section ol.loweralpha>li,.rst-content .toctree-wrapper ol.loweralpha,.rst-content .toctree-wrapper ol.loweralpha>li,.rst-content section ol.loweralpha,.rst-content section ol.loweralpha>li{list-style:lower-alpha}.rst-content .section ol.upperalpha,.rst-content .section ol.upperalpha>li,.rst-content .toctree-wrapper ol.upperalpha,.rst-content .toctree-wrapper ol.upperalpha>li,.rst-content section ol.upperalpha,.rst-content section ol.upperalpha>li{list-style:upper-alpha}.rst-content .section ol li>*,.rst-content .section ul li>*,.rst-content .toctree-wrapper ol li>*,.rst-content .toctree-wrapper ul li>*,.rst-content section ol li>*,.rst-content section ul li>*{margin-top:12px;margin-bottom:12px}.rst-content .section ol li>:first-child,.rst-content .section ul li>:first-child,.rst-content .toctree-wrapper ol li>:first-child,.rst-content .toctree-wrapper ul li>:first-child,.rst-content section ol li>:first-child,.rst-content section ul li>:first-child{margin-top:0}.rst-content .section ol li>p,.rst-content .section ol li>p:last-child,.rst-content .section ul li>p,.rst-content .section ul li>p:last-child,.rst-content .toctree-wrapper ol li>p,.rst-content .toctree-wrapper ol li>p:last-child,.rst-content .toctree-wrapper ul li>p,.rst-content .toctree-wrapper ul li>p:last-child,.rst-content section ol li>p,.rst-content section ol li>p:last-child,.rst-content section ul li>p,.rst-content section ul li>p:last-child{margin-bottom:12px}.rst-content .section ol li>p:only-child,.rst-content .section ol li>p:only-child:last-child,.rst-content .section ul li>p:only-child,.rst-content .section ul li>p:only-child:last-child,.rst-content .toctree-wrapper ol li>p:only-child,.rst-content .toctree-wrapper ol li>p:only-child:last-child,.rst-content .toctree-wrapper ul li>p:only-child,.rst-content .toctree-wrapper ul li>p:only-child:last-child,.rst-content section ol li>p:only-child,.rst-content section ol li>p:only-child:last-child,.rst-content section ul li>p:only-child,.rst-content section ul li>p:only-child:last-child{margin-bottom:0}.rst-content .section ol li>ol,.rst-content .section ol li>ul,.rst-content .section ul li>ol,.rst-content .section ul li>ul,.rst-content .toctree-wrapper ol li>ol,.rst-content .toctree-wrapper ol li>ul,.rst-content .toctree-wrapper ul li>ol,.rst-content .toctree-wrapper ul li>ul,.rst-content section ol li>ol,.rst-content section ol li>ul,.rst-content section ul li>ol,.rst-content section ul li>ul{margin-bottom:12px}.rst-content .section ol.simple li>*,.rst-content .section ol.simple li ol,.rst-content .section ol.simple li ul,.rst-content .section ul.simple li>*,.rst-content .section ul.simple li ol,.rst-content .section ul.simple li ul,.rst-content .toctree-wrapper ol.simple li>*,.rst-content .toctree-wrapper ol.simple li ol,.rst-content .toctree-wrapper ol.simple li ul,.rst-content .toctree-wrapper ul.simple li>*,.rst-content .toctree-wrapper ul.simple li ol,.rst-content .toctree-wrapper ul.simple li ul,.rst-content section ol.simple li>*,.rst-content section ol.simple li ol,.rst-content section ol.simple li ul,.rst-content section ul.simple li>*,.rst-content section ul.simple li ol,.rst-content section ul.simple li ul{margin-top:0;margin-bottom:0}.rst-content .line-block{margin-left:0;margin-bottom:24px;line-height:24px}.rst-content .line-block .line-block{margin-left:24px;margin-bottom:0}.rst-content .topic-title{font-weight:700;margin-bottom:12px}.rst-content .toc-backref{color:#404040}.rst-content .align-right{float:right;margin:0 0 24px 24px}.rst-content .align-left{float:left;margin:0 24px 24px 0}.rst-content .align-center{margin:auto}.rst-content .align-center:not(table){display:block}.rst-content .code-block-caption .headerlink,.rst-content .eqno .headerlink,.rst-content .toctree-wrapper>p.caption .headerlink,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content p .headerlink,.rst-content table>caption .headerlink{opacity:0;font-size:14px;font-family:FontAwesome;margin-left:.5em}.rst-content .code-block-caption .headerlink:focus,.rst-content .code-block-caption:hover .headerlink,.rst-content .eqno .headerlink:focus,.rst-content .eqno:hover .headerlink,.rst-content .toctree-wrapper>p.caption .headerlink:focus,.rst-content .toctree-wrapper>p.caption:hover .headerlink,.rst-content dl dt .headerlink:focus,.rst-content dl dt:hover .headerlink,.rst-content h1 .headerlink:focus,.rst-content h1:hover .headerlink,.rst-content h2 .headerlink:focus,.rst-content h2:hover .headerlink,.rst-content h3 .headerlink:focus,.rst-content h3:hover .headerlink,.rst-content h4 .headerlink:focus,.rst-content h4:hover .headerlink,.rst-content h5 .headerlink:focus,.rst-content h5:hover .headerlink,.rst-content h6 .headerlink:focus,.rst-content h6:hover .headerlink,.rst-content p.caption .headerlink:focus,.rst-content p.caption:hover .headerlink,.rst-content p .headerlink:focus,.rst-content p:hover .headerlink,.rst-content table>caption .headerlink:focus,.rst-content table>caption:hover .headerlink{opacity:1}.rst-content p a{overflow-wrap:anywhere}.rst-content .wy-table td p,.rst-content .wy-table td ul,.rst-content .wy-table th p,.rst-content .wy-table th ul,.rst-content table.docutils td p,.rst-content table.docutils td ul,.rst-content table.docutils th p,.rst-content table.docutils th ul,.rst-content table.field-list td p,.rst-content table.field-list td ul,.rst-content table.field-list th p,.rst-content table.field-list th ul{font-size:inherit}.rst-content .btn:focus{outline:2px solid}.rst-content table>caption .headerlink:after{font-size:12px}.rst-content .centered{text-align:center}.rst-content .sidebar{float:right;width:40%;display:block;margin:0 0 24px 24px;padding:24px;background:#f3f6f6;border:1px solid #e1e4e5}.rst-content .sidebar dl,.rst-content .sidebar p,.rst-content .sidebar ul{font-size:90%}.rst-content .sidebar .last,.rst-content .sidebar>:last-child{margin-bottom:0}.rst-content .sidebar .sidebar-title{display:block;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif;font-weight:700;background:#e1e4e5;padding:6px 12px;margin:-24px -24px 24px;font-size:100%}.rst-content .highlighted{background:#f1c40f;box-shadow:0 0 0 2px #f1c40f;display:inline;font-weight:700}.rst-content .citation-reference,.rst-content .footnote-reference{vertical-align:baseline;position:relative;top:-.4em;line-height:0;font-size:90%}.rst-content .citation-reference>span.fn-bracket,.rst-content .footnote-reference>span.fn-bracket{display:none}.rst-content .hlist{width:100%}.rst-content dl dt span.classifier:before{content:" : "}.rst-content dl dt span.classifier-delimiter{display:none!important}html.writer-html4 .rst-content table.docutils.citation,html.writer-html4 .rst-content table.docutils.footnote{background:none;border:none}html.writer-html4 .rst-content table.docutils.citation td,html.writer-html4 .rst-content table.docutils.citation tr,html.writer-html4 .rst-content table.docutils.footnote td,html.writer-html4 .rst-content table.docutils.footnote tr{border:none;background-color:transparent!important;white-space:normal}html.writer-html4 .rst-content table.docutils.citation td.label,html.writer-html4 .rst-content table.docutils.footnote td.label{padding-left:0;padding-right:0;vertical-align:top}html.writer-html5 .rst-content dl.citation,html.writer-html5 .rst-content dl.field-list,html.writer-html5 .rst-content dl.footnote{display:grid;grid-template-columns:auto minmax(80%,95%)}html.writer-html5 .rst-content dl.citation>dt,html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dt{display:inline-grid;grid-template-columns:max-content auto}html.writer-html5 .rst-content aside.citation,html.writer-html5 .rst-content aside.footnote,html.writer-html5 .rst-content div.citation{display:grid;grid-template-columns:auto auto minmax(.65rem,auto) minmax(40%,95%)}html.writer-html5 .rst-content aside.citation>span.label,html.writer-html5 .rst-content aside.footnote>span.label,html.writer-html5 .rst-content div.citation>span.label{grid-column-start:1;grid-column-end:2}html.writer-html5 .rst-content aside.citation>span.backrefs,html.writer-html5 .rst-content aside.footnote>span.backrefs,html.writer-html5 .rst-content div.citation>span.backrefs{grid-column-start:2;grid-column-end:3;grid-row-start:1;grid-row-end:3}html.writer-html5 .rst-content aside.citation>p,html.writer-html5 .rst-content aside.footnote>p,html.writer-html5 .rst-content div.citation>p{grid-column-start:4;grid-column-end:5}html.writer-html5 .rst-content dl.citation,html.writer-html5 .rst-content dl.field-list,html.writer-html5 .rst-content dl.footnote{margin-bottom:24px}html.writer-html5 .rst-content dl.citation>dt,html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dt{padding-left:1rem}html.writer-html5 .rst-content dl.citation>dd,html.writer-html5 .rst-content dl.citation>dt,html.writer-html5 .rst-content dl.field-list>dd,html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dd,html.writer-html5 .rst-content dl.footnote>dt{margin-bottom:0}html.writer-html5 .rst-content dl.citation,html.writer-html5 .rst-content dl.footnote{font-size:.9rem}html.writer-html5 .rst-content dl.citation>dt,html.writer-html5 .rst-content dl.footnote>dt{margin:0 .5rem .5rem 0;line-height:1.2rem;word-break:break-all;font-weight:400}html.writer-html5 .rst-content dl.citation>dt>span.brackets:before,html.writer-html5 .rst-content dl.footnote>dt>span.brackets:before{content:"["}html.writer-html5 .rst-content dl.citation>dt>span.brackets:after,html.writer-html5 .rst-content dl.footnote>dt>span.brackets:after{content:"]"}html.writer-html5 .rst-content dl.citation>dt>span.fn-backref,html.writer-html5 .rst-content dl.footnote>dt>span.fn-backref{text-align:left;font-style:italic;margin-left:.65rem;word-break:break-word;word-spacing:-.1rem;max-width:5rem}html.writer-html5 .rst-content dl.citation>dt>span.fn-backref>a,html.writer-html5 .rst-content dl.footnote>dt>span.fn-backref>a{word-break:keep-all}html.writer-html5 .rst-content dl.citation>dt>span.fn-backref>a:not(:first-child):before,html.writer-html5 .rst-content dl.footnote>dt>span.fn-backref>a:not(:first-child):before{content:" "}html.writer-html5 .rst-content dl.citation>dd,html.writer-html5 .rst-content dl.footnote>dd{margin:0 0 .5rem;line-height:1.2rem}html.writer-html5 .rst-content dl.citation>dd p,html.writer-html5 .rst-content dl.footnote>dd p{font-size:.9rem}html.writer-html5 .rst-content aside.citation,html.writer-html5 .rst-content aside.footnote,html.writer-html5 .rst-content div.citation{padding-left:1rem;padding-right:1rem;font-size:.9rem;line-height:1.2rem}html.writer-html5 .rst-content aside.citation p,html.writer-html5 .rst-content aside.footnote p,html.writer-html5 .rst-content div.citation p{font-size:.9rem;line-height:1.2rem;margin-bottom:12px}html.writer-html5 .rst-content aside.citation span.backrefs,html.writer-html5 .rst-content aside.footnote span.backrefs,html.writer-html5 .rst-content div.citation span.backrefs{text-align:left;font-style:italic;margin-left:.65rem;word-break:break-word;word-spacing:-.1rem;max-width:5rem}html.writer-html5 .rst-content aside.citation span.backrefs>a,html.writer-html5 .rst-content aside.footnote span.backrefs>a,html.writer-html5 .rst-content div.citation span.backrefs>a{word-break:keep-all}html.writer-html5 .rst-content aside.citation span.backrefs>a:not(:first-child):before,html.writer-html5 .rst-content aside.footnote span.backrefs>a:not(:first-child):before,html.writer-html5 .rst-content div.citation span.backrefs>a:not(:first-child):before{content:" "}html.writer-html5 .rst-content aside.citation span.label,html.writer-html5 .rst-content aside.footnote span.label,html.writer-html5 .rst-content div.citation span.label{line-height:1.2rem}html.writer-html5 .rst-content aside.citation-list,html.writer-html5 .rst-content aside.footnote-list,html.writer-html5 .rst-content div.citation-list{margin-bottom:24px}html.writer-html5 .rst-content dl.option-list kbd{font-size:.9rem}.rst-content table.docutils.footnote,html.writer-html4 .rst-content table.docutils.citation,html.writer-html5 .rst-content aside.footnote,html.writer-html5 .rst-content aside.footnote-list aside.footnote,html.writer-html5 .rst-content div.citation-list>div.citation,html.writer-html5 .rst-content dl.citation,html.writer-html5 .rst-content dl.footnote{color:grey}.rst-content table.docutils.footnote code,.rst-content table.docutils.footnote tt,html.writer-html4 .rst-content table.docutils.citation code,html.writer-html4 .rst-content table.docutils.citation tt,html.writer-html5 .rst-content aside.footnote-list aside.footnote code,html.writer-html5 .rst-content aside.footnote-list aside.footnote tt,html.writer-html5 .rst-content aside.footnote code,html.writer-html5 .rst-content aside.footnote tt,html.writer-html5 .rst-content div.citation-list>div.citation code,html.writer-html5 .rst-content div.citation-list>div.citation tt,html.writer-html5 .rst-content dl.citation code,html.writer-html5 .rst-content dl.citation tt,html.writer-html5 .rst-content dl.footnote code,html.writer-html5 .rst-content dl.footnote tt{color:#555}.rst-content .wy-table-responsive.citation,.rst-content .wy-table-responsive.footnote{margin-bottom:0}.rst-content .wy-table-responsive.citation+:not(.citation),.rst-content .wy-table-responsive.footnote+:not(.footnote){margin-top:24px}.rst-content .wy-table-responsive.citation:last-child,.rst-content .wy-table-responsive.footnote:last-child{margin-bottom:24px}.rst-content table.docutils th{border-color:#e1e4e5}html.writer-html5 .rst-content table.docutils th{border:1px solid #e1e4e5}html.writer-html5 .rst-content table.docutils td>p,html.writer-html5 .rst-content table.docutils th>p{line-height:1rem;margin-bottom:0;font-size:.9rem}.rst-content table.docutils td .last,.rst-content table.docutils td .last>:last-child{margin-bottom:0}.rst-content table.field-list,.rst-content table.field-list td{border:none}.rst-content table.field-list td p{line-height:inherit}.rst-content table.field-list td>strong{display:inline-block}.rst-content table.field-list .field-name{padding-right:10px;text-align:left;white-space:nowrap}.rst-content table.field-list .field-body{text-align:left}.rst-content code,.rst-content tt{color:#000;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;padding:2px 5px}.rst-content code big,.rst-content code em,.rst-content tt big,.rst-content tt em{font-size:100%!important;line-height:normal}.rst-content code.literal,.rst-content tt.literal{color:#e74c3c;white-space:normal}.rst-content code.xref,.rst-content tt.xref,a .rst-content code,a .rst-content tt{font-weight:700;color:#404040;overflow-wrap:normal}.rst-content kbd,.rst-content pre,.rst-content samp{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace}.rst-content a code,.rst-content a tt{color:#2980b9}.rst-content dl{margin-bottom:24px}.rst-content dl dt{font-weight:700;margin-bottom:12px}.rst-content dl ol,.rst-content dl p,.rst-content dl table,.rst-content dl ul{margin-bottom:12px}.rst-content dl dd{margin:0 0 12px 24px;line-height:24px}.rst-content dl dd>ol:last-child,.rst-content dl dd>p:last-child,.rst-content dl dd>table:last-child,.rst-content dl dd>ul:last-child{margin-bottom:0}html.writer-html4 .rst-content dl:not(.docutils),html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple){margin-bottom:24px}html.writer-html4 .rst-content dl:not(.docutils)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt{display:table;margin:6px 0;font-size:90%;line-height:normal;background:#e7f2fa;color:#2980b9;border-top:3px solid #6ab0de;padding:6px;position:relative}html.writer-html4 .rst-content dl:not(.docutils)>dt:before,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt:before{color:#6ab0de}html.writer-html4 .rst-content dl:not(.docutils)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) dl:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt{margin-bottom:6px;border:none;border-left:3px solid #ccc;background:#f0f0f0;color:#555}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) dl:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils)>dt:first-child,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple)>dt:first-child{margin-top:0}html.writer-html4 .rst-content dl:not(.docutils) code.descclassname,html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descclassname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) code.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) tt.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) tt.descname{background-color:transparent;border:none;padding:0;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) tt.descname{font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .optional,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .optional{display:inline-block;padding:0 4px;color:#000;font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .property,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .property{display:inline-block;padding-right:8px;max-width:100%}html.writer-html4 .rst-content dl:not(.docutils) .k,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .k{font-style:italic}html.writer-html4 .rst-content dl:not(.docutils) .descclassname,html.writer-html4 .rst-content dl:not(.docutils) .descname,html.writer-html4 .rst-content dl:not(.docutils) .sig-name,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.citation):not(.glossary):not(.simple) .sig-name{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;color:#000}.rst-content .viewcode-back,.rst-content .viewcode-link{display:inline-block;color:#27ae60;font-size:80%;padding-left:24px}.rst-content .viewcode-back{display:block;float:right}.rst-content p.rubric{margin-bottom:12px;font-weight:700}.rst-content code.download,.rst-content tt.download{background:inherit;padding:inherit;font-weight:400;font-family:inherit;font-size:inherit;color:inherit;border:inherit;white-space:inherit}.rst-content code.download span:first-child,.rst-content tt.download span:first-child{-webkit-font-smoothing:subpixel-antialiased}.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{margin-right:4px}.rst-content .guilabel,.rst-content .menuselection{font-size:80%;font-weight:700;border-radius:4px;padding:2.4px 6px;margin:auto 2px}.rst-content .guilabel,.rst-content .menuselection{border:1px solid #7fbbe3;background:#e7f2fa}.rst-content :not(dl.option-list)>:not(dt):not(kbd):not(.kbd)>.kbd,.rst-content :not(dl.option-list)>:not(dt):not(kbd):not(.kbd)>kbd{color:inherit;font-size:80%;background-color:#fff;border:1px solid #a6a6a6;border-radius:4px;box-shadow:0 2px grey;padding:2.4px 6px;margin:auto 0}.rst-content .versionmodified{font-style:italic}@media screen and (max-width:480px){.rst-content .sidebar{width:100%}}span[id*=MathJax-Span]{color:#404040}.math{text-align:center}@font-face{font-family:Lato;src:url(fonts/lato-normal.woff2?bd03a2cc277bbbc338d464e679fe9942) format("woff2"),url(fonts/lato-normal.woff?27bd77b9162d388cb8d4c4217c7c5e2a) format("woff");font-weight:400;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold.woff2?cccb897485813c7c256901dbca54ecf2) format("woff2"),url(fonts/lato-bold.woff?d878b6c29b10beca227e9eef4246111b) format("woff");font-weight:700;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold-italic.woff2?0b6bb6725576b072c5d0b02ecdd1900d) format("woff2"),url(fonts/lato-bold-italic.woff?9c7e4e9eb485b4a121c760e61bc3707c) format("woff");font-weight:700;font-style:italic;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-normal-italic.woff2?4eb103b4d12be57cb1d040ed5e162e9d) format("woff2"),url(fonts/lato-normal-italic.woff?f28f2d6482446544ef1ea1ccc6dd5892) format("woff");font-weight:400;font-style:italic;font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:400;src:url(fonts/Roboto-Slab-Regular.woff2?7abf5b8d04d26a2cafea937019bca958) format("woff2"),url(fonts/Roboto-Slab-Regular.woff?c1be9284088d487c5e3ff0a10a92e58c) format("woff");font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:700;src:url(fonts/Roboto-Slab-Bold.woff2?9984f4a9bda09be08e83f2506954adbe) format("woff2"),url(fonts/Roboto-Slab-Bold.woff?bed5564a116b05148e3b3bea6fb1162a) format("woff");font-display:block} \ No newline at end of file diff --git a/0.26/_static/doctools.js b/0.26/_static/doctools.js new file mode 100644 index 000000000..527b876ca --- /dev/null +++ b/0.26/_static/doctools.js @@ -0,0 +1,156 @@ +/* + * doctools.js + * ~~~~~~~~~~~ + * + * Base JavaScript utilities for all Sphinx HTML documentation. + * + * :copyright: Copyright 2007-2022 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ +"use strict"; + +const BLACKLISTED_KEY_CONTROL_ELEMENTS = new Set([ + "TEXTAREA", + "INPUT", + "SELECT", + "BUTTON", +]); + +const _ready = (callback) => { + if (document.readyState !== "loading") { + callback(); + } else { + document.addEventListener("DOMContentLoaded", callback); + } +}; + +/** + * Small JavaScript module for the documentation. + */ +const Documentation = { + init: () => { + Documentation.initDomainIndexTable(); + Documentation.initOnKeyListeners(); + }, + + /** + * i18n support + */ + TRANSLATIONS: {}, + PLURAL_EXPR: (n) => (n === 1 ? 0 : 1), + LOCALE: "unknown", + + // gettext and ngettext don't access this so that the functions + // can safely bound to a different name (_ = Documentation.gettext) + gettext: (string) => { + const translated = Documentation.TRANSLATIONS[string]; + switch (typeof translated) { + case "undefined": + return string; // no translation + case "string": + return translated; // translation exists + default: + return translated[0]; // (singular, plural) translation tuple exists + } + }, + + ngettext: (singular, plural, n) => { + const translated = Documentation.TRANSLATIONS[singular]; + if (typeof translated !== "undefined") + return translated[Documentation.PLURAL_EXPR(n)]; + return n === 1 ? singular : plural; + }, + + addTranslations: (catalog) => { + Object.assign(Documentation.TRANSLATIONS, catalog.messages); + Documentation.PLURAL_EXPR = new Function( + "n", + `return (${catalog.plural_expr})` + ); + Documentation.LOCALE = catalog.locale; + }, + + /** + * helper function to focus on search bar + */ + focusSearchBar: () => { + document.querySelectorAll("input[name=q]")[0]?.focus(); + }, + + /** + * Initialise the domain index toggle buttons + */ + initDomainIndexTable: () => { + const toggler = (el) => { + const idNumber = el.id.substr(7); + const toggledRows = document.querySelectorAll(`tr.cg-${idNumber}`); + if (el.src.substr(-9) === "minus.png") { + el.src = `${el.src.substr(0, el.src.length - 9)}plus.png`; + toggledRows.forEach((el) => (el.style.display = "none")); + } else { + el.src = `${el.src.substr(0, el.src.length - 8)}minus.png`; + toggledRows.forEach((el) => (el.style.display = "")); + } + }; + + const togglerElements = document.querySelectorAll("img.toggler"); + togglerElements.forEach((el) => + el.addEventListener("click", (event) => toggler(event.currentTarget)) + ); + togglerElements.forEach((el) => (el.style.display = "")); + if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) togglerElements.forEach(toggler); + }, + + initOnKeyListeners: () => { + // only install a listener if it is really needed + if ( + !DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS && + !DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS + ) + return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.altKey || event.ctrlKey || event.metaKey) return; + + if (!event.shiftKey) { + switch (event.key) { + case "ArrowLeft": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const prevLink = document.querySelector('link[rel="prev"]'); + if (prevLink && prevLink.href) { + window.location.href = prevLink.href; + event.preventDefault(); + } + break; + case "ArrowRight": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const nextLink = document.querySelector('link[rel="next"]'); + if (nextLink && nextLink.href) { + window.location.href = nextLink.href; + event.preventDefault(); + } + break; + } + } + + // some keyboard layouts may need Shift to get / + switch (event.key) { + case "/": + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) break; + Documentation.focusSearchBar(); + event.preventDefault(); + } + }); + }, +}; + +// quick alias for translations +const _ = Documentation.gettext; + +_ready(Documentation.init); diff --git a/0.26/_static/documentation_options.js b/0.26/_static/documentation_options.js new file mode 100644 index 000000000..b57ae3b83 --- /dev/null +++ b/0.26/_static/documentation_options.js @@ -0,0 +1,14 @@ +var DOCUMENTATION_OPTIONS = { + URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'), + VERSION: '', + LANGUAGE: 'en', + COLLAPSE_INDEX: false, + BUILDER: 'html', + FILE_SUFFIX: '.html', + LINK_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt', + NAVIGATION_WITH_KEYS: false, + SHOW_SEARCH_SUMMARY: true, + ENABLE_SEARCH_SHORTCUTS: true, +}; \ No newline at end of file diff --git a/0.26/_static/file.png b/0.26/_static/file.png new file mode 100644 index 000000000..a858a410e Binary files /dev/null and b/0.26/_static/file.png differ diff --git a/0.26/_static/jquery-3.6.0.js b/0.26/_static/jquery-3.6.0.js new file mode 100644 index 000000000..fc6c299b7 --- /dev/null +++ b/0.26/_static/jquery-3.6.0.js @@ -0,0 +1,10881 @@ +/*! + * jQuery JavaScript Library v3.6.0 + * https://jquery.com/ + * + * Includes Sizzle.js + * https://sizzlejs.com/ + * + * Copyright OpenJS Foundation and other contributors + * Released under the MIT license + * https://jquery.org/license + * + * Date: 2021-03-02T17:08Z + */ +( function( global, factory ) { + + "use strict"; + + if ( typeof module === "object" && typeof module.exports === "object" ) { + + // For CommonJS and CommonJS-like environments where a proper `window` + // is present, execute the factory and get jQuery. + // For environments that do not have a `window` with a `document` + // (such as Node.js), expose a factory as module.exports. + // This accentuates the need for the creation of a real `window`. + // e.g. var jQuery = require("jquery")(window); + // See ticket #14549 for more info. + module.exports = global.document ? + factory( global, true ) : + function( w ) { + if ( !w.document ) { + throw new Error( "jQuery requires a window with a document" ); + } + return factory( w ); + }; + } else { + factory( global ); + } + +// Pass this if window is not defined yet +} )( typeof window !== "undefined" ? window : this, function( window, noGlobal ) { + +// Edge <= 12 - 13+, Firefox <=18 - 45+, IE 10 - 11, Safari 5.1 - 9+, iOS 6 - 9.1 +// throw exceptions when non-strict code (e.g., ASP.NET 4.5) accesses strict mode +// arguments.callee.caller (trac-13335). But as of jQuery 3.0 (2016), strict mode should be common +// enough that all such attempts are guarded in a try block. +"use strict"; + +var arr = []; + +var getProto = Object.getPrototypeOf; + +var slice = arr.slice; + +var flat = arr.flat ? function( array ) { + return arr.flat.call( array ); +} : function( array ) { + return arr.concat.apply( [], array ); +}; + + +var push = arr.push; + +var indexOf = arr.indexOf; + +var class2type = {}; + +var toString = class2type.toString; + +var hasOwn = class2type.hasOwnProperty; + +var fnToString = hasOwn.toString; + +var ObjectFunctionString = fnToString.call( Object ); + +var support = {}; + +var isFunction = function isFunction( obj ) { + + // Support: Chrome <=57, Firefox <=52 + // In some browsers, typeof returns "function" for HTML elements + // (i.e., `typeof document.createElement( "object" ) === "function"`). + // We don't want to classify *any* DOM node as a function. + // Support: QtWeb <=3.8.5, WebKit <=534.34, wkhtmltopdf tool <=0.12.5 + // Plus for old WebKit, typeof returns "function" for HTML collections + // (e.g., `typeof document.getElementsByTagName("div") === "function"`). (gh-4756) + return typeof obj === "function" && typeof obj.nodeType !== "number" && + typeof obj.item !== "function"; + }; + + +var isWindow = function isWindow( obj ) { + return obj != null && obj === obj.window; + }; + + +var document = window.document; + + + + var preservedScriptAttributes = { + type: true, + src: true, + nonce: true, + noModule: true + }; + + function DOMEval( code, node, doc ) { + doc = doc || document; + + var i, val, + script = doc.createElement( "script" ); + + script.text = code; + if ( node ) { + for ( i in preservedScriptAttributes ) { + + // Support: Firefox 64+, Edge 18+ + // Some browsers don't support the "nonce" property on scripts. + // On the other hand, just using `getAttribute` is not enough as + // the `nonce` attribute is reset to an empty string whenever it + // becomes browsing-context connected. + // See https://github.com/whatwg/html/issues/2369 + // See https://html.spec.whatwg.org/#nonce-attributes + // The `node.getAttribute` check was added for the sake of + // `jQuery.globalEval` so that it can fake a nonce-containing node + // via an object. + val = node[ i ] || node.getAttribute && node.getAttribute( i ); + if ( val ) { + script.setAttribute( i, val ); + } + } + } + doc.head.appendChild( script ).parentNode.removeChild( script ); + } + + +function toType( obj ) { + if ( obj == null ) { + return obj + ""; + } + + // Support: Android <=2.3 only (functionish RegExp) + return typeof obj === "object" || typeof obj === "function" ? + class2type[ toString.call( obj ) ] || "object" : + typeof obj; +} +/* global Symbol */ +// Defining this global in .eslintrc.json would create a danger of using the global +// unguarded in another place, it seems safer to define global only for this module + + + +var + version = "3.6.0", + + // Define a local copy of jQuery + jQuery = function( selector, context ) { + + // The jQuery object is actually just the init constructor 'enhanced' + // Need init if jQuery is called (just allow error to be thrown if not included) + return new jQuery.fn.init( selector, context ); + }; + +jQuery.fn = jQuery.prototype = { + + // The current version of jQuery being used + jquery: version, + + constructor: jQuery, + + // The default length of a jQuery object is 0 + length: 0, + + toArray: function() { + return slice.call( this ); + }, + + // Get the Nth element in the matched element set OR + // Get the whole matched element set as a clean array + get: function( num ) { + + // Return all the elements in a clean array + if ( num == null ) { + return slice.call( this ); + } + + // Return just the one element from the set + return num < 0 ? this[ num + this.length ] : this[ num ]; + }, + + // Take an array of elements and push it onto the stack + // (returning the new matched element set) + pushStack: function( elems ) { + + // Build a new jQuery matched element set + var ret = jQuery.merge( this.constructor(), elems ); + + // Add the old object onto the stack (as a reference) + ret.prevObject = this; + + // Return the newly-formed element set + return ret; + }, + + // Execute a callback for every element in the matched set. + each: function( callback ) { + return jQuery.each( this, callback ); + }, + + map: function( callback ) { + return this.pushStack( jQuery.map( this, function( elem, i ) { + return callback.call( elem, i, elem ); + } ) ); + }, + + slice: function() { + return this.pushStack( slice.apply( this, arguments ) ); + }, + + first: function() { + return this.eq( 0 ); + }, + + last: function() { + return this.eq( -1 ); + }, + + even: function() { + return this.pushStack( jQuery.grep( this, function( _elem, i ) { + return ( i + 1 ) % 2; + } ) ); + }, + + odd: function() { + return this.pushStack( jQuery.grep( this, function( _elem, i ) { + return i % 2; + } ) ); + }, + + eq: function( i ) { + var len = this.length, + j = +i + ( i < 0 ? len : 0 ); + return this.pushStack( j >= 0 && j < len ? [ this[ j ] ] : [] ); + }, + + end: function() { + return this.prevObject || this.constructor(); + }, + + // For internal use only. + // Behaves like an Array's method, not like a jQuery method. + push: push, + sort: arr.sort, + splice: arr.splice +}; + +jQuery.extend = jQuery.fn.extend = function() { + var options, name, src, copy, copyIsArray, clone, + target = arguments[ 0 ] || {}, + i = 1, + length = arguments.length, + deep = false; + + // Handle a deep copy situation + if ( typeof target === "boolean" ) { + deep = target; + + // Skip the boolean and the target + target = arguments[ i ] || {}; + i++; + } + + // Handle case when target is a string or something (possible in deep copy) + if ( typeof target !== "object" && !isFunction( target ) ) { + target = {}; + } + + // Extend jQuery itself if only one argument is passed + if ( i === length ) { + target = this; + i--; + } + + for ( ; i < length; i++ ) { + + // Only deal with non-null/undefined values + if ( ( options = arguments[ i ] ) != null ) { + + // Extend the base object + for ( name in options ) { + copy = options[ name ]; + + // Prevent Object.prototype pollution + // Prevent never-ending loop + if ( name === "__proto__" || target === copy ) { + continue; + } + + // Recurse if we're merging plain objects or arrays + if ( deep && copy && ( jQuery.isPlainObject( copy ) || + ( copyIsArray = Array.isArray( copy ) ) ) ) { + src = target[ name ]; + + // Ensure proper type for the source value + if ( copyIsArray && !Array.isArray( src ) ) { + clone = []; + } else if ( !copyIsArray && !jQuery.isPlainObject( src ) ) { + clone = {}; + } else { + clone = src; + } + copyIsArray = false; + + // Never move original objects, clone them + target[ name ] = jQuery.extend( deep, clone, copy ); + + // Don't bring in undefined values + } else if ( copy !== undefined ) { + target[ name ] = copy; + } + } + } + } + + // Return the modified object + return target; +}; + +jQuery.extend( { + + // Unique for each copy of jQuery on the page + expando: "jQuery" + ( version + Math.random() ).replace( /\D/g, "" ), + + // Assume jQuery is ready without the ready module + isReady: true, + + error: function( msg ) { + throw new Error( msg ); + }, + + noop: function() {}, + + isPlainObject: function( obj ) { + var proto, Ctor; + + // Detect obvious negatives + // Use toString instead of jQuery.type to catch host objects + if ( !obj || toString.call( obj ) !== "[object Object]" ) { + return false; + } + + proto = getProto( obj ); + + // Objects with no prototype (e.g., `Object.create( null )`) are plain + if ( !proto ) { + return true; + } + + // Objects with prototype are plain iff they were constructed by a global Object function + Ctor = hasOwn.call( proto, "constructor" ) && proto.constructor; + return typeof Ctor === "function" && fnToString.call( Ctor ) === ObjectFunctionString; + }, + + isEmptyObject: function( obj ) { + var name; + + for ( name in obj ) { + return false; + } + return true; + }, + + // Evaluates a script in a provided context; falls back to the global one + // if not specified. + globalEval: function( code, options, doc ) { + DOMEval( code, { nonce: options && options.nonce }, doc ); + }, + + each: function( obj, callback ) { + var length, i = 0; + + if ( isArrayLike( obj ) ) { + length = obj.length; + for ( ; i < length; i++ ) { + if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { + break; + } + } + } else { + for ( i in obj ) { + if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { + break; + } + } + } + + return obj; + }, + + // results is for internal usage only + makeArray: function( arr, results ) { + var ret = results || []; + + if ( arr != null ) { + if ( isArrayLike( Object( arr ) ) ) { + jQuery.merge( ret, + typeof arr === "string" ? + [ arr ] : arr + ); + } else { + push.call( ret, arr ); + } + } + + return ret; + }, + + inArray: function( elem, arr, i ) { + return arr == null ? -1 : indexOf.call( arr, elem, i ); + }, + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + merge: function( first, second ) { + var len = +second.length, + j = 0, + i = first.length; + + for ( ; j < len; j++ ) { + first[ i++ ] = second[ j ]; + } + + first.length = i; + + return first; + }, + + grep: function( elems, callback, invert ) { + var callbackInverse, + matches = [], + i = 0, + length = elems.length, + callbackExpect = !invert; + + // Go through the array, only saving the items + // that pass the validator function + for ( ; i < length; i++ ) { + callbackInverse = !callback( elems[ i ], i ); + if ( callbackInverse !== callbackExpect ) { + matches.push( elems[ i ] ); + } + } + + return matches; + }, + + // arg is for internal usage only + map: function( elems, callback, arg ) { + var length, value, + i = 0, + ret = []; + + // Go through the array, translating each of the items to their new values + if ( isArrayLike( elems ) ) { + length = elems.length; + for ( ; i < length; i++ ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret.push( value ); + } + } + + // Go through every key on the object, + } else { + for ( i in elems ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret.push( value ); + } + } + } + + // Flatten any nested arrays + return flat( ret ); + }, + + // A global GUID counter for objects + guid: 1, + + // jQuery.support is not used in Core but other projects attach their + // properties to it so it needs to exist. + support: support +} ); + +if ( typeof Symbol === "function" ) { + jQuery.fn[ Symbol.iterator ] = arr[ Symbol.iterator ]; +} + +// Populate the class2type map +jQuery.each( "Boolean Number String Function Array Date RegExp Object Error Symbol".split( " " ), + function( _i, name ) { + class2type[ "[object " + name + "]" ] = name.toLowerCase(); + } ); + +function isArrayLike( obj ) { + + // Support: real iOS 8.2 only (not reproducible in simulator) + // `in` check used to prevent JIT error (gh-2145) + // hasOwn isn't used here due to false negatives + // regarding Nodelist length in IE + var length = !!obj && "length" in obj && obj.length, + type = toType( obj ); + + if ( isFunction( obj ) || isWindow( obj ) ) { + return false; + } + + return type === "array" || length === 0 || + typeof length === "number" && length > 0 && ( length - 1 ) in obj; +} +var Sizzle = +/*! + * Sizzle CSS Selector Engine v2.3.6 + * https://sizzlejs.com/ + * + * Copyright JS Foundation and other contributors + * Released under the MIT license + * https://js.foundation/ + * + * Date: 2021-02-16 + */ +( function( window ) { +var i, + support, + Expr, + getText, + isXML, + tokenize, + compile, + select, + outermostContext, + sortInput, + hasDuplicate, + + // Local document vars + setDocument, + document, + docElem, + documentIsHTML, + rbuggyQSA, + rbuggyMatches, + matches, + contains, + + // Instance-specific data + expando = "sizzle" + 1 * new Date(), + preferredDoc = window.document, + dirruns = 0, + done = 0, + classCache = createCache(), + tokenCache = createCache(), + compilerCache = createCache(), + nonnativeSelectorCache = createCache(), + sortOrder = function( a, b ) { + if ( a === b ) { + hasDuplicate = true; + } + return 0; + }, + + // Instance methods + hasOwn = ( {} ).hasOwnProperty, + arr = [], + pop = arr.pop, + pushNative = arr.push, + push = arr.push, + slice = arr.slice, + + // Use a stripped-down indexOf as it's faster than native + // https://jsperf.com/thor-indexof-vs-for/5 + indexOf = function( list, elem ) { + var i = 0, + len = list.length; + for ( ; i < len; i++ ) { + if ( list[ i ] === elem ) { + return i; + } + } + return -1; + }, + + booleans = "checked|selected|async|autofocus|autoplay|controls|defer|disabled|hidden|" + + "ismap|loop|multiple|open|readonly|required|scoped", + + // Regular expressions + + // http://www.w3.org/TR/css3-selectors/#whitespace + whitespace = "[\\x20\\t\\r\\n\\f]", + + // https://www.w3.org/TR/css-syntax-3/#ident-token-diagram + identifier = "(?:\\\\[\\da-fA-F]{1,6}" + whitespace + + "?|\\\\[^\\r\\n\\f]|[\\w-]|[^\0-\\x7f])+", + + // Attribute selectors: http://www.w3.org/TR/selectors/#attribute-selectors + attributes = "\\[" + whitespace + "*(" + identifier + ")(?:" + whitespace + + + // Operator (capture 2) + "*([*^$|!~]?=)" + whitespace + + + // "Attribute values must be CSS identifiers [capture 5] + // or strings [capture 3 or capture 4]" + "*(?:'((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\"|(" + identifier + "))|)" + + whitespace + "*\\]", + + pseudos = ":(" + identifier + ")(?:\\((" + + + // To reduce the number of selectors needing tokenize in the preFilter, prefer arguments: + // 1. quoted (capture 3; capture 4 or capture 5) + "('((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\")|" + + + // 2. simple (capture 6) + "((?:\\\\.|[^\\\\()[\\]]|" + attributes + ")*)|" + + + // 3. anything else (capture 2) + ".*" + + ")\\)|)", + + // Leading and non-escaped trailing whitespace, capturing some non-whitespace characters preceding the latter + rwhitespace = new RegExp( whitespace + "+", "g" ), + rtrim = new RegExp( "^" + whitespace + "+|((?:^|[^\\\\])(?:\\\\.)*)" + + whitespace + "+$", "g" ), + + rcomma = new RegExp( "^" + whitespace + "*," + whitespace + "*" ), + rcombinators = new RegExp( "^" + whitespace + "*([>+~]|" + whitespace + ")" + whitespace + + "*" ), + rdescend = new RegExp( whitespace + "|>" ), + + rpseudo = new RegExp( pseudos ), + ridentifier = new RegExp( "^" + identifier + "$" ), + + matchExpr = { + "ID": new RegExp( "^#(" + identifier + ")" ), + "CLASS": new RegExp( "^\\.(" + identifier + ")" ), + "TAG": new RegExp( "^(" + identifier + "|[*])" ), + "ATTR": new RegExp( "^" + attributes ), + "PSEUDO": new RegExp( "^" + pseudos ), + "CHILD": new RegExp( "^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\(" + + whitespace + "*(even|odd|(([+-]|)(\\d*)n|)" + whitespace + "*(?:([+-]|)" + + whitespace + "*(\\d+)|))" + whitespace + "*\\)|)", "i" ), + "bool": new RegExp( "^(?:" + booleans + ")$", "i" ), + + // For use in libraries implementing .is() + // We use this for POS matching in `select` + "needsContext": new RegExp( "^" + whitespace + + "*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\(" + whitespace + + "*((?:-\\d)?\\d*)" + whitespace + "*\\)|)(?=[^-]|$)", "i" ) + }, + + rhtml = /HTML$/i, + rinputs = /^(?:input|select|textarea|button)$/i, + rheader = /^h\d$/i, + + rnative = /^[^{]+\{\s*\[native \w/, + + // Easily-parseable/retrievable ID or TAG or CLASS selectors + rquickExpr = /^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/, + + rsibling = /[+~]/, + + // CSS escapes + // http://www.w3.org/TR/CSS21/syndata.html#escaped-characters + runescape = new RegExp( "\\\\[\\da-fA-F]{1,6}" + whitespace + "?|\\\\([^\\r\\n\\f])", "g" ), + funescape = function( escape, nonHex ) { + var high = "0x" + escape.slice( 1 ) - 0x10000; + + return nonHex ? + + // Strip the backslash prefix from a non-hex escape sequence + nonHex : + + // Replace a hexadecimal escape sequence with the encoded Unicode code point + // Support: IE <=11+ + // For values outside the Basic Multilingual Plane (BMP), manually construct a + // surrogate pair + high < 0 ? + String.fromCharCode( high + 0x10000 ) : + String.fromCharCode( high >> 10 | 0xD800, high & 0x3FF | 0xDC00 ); + }, + + // CSS string/identifier serialization + // https://drafts.csswg.org/cssom/#common-serializing-idioms + rcssescape = /([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g, + fcssescape = function( ch, asCodePoint ) { + if ( asCodePoint ) { + + // U+0000 NULL becomes U+FFFD REPLACEMENT CHARACTER + if ( ch === "\0" ) { + return "\uFFFD"; + } + + // Control characters and (dependent upon position) numbers get escaped as code points + return ch.slice( 0, -1 ) + "\\" + + ch.charCodeAt( ch.length - 1 ).toString( 16 ) + " "; + } + + // Other potentially-special ASCII characters get backslash-escaped + return "\\" + ch; + }, + + // Used for iframes + // See setDocument() + // Removing the function wrapper causes a "Permission Denied" + // error in IE + unloadHandler = function() { + setDocument(); + }, + + inDisabledFieldset = addCombinator( + function( elem ) { + return elem.disabled === true && elem.nodeName.toLowerCase() === "fieldset"; + }, + { dir: "parentNode", next: "legend" } + ); + +// Optimize for push.apply( _, NodeList ) +try { + push.apply( + ( arr = slice.call( preferredDoc.childNodes ) ), + preferredDoc.childNodes + ); + + // Support: Android<4.0 + // Detect silently failing push.apply + // eslint-disable-next-line no-unused-expressions + arr[ preferredDoc.childNodes.length ].nodeType; +} catch ( e ) { + push = { apply: arr.length ? + + // Leverage slice if possible + function( target, els ) { + pushNative.apply( target, slice.call( els ) ); + } : + + // Support: IE<9 + // Otherwise append directly + function( target, els ) { + var j = target.length, + i = 0; + + // Can't trust NodeList.length + while ( ( target[ j++ ] = els[ i++ ] ) ) {} + target.length = j - 1; + } + }; +} + +function Sizzle( selector, context, results, seed ) { + var m, i, elem, nid, match, groups, newSelector, + newContext = context && context.ownerDocument, + + // nodeType defaults to 9, since context defaults to document + nodeType = context ? context.nodeType : 9; + + results = results || []; + + // Return early from calls with invalid selector or context + if ( typeof selector !== "string" || !selector || + nodeType !== 1 && nodeType !== 9 && nodeType !== 11 ) { + + return results; + } + + // Try to shortcut find operations (as opposed to filters) in HTML documents + if ( !seed ) { + setDocument( context ); + context = context || document; + + if ( documentIsHTML ) { + + // If the selector is sufficiently simple, try using a "get*By*" DOM method + // (excepting DocumentFragment context, where the methods don't exist) + if ( nodeType !== 11 && ( match = rquickExpr.exec( selector ) ) ) { + + // ID selector + if ( ( m = match[ 1 ] ) ) { + + // Document context + if ( nodeType === 9 ) { + if ( ( elem = context.getElementById( m ) ) ) { + + // Support: IE, Opera, Webkit + // TODO: identify versions + // getElementById can match elements by name instead of ID + if ( elem.id === m ) { + results.push( elem ); + return results; + } + } else { + return results; + } + + // Element context + } else { + + // Support: IE, Opera, Webkit + // TODO: identify versions + // getElementById can match elements by name instead of ID + if ( newContext && ( elem = newContext.getElementById( m ) ) && + contains( context, elem ) && + elem.id === m ) { + + results.push( elem ); + return results; + } + } + + // Type selector + } else if ( match[ 2 ] ) { + push.apply( results, context.getElementsByTagName( selector ) ); + return results; + + // Class selector + } else if ( ( m = match[ 3 ] ) && support.getElementsByClassName && + context.getElementsByClassName ) { + + push.apply( results, context.getElementsByClassName( m ) ); + return results; + } + } + + // Take advantage of querySelectorAll + if ( support.qsa && + !nonnativeSelectorCache[ selector + " " ] && + ( !rbuggyQSA || !rbuggyQSA.test( selector ) ) && + + // Support: IE 8 only + // Exclude object elements + ( nodeType !== 1 || context.nodeName.toLowerCase() !== "object" ) ) { + + newSelector = selector; + newContext = context; + + // qSA considers elements outside a scoping root when evaluating child or + // descendant combinators, which is not what we want. + // In such cases, we work around the behavior by prefixing every selector in the + // list with an ID selector referencing the scope context. + // The technique has to be used as well when a leading combinator is used + // as such selectors are not recognized by querySelectorAll. + // Thanks to Andrew Dupont for this technique. + if ( nodeType === 1 && + ( rdescend.test( selector ) || rcombinators.test( selector ) ) ) { + + // Expand context for sibling selectors + newContext = rsibling.test( selector ) && testContext( context.parentNode ) || + context; + + // We can use :scope instead of the ID hack if the browser + // supports it & if we're not changing the context. + if ( newContext !== context || !support.scope ) { + + // Capture the context ID, setting it first if necessary + if ( ( nid = context.getAttribute( "id" ) ) ) { + nid = nid.replace( rcssescape, fcssescape ); + } else { + context.setAttribute( "id", ( nid = expando ) ); + } + } + + // Prefix every selector in the list + groups = tokenize( selector ); + i = groups.length; + while ( i-- ) { + groups[ i ] = ( nid ? "#" + nid : ":scope" ) + " " + + toSelector( groups[ i ] ); + } + newSelector = groups.join( "," ); + } + + try { + push.apply( results, + newContext.querySelectorAll( newSelector ) + ); + return results; + } catch ( qsaError ) { + nonnativeSelectorCache( selector, true ); + } finally { + if ( nid === expando ) { + context.removeAttribute( "id" ); + } + } + } + } + } + + // All others + return select( selector.replace( rtrim, "$1" ), context, results, seed ); +} + +/** + * Create key-value caches of limited size + * @returns {function(string, object)} Returns the Object data after storing it on itself with + * property name the (space-suffixed) string and (if the cache is larger than Expr.cacheLength) + * deleting the oldest entry + */ +function createCache() { + var keys = []; + + function cache( key, value ) { + + // Use (key + " ") to avoid collision with native prototype properties (see Issue #157) + if ( keys.push( key + " " ) > Expr.cacheLength ) { + + // Only keep the most recent entries + delete cache[ keys.shift() ]; + } + return ( cache[ key + " " ] = value ); + } + return cache; +} + +/** + * Mark a function for special use by Sizzle + * @param {Function} fn The function to mark + */ +function markFunction( fn ) { + fn[ expando ] = true; + return fn; +} + +/** + * Support testing using an element + * @param {Function} fn Passed the created element and returns a boolean result + */ +function assert( fn ) { + var el = document.createElement( "fieldset" ); + + try { + return !!fn( el ); + } catch ( e ) { + return false; + } finally { + + // Remove from its parent by default + if ( el.parentNode ) { + el.parentNode.removeChild( el ); + } + + // release memory in IE + el = null; + } +} + +/** + * Adds the same handler for all of the specified attrs + * @param {String} attrs Pipe-separated list of attributes + * @param {Function} handler The method that will be applied + */ +function addHandle( attrs, handler ) { + var arr = attrs.split( "|" ), + i = arr.length; + + while ( i-- ) { + Expr.attrHandle[ arr[ i ] ] = handler; + } +} + +/** + * Checks document order of two siblings + * @param {Element} a + * @param {Element} b + * @returns {Number} Returns less than 0 if a precedes b, greater than 0 if a follows b + */ +function siblingCheck( a, b ) { + var cur = b && a, + diff = cur && a.nodeType === 1 && b.nodeType === 1 && + a.sourceIndex - b.sourceIndex; + + // Use IE sourceIndex if available on both nodes + if ( diff ) { + return diff; + } + + // Check if b follows a + if ( cur ) { + while ( ( cur = cur.nextSibling ) ) { + if ( cur === b ) { + return -1; + } + } + } + + return a ? 1 : -1; +} + +/** + * Returns a function to use in pseudos for input types + * @param {String} type + */ +function createInputPseudo( type ) { + return function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && elem.type === type; + }; +} + +/** + * Returns a function to use in pseudos for buttons + * @param {String} type + */ +function createButtonPseudo( type ) { + return function( elem ) { + var name = elem.nodeName.toLowerCase(); + return ( name === "input" || name === "button" ) && elem.type === type; + }; +} + +/** + * Returns a function to use in pseudos for :enabled/:disabled + * @param {Boolean} disabled true for :disabled; false for :enabled + */ +function createDisabledPseudo( disabled ) { + + // Known :disabled false positives: fieldset[disabled] > legend:nth-of-type(n+2) :can-disable + return function( elem ) { + + // Only certain elements can match :enabled or :disabled + // https://html.spec.whatwg.org/multipage/scripting.html#selector-enabled + // https://html.spec.whatwg.org/multipage/scripting.html#selector-disabled + if ( "form" in elem ) { + + // Check for inherited disabledness on relevant non-disabled elements: + // * listed form-associated elements in a disabled fieldset + // https://html.spec.whatwg.org/multipage/forms.html#category-listed + // https://html.spec.whatwg.org/multipage/forms.html#concept-fe-disabled + // * option elements in a disabled optgroup + // https://html.spec.whatwg.org/multipage/forms.html#concept-option-disabled + // All such elements have a "form" property. + if ( elem.parentNode && elem.disabled === false ) { + + // Option elements defer to a parent optgroup if present + if ( "label" in elem ) { + if ( "label" in elem.parentNode ) { + return elem.parentNode.disabled === disabled; + } else { + return elem.disabled === disabled; + } + } + + // Support: IE 6 - 11 + // Use the isDisabled shortcut property to check for disabled fieldset ancestors + return elem.isDisabled === disabled || + + // Where there is no isDisabled, check manually + /* jshint -W018 */ + elem.isDisabled !== !disabled && + inDisabledFieldset( elem ) === disabled; + } + + return elem.disabled === disabled; + + // Try to winnow out elements that can't be disabled before trusting the disabled property. + // Some victims get caught in our net (label, legend, menu, track), but it shouldn't + // even exist on them, let alone have a boolean value. + } else if ( "label" in elem ) { + return elem.disabled === disabled; + } + + // Remaining elements are neither :enabled nor :disabled + return false; + }; +} + +/** + * Returns a function to use in pseudos for positionals + * @param {Function} fn + */ +function createPositionalPseudo( fn ) { + return markFunction( function( argument ) { + argument = +argument; + return markFunction( function( seed, matches ) { + var j, + matchIndexes = fn( [], seed.length, argument ), + i = matchIndexes.length; + + // Match elements found at the specified indexes + while ( i-- ) { + if ( seed[ ( j = matchIndexes[ i ] ) ] ) { + seed[ j ] = !( matches[ j ] = seed[ j ] ); + } + } + } ); + } ); +} + +/** + * Checks a node for validity as a Sizzle context + * @param {Element|Object=} context + * @returns {Element|Object|Boolean} The input node if acceptable, otherwise a falsy value + */ +function testContext( context ) { + return context && typeof context.getElementsByTagName !== "undefined" && context; +} + +// Expose support vars for convenience +support = Sizzle.support = {}; + +/** + * Detects XML nodes + * @param {Element|Object} elem An element or a document + * @returns {Boolean} True iff elem is a non-HTML XML node + */ +isXML = Sizzle.isXML = function( elem ) { + var namespace = elem && elem.namespaceURI, + docElem = elem && ( elem.ownerDocument || elem ).documentElement; + + // Support: IE <=8 + // Assume HTML when documentElement doesn't yet exist, such as inside loading iframes + // https://bugs.jquery.com/ticket/4833 + return !rhtml.test( namespace || docElem && docElem.nodeName || "HTML" ); +}; + +/** + * Sets document-related variables once based on the current document + * @param {Element|Object} [doc] An element or document object to use to set the document + * @returns {Object} Returns the current document + */ +setDocument = Sizzle.setDocument = function( node ) { + var hasCompare, subWindow, + doc = node ? node.ownerDocument || node : preferredDoc; + + // Return early if doc is invalid or already selected + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( doc == document || doc.nodeType !== 9 || !doc.documentElement ) { + return document; + } + + // Update global variables + document = doc; + docElem = document.documentElement; + documentIsHTML = !isXML( document ); + + // Support: IE 9 - 11+, Edge 12 - 18+ + // Accessing iframe documents after unload throws "permission denied" errors (jQuery #13936) + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( preferredDoc != document && + ( subWindow = document.defaultView ) && subWindow.top !== subWindow ) { + + // Support: IE 11, Edge + if ( subWindow.addEventListener ) { + subWindow.addEventListener( "unload", unloadHandler, false ); + + // Support: IE 9 - 10 only + } else if ( subWindow.attachEvent ) { + subWindow.attachEvent( "onunload", unloadHandler ); + } + } + + // Support: IE 8 - 11+, Edge 12 - 18+, Chrome <=16 - 25 only, Firefox <=3.6 - 31 only, + // Safari 4 - 5 only, Opera <=11.6 - 12.x only + // IE/Edge & older browsers don't support the :scope pseudo-class. + // Support: Safari 6.0 only + // Safari 6.0 supports :scope but it's an alias of :root there. + support.scope = assert( function( el ) { + docElem.appendChild( el ).appendChild( document.createElement( "div" ) ); + return typeof el.querySelectorAll !== "undefined" && + !el.querySelectorAll( ":scope fieldset div" ).length; + } ); + + /* Attributes + ---------------------------------------------------------------------- */ + + // Support: IE<8 + // Verify that getAttribute really returns attributes and not properties + // (excepting IE8 booleans) + support.attributes = assert( function( el ) { + el.className = "i"; + return !el.getAttribute( "className" ); + } ); + + /* getElement(s)By* + ---------------------------------------------------------------------- */ + + // Check if getElementsByTagName("*") returns only elements + support.getElementsByTagName = assert( function( el ) { + el.appendChild( document.createComment( "" ) ); + return !el.getElementsByTagName( "*" ).length; + } ); + + // Support: IE<9 + support.getElementsByClassName = rnative.test( document.getElementsByClassName ); + + // Support: IE<10 + // Check if getElementById returns elements by name + // The broken getElementById methods don't pick up programmatically-set names, + // so use a roundabout getElementsByName test + support.getById = assert( function( el ) { + docElem.appendChild( el ).id = expando; + return !document.getElementsByName || !document.getElementsByName( expando ).length; + } ); + + // ID filter and find + if ( support.getById ) { + Expr.filter[ "ID" ] = function( id ) { + var attrId = id.replace( runescape, funescape ); + return function( elem ) { + return elem.getAttribute( "id" ) === attrId; + }; + }; + Expr.find[ "ID" ] = function( id, context ) { + if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { + var elem = context.getElementById( id ); + return elem ? [ elem ] : []; + } + }; + } else { + Expr.filter[ "ID" ] = function( id ) { + var attrId = id.replace( runescape, funescape ); + return function( elem ) { + var node = typeof elem.getAttributeNode !== "undefined" && + elem.getAttributeNode( "id" ); + return node && node.value === attrId; + }; + }; + + // Support: IE 6 - 7 only + // getElementById is not reliable as a find shortcut + Expr.find[ "ID" ] = function( id, context ) { + if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { + var node, i, elems, + elem = context.getElementById( id ); + + if ( elem ) { + + // Verify the id attribute + node = elem.getAttributeNode( "id" ); + if ( node && node.value === id ) { + return [ elem ]; + } + + // Fall back on getElementsByName + elems = context.getElementsByName( id ); + i = 0; + while ( ( elem = elems[ i++ ] ) ) { + node = elem.getAttributeNode( "id" ); + if ( node && node.value === id ) { + return [ elem ]; + } + } + } + + return []; + } + }; + } + + // Tag + Expr.find[ "TAG" ] = support.getElementsByTagName ? + function( tag, context ) { + if ( typeof context.getElementsByTagName !== "undefined" ) { + return context.getElementsByTagName( tag ); + + // DocumentFragment nodes don't have gEBTN + } else if ( support.qsa ) { + return context.querySelectorAll( tag ); + } + } : + + function( tag, context ) { + var elem, + tmp = [], + i = 0, + + // By happy coincidence, a (broken) gEBTN appears on DocumentFragment nodes too + results = context.getElementsByTagName( tag ); + + // Filter out possible comments + if ( tag === "*" ) { + while ( ( elem = results[ i++ ] ) ) { + if ( elem.nodeType === 1 ) { + tmp.push( elem ); + } + } + + return tmp; + } + return results; + }; + + // Class + Expr.find[ "CLASS" ] = support.getElementsByClassName && function( className, context ) { + if ( typeof context.getElementsByClassName !== "undefined" && documentIsHTML ) { + return context.getElementsByClassName( className ); + } + }; + + /* QSA/matchesSelector + ---------------------------------------------------------------------- */ + + // QSA and matchesSelector support + + // matchesSelector(:active) reports false when true (IE9/Opera 11.5) + rbuggyMatches = []; + + // qSa(:focus) reports false when true (Chrome 21) + // We allow this because of a bug in IE8/9 that throws an error + // whenever `document.activeElement` is accessed on an iframe + // So, we allow :focus to pass through QSA all the time to avoid the IE error + // See https://bugs.jquery.com/ticket/13378 + rbuggyQSA = []; + + if ( ( support.qsa = rnative.test( document.querySelectorAll ) ) ) { + + // Build QSA regex + // Regex strategy adopted from Diego Perini + assert( function( el ) { + + var input; + + // Select is set to empty string on purpose + // This is to test IE's treatment of not explicitly + // setting a boolean content attribute, + // since its presence should be enough + // https://bugs.jquery.com/ticket/12359 + docElem.appendChild( el ).innerHTML = "" + + ""; + + // Support: IE8, Opera 11-12.16 + // Nothing should be selected when empty strings follow ^= or $= or *= + // The test attribute must be unknown in Opera but "safe" for WinRT + // https://msdn.microsoft.com/en-us/library/ie/hh465388.aspx#attribute_section + if ( el.querySelectorAll( "[msallowcapture^='']" ).length ) { + rbuggyQSA.push( "[*^$]=" + whitespace + "*(?:''|\"\")" ); + } + + // Support: IE8 + // Boolean attributes and "value" are not treated correctly + if ( !el.querySelectorAll( "[selected]" ).length ) { + rbuggyQSA.push( "\\[" + whitespace + "*(?:value|" + booleans + ")" ); + } + + // Support: Chrome<29, Android<4.4, Safari<7.0+, iOS<7.0+, PhantomJS<1.9.8+ + if ( !el.querySelectorAll( "[id~=" + expando + "-]" ).length ) { + rbuggyQSA.push( "~=" ); + } + + // Support: IE 11+, Edge 15 - 18+ + // IE 11/Edge don't find elements on a `[name='']` query in some cases. + // Adding a temporary attribute to the document before the selection works + // around the issue. + // Interestingly, IE 10 & older don't seem to have the issue. + input = document.createElement( "input" ); + input.setAttribute( "name", "" ); + el.appendChild( input ); + if ( !el.querySelectorAll( "[name='']" ).length ) { + rbuggyQSA.push( "\\[" + whitespace + "*name" + whitespace + "*=" + + whitespace + "*(?:''|\"\")" ); + } + + // Webkit/Opera - :checked should return selected option elements + // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked + // IE8 throws error here and will not see later tests + if ( !el.querySelectorAll( ":checked" ).length ) { + rbuggyQSA.push( ":checked" ); + } + + // Support: Safari 8+, iOS 8+ + // https://bugs.webkit.org/show_bug.cgi?id=136851 + // In-page `selector#id sibling-combinator selector` fails + if ( !el.querySelectorAll( "a#" + expando + "+*" ).length ) { + rbuggyQSA.push( ".#.+[+~]" ); + } + + // Support: Firefox <=3.6 - 5 only + // Old Firefox doesn't throw on a badly-escaped identifier. + el.querySelectorAll( "\\\f" ); + rbuggyQSA.push( "[\\r\\n\\f]" ); + } ); + + assert( function( el ) { + el.innerHTML = "" + + ""; + + // Support: Windows 8 Native Apps + // The type and name attributes are restricted during .innerHTML assignment + var input = document.createElement( "input" ); + input.setAttribute( "type", "hidden" ); + el.appendChild( input ).setAttribute( "name", "D" ); + + // Support: IE8 + // Enforce case-sensitivity of name attribute + if ( el.querySelectorAll( "[name=d]" ).length ) { + rbuggyQSA.push( "name" + whitespace + "*[*^$|!~]?=" ); + } + + // FF 3.5 - :enabled/:disabled and hidden elements (hidden elements are still enabled) + // IE8 throws error here and will not see later tests + if ( el.querySelectorAll( ":enabled" ).length !== 2 ) { + rbuggyQSA.push( ":enabled", ":disabled" ); + } + + // Support: IE9-11+ + // IE's :disabled selector does not pick up the children of disabled fieldsets + docElem.appendChild( el ).disabled = true; + if ( el.querySelectorAll( ":disabled" ).length !== 2 ) { + rbuggyQSA.push( ":enabled", ":disabled" ); + } + + // Support: Opera 10 - 11 only + // Opera 10-11 does not throw on post-comma invalid pseudos + el.querySelectorAll( "*,:x" ); + rbuggyQSA.push( ",.*:" ); + } ); + } + + if ( ( support.matchesSelector = rnative.test( ( matches = docElem.matches || + docElem.webkitMatchesSelector || + docElem.mozMatchesSelector || + docElem.oMatchesSelector || + docElem.msMatchesSelector ) ) ) ) { + + assert( function( el ) { + + // Check to see if it's possible to do matchesSelector + // on a disconnected node (IE 9) + support.disconnectedMatch = matches.call( el, "*" ); + + // This should fail with an exception + // Gecko does not error, returns false instead + matches.call( el, "[s!='']:x" ); + rbuggyMatches.push( "!=", pseudos ); + } ); + } + + rbuggyQSA = rbuggyQSA.length && new RegExp( rbuggyQSA.join( "|" ) ); + rbuggyMatches = rbuggyMatches.length && new RegExp( rbuggyMatches.join( "|" ) ); + + /* Contains + ---------------------------------------------------------------------- */ + hasCompare = rnative.test( docElem.compareDocumentPosition ); + + // Element contains another + // Purposefully self-exclusive + // As in, an element does not contain itself + contains = hasCompare || rnative.test( docElem.contains ) ? + function( a, b ) { + var adown = a.nodeType === 9 ? a.documentElement : a, + bup = b && b.parentNode; + return a === bup || !!( bup && bup.nodeType === 1 && ( + adown.contains ? + adown.contains( bup ) : + a.compareDocumentPosition && a.compareDocumentPosition( bup ) & 16 + ) ); + } : + function( a, b ) { + if ( b ) { + while ( ( b = b.parentNode ) ) { + if ( b === a ) { + return true; + } + } + } + return false; + }; + + /* Sorting + ---------------------------------------------------------------------- */ + + // Document order sorting + sortOrder = hasCompare ? + function( a, b ) { + + // Flag for duplicate removal + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + // Sort on method existence if only one input has compareDocumentPosition + var compare = !a.compareDocumentPosition - !b.compareDocumentPosition; + if ( compare ) { + return compare; + } + + // Calculate position if both inputs belong to the same document + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + compare = ( a.ownerDocument || a ) == ( b.ownerDocument || b ) ? + a.compareDocumentPosition( b ) : + + // Otherwise we know they are disconnected + 1; + + // Disconnected nodes + if ( compare & 1 || + ( !support.sortDetached && b.compareDocumentPosition( a ) === compare ) ) { + + // Choose the first element that is related to our preferred document + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( a == document || a.ownerDocument == preferredDoc && + contains( preferredDoc, a ) ) { + return -1; + } + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( b == document || b.ownerDocument == preferredDoc && + contains( preferredDoc, b ) ) { + return 1; + } + + // Maintain original order + return sortInput ? + ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : + 0; + } + + return compare & 4 ? -1 : 1; + } : + function( a, b ) { + + // Exit early if the nodes are identical + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + var cur, + i = 0, + aup = a.parentNode, + bup = b.parentNode, + ap = [ a ], + bp = [ b ]; + + // Parentless nodes are either documents or disconnected + if ( !aup || !bup ) { + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + /* eslint-disable eqeqeq */ + return a == document ? -1 : + b == document ? 1 : + /* eslint-enable eqeqeq */ + aup ? -1 : + bup ? 1 : + sortInput ? + ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : + 0; + + // If the nodes are siblings, we can do a quick check + } else if ( aup === bup ) { + return siblingCheck( a, b ); + } + + // Otherwise we need full lists of their ancestors for comparison + cur = a; + while ( ( cur = cur.parentNode ) ) { + ap.unshift( cur ); + } + cur = b; + while ( ( cur = cur.parentNode ) ) { + bp.unshift( cur ); + } + + // Walk down the tree looking for a discrepancy + while ( ap[ i ] === bp[ i ] ) { + i++; + } + + return i ? + + // Do a sibling check if the nodes have a common ancestor + siblingCheck( ap[ i ], bp[ i ] ) : + + // Otherwise nodes in our document sort first + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + /* eslint-disable eqeqeq */ + ap[ i ] == preferredDoc ? -1 : + bp[ i ] == preferredDoc ? 1 : + /* eslint-enable eqeqeq */ + 0; + }; + + return document; +}; + +Sizzle.matches = function( expr, elements ) { + return Sizzle( expr, null, null, elements ); +}; + +Sizzle.matchesSelector = function( elem, expr ) { + setDocument( elem ); + + if ( support.matchesSelector && documentIsHTML && + !nonnativeSelectorCache[ expr + " " ] && + ( !rbuggyMatches || !rbuggyMatches.test( expr ) ) && + ( !rbuggyQSA || !rbuggyQSA.test( expr ) ) ) { + + try { + var ret = matches.call( elem, expr ); + + // IE 9's matchesSelector returns false on disconnected nodes + if ( ret || support.disconnectedMatch || + + // As well, disconnected nodes are said to be in a document + // fragment in IE 9 + elem.document && elem.document.nodeType !== 11 ) { + return ret; + } + } catch ( e ) { + nonnativeSelectorCache( expr, true ); + } + } + + return Sizzle( expr, document, null, [ elem ] ).length > 0; +}; + +Sizzle.contains = function( context, elem ) { + + // Set document vars if needed + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( ( context.ownerDocument || context ) != document ) { + setDocument( context ); + } + return contains( context, elem ); +}; + +Sizzle.attr = function( elem, name ) { + + // Set document vars if needed + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( ( elem.ownerDocument || elem ) != document ) { + setDocument( elem ); + } + + var fn = Expr.attrHandle[ name.toLowerCase() ], + + // Don't get fooled by Object.prototype properties (jQuery #13807) + val = fn && hasOwn.call( Expr.attrHandle, name.toLowerCase() ) ? + fn( elem, name, !documentIsHTML ) : + undefined; + + return val !== undefined ? + val : + support.attributes || !documentIsHTML ? + elem.getAttribute( name ) : + ( val = elem.getAttributeNode( name ) ) && val.specified ? + val.value : + null; +}; + +Sizzle.escape = function( sel ) { + return ( sel + "" ).replace( rcssescape, fcssescape ); +}; + +Sizzle.error = function( msg ) { + throw new Error( "Syntax error, unrecognized expression: " + msg ); +}; + +/** + * Document sorting and removing duplicates + * @param {ArrayLike} results + */ +Sizzle.uniqueSort = function( results ) { + var elem, + duplicates = [], + j = 0, + i = 0; + + // Unless we *know* we can detect duplicates, assume their presence + hasDuplicate = !support.detectDuplicates; + sortInput = !support.sortStable && results.slice( 0 ); + results.sort( sortOrder ); + + if ( hasDuplicate ) { + while ( ( elem = results[ i++ ] ) ) { + if ( elem === results[ i ] ) { + j = duplicates.push( i ); + } + } + while ( j-- ) { + results.splice( duplicates[ j ], 1 ); + } + } + + // Clear input after sorting to release objects + // See https://github.com/jquery/sizzle/pull/225 + sortInput = null; + + return results; +}; + +/** + * Utility function for retrieving the text value of an array of DOM nodes + * @param {Array|Element} elem + */ +getText = Sizzle.getText = function( elem ) { + var node, + ret = "", + i = 0, + nodeType = elem.nodeType; + + if ( !nodeType ) { + + // If no nodeType, this is expected to be an array + while ( ( node = elem[ i++ ] ) ) { + + // Do not traverse comment nodes + ret += getText( node ); + } + } else if ( nodeType === 1 || nodeType === 9 || nodeType === 11 ) { + + // Use textContent for elements + // innerText usage removed for consistency of new lines (jQuery #11153) + if ( typeof elem.textContent === "string" ) { + return elem.textContent; + } else { + + // Traverse its children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { + ret += getText( elem ); + } + } + } else if ( nodeType === 3 || nodeType === 4 ) { + return elem.nodeValue; + } + + // Do not include comment or processing instruction nodes + + return ret; +}; + +Expr = Sizzle.selectors = { + + // Can be adjusted by the user + cacheLength: 50, + + createPseudo: markFunction, + + match: matchExpr, + + attrHandle: {}, + + find: {}, + + relative: { + ">": { dir: "parentNode", first: true }, + " ": { dir: "parentNode" }, + "+": { dir: "previousSibling", first: true }, + "~": { dir: "previousSibling" } + }, + + preFilter: { + "ATTR": function( match ) { + match[ 1 ] = match[ 1 ].replace( runescape, funescape ); + + // Move the given value to match[3] whether quoted or unquoted + match[ 3 ] = ( match[ 3 ] || match[ 4 ] || + match[ 5 ] || "" ).replace( runescape, funescape ); + + if ( match[ 2 ] === "~=" ) { + match[ 3 ] = " " + match[ 3 ] + " "; + } + + return match.slice( 0, 4 ); + }, + + "CHILD": function( match ) { + + /* matches from matchExpr["CHILD"] + 1 type (only|nth|...) + 2 what (child|of-type) + 3 argument (even|odd|\d*|\d*n([+-]\d+)?|...) + 4 xn-component of xn+y argument ([+-]?\d*n|) + 5 sign of xn-component + 6 x of xn-component + 7 sign of y-component + 8 y of y-component + */ + match[ 1 ] = match[ 1 ].toLowerCase(); + + if ( match[ 1 ].slice( 0, 3 ) === "nth" ) { + + // nth-* requires argument + if ( !match[ 3 ] ) { + Sizzle.error( match[ 0 ] ); + } + + // numeric x and y parameters for Expr.filter.CHILD + // remember that false/true cast respectively to 0/1 + match[ 4 ] = +( match[ 4 ] ? + match[ 5 ] + ( match[ 6 ] || 1 ) : + 2 * ( match[ 3 ] === "even" || match[ 3 ] === "odd" ) ); + match[ 5 ] = +( ( match[ 7 ] + match[ 8 ] ) || match[ 3 ] === "odd" ); + + // other types prohibit arguments + } else if ( match[ 3 ] ) { + Sizzle.error( match[ 0 ] ); + } + + return match; + }, + + "PSEUDO": function( match ) { + var excess, + unquoted = !match[ 6 ] && match[ 2 ]; + + if ( matchExpr[ "CHILD" ].test( match[ 0 ] ) ) { + return null; + } + + // Accept quoted arguments as-is + if ( match[ 3 ] ) { + match[ 2 ] = match[ 4 ] || match[ 5 ] || ""; + + // Strip excess characters from unquoted arguments + } else if ( unquoted && rpseudo.test( unquoted ) && + + // Get excess from tokenize (recursively) + ( excess = tokenize( unquoted, true ) ) && + + // advance to the next closing parenthesis + ( excess = unquoted.indexOf( ")", unquoted.length - excess ) - unquoted.length ) ) { + + // excess is a negative index + match[ 0 ] = match[ 0 ].slice( 0, excess ); + match[ 2 ] = unquoted.slice( 0, excess ); + } + + // Return only captures needed by the pseudo filter method (type and argument) + return match.slice( 0, 3 ); + } + }, + + filter: { + + "TAG": function( nodeNameSelector ) { + var nodeName = nodeNameSelector.replace( runescape, funescape ).toLowerCase(); + return nodeNameSelector === "*" ? + function() { + return true; + } : + function( elem ) { + return elem.nodeName && elem.nodeName.toLowerCase() === nodeName; + }; + }, + + "CLASS": function( className ) { + var pattern = classCache[ className + " " ]; + + return pattern || + ( pattern = new RegExp( "(^|" + whitespace + + ")" + className + "(" + whitespace + "|$)" ) ) && classCache( + className, function( elem ) { + return pattern.test( + typeof elem.className === "string" && elem.className || + typeof elem.getAttribute !== "undefined" && + elem.getAttribute( "class" ) || + "" + ); + } ); + }, + + "ATTR": function( name, operator, check ) { + return function( elem ) { + var result = Sizzle.attr( elem, name ); + + if ( result == null ) { + return operator === "!="; + } + if ( !operator ) { + return true; + } + + result += ""; + + /* eslint-disable max-len */ + + return operator === "=" ? result === check : + operator === "!=" ? result !== check : + operator === "^=" ? check && result.indexOf( check ) === 0 : + operator === "*=" ? check && result.indexOf( check ) > -1 : + operator === "$=" ? check && result.slice( -check.length ) === check : + operator === "~=" ? ( " " + result.replace( rwhitespace, " " ) + " " ).indexOf( check ) > -1 : + operator === "|=" ? result === check || result.slice( 0, check.length + 1 ) === check + "-" : + false; + /* eslint-enable max-len */ + + }; + }, + + "CHILD": function( type, what, _argument, first, last ) { + var simple = type.slice( 0, 3 ) !== "nth", + forward = type.slice( -4 ) !== "last", + ofType = what === "of-type"; + + return first === 1 && last === 0 ? + + // Shortcut for :nth-*(n) + function( elem ) { + return !!elem.parentNode; + } : + + function( elem, _context, xml ) { + var cache, uniqueCache, outerCache, node, nodeIndex, start, + dir = simple !== forward ? "nextSibling" : "previousSibling", + parent = elem.parentNode, + name = ofType && elem.nodeName.toLowerCase(), + useCache = !xml && !ofType, + diff = false; + + if ( parent ) { + + // :(first|last|only)-(child|of-type) + if ( simple ) { + while ( dir ) { + node = elem; + while ( ( node = node[ dir ] ) ) { + if ( ofType ? + node.nodeName.toLowerCase() === name : + node.nodeType === 1 ) { + + return false; + } + } + + // Reverse direction for :only-* (if we haven't yet done so) + start = dir = type === "only" && !start && "nextSibling"; + } + return true; + } + + start = [ forward ? parent.firstChild : parent.lastChild ]; + + // non-xml :nth-child(...) stores cache data on `parent` + if ( forward && useCache ) { + + // Seek `elem` from a previously-cached index + + // ...in a gzip-friendly way + node = parent; + outerCache = node[ expando ] || ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + cache = uniqueCache[ type ] || []; + nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; + diff = nodeIndex && cache[ 2 ]; + node = nodeIndex && parent.childNodes[ nodeIndex ]; + + while ( ( node = ++nodeIndex && node && node[ dir ] || + + // Fallback to seeking `elem` from the start + ( diff = nodeIndex = 0 ) || start.pop() ) ) { + + // When found, cache indexes on `parent` and break + if ( node.nodeType === 1 && ++diff && node === elem ) { + uniqueCache[ type ] = [ dirruns, nodeIndex, diff ]; + break; + } + } + + } else { + + // Use previously-cached element index if available + if ( useCache ) { + + // ...in a gzip-friendly way + node = elem; + outerCache = node[ expando ] || ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + cache = uniqueCache[ type ] || []; + nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; + diff = nodeIndex; + } + + // xml :nth-child(...) + // or :nth-last-child(...) or :nth(-last)?-of-type(...) + if ( diff === false ) { + + // Use the same loop as above to seek `elem` from the start + while ( ( node = ++nodeIndex && node && node[ dir ] || + ( diff = nodeIndex = 0 ) || start.pop() ) ) { + + if ( ( ofType ? + node.nodeName.toLowerCase() === name : + node.nodeType === 1 ) && + ++diff ) { + + // Cache the index of each encountered element + if ( useCache ) { + outerCache = node[ expando ] || + ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + uniqueCache[ type ] = [ dirruns, diff ]; + } + + if ( node === elem ) { + break; + } + } + } + } + } + + // Incorporate the offset, then check against cycle size + diff -= last; + return diff === first || ( diff % first === 0 && diff / first >= 0 ); + } + }; + }, + + "PSEUDO": function( pseudo, argument ) { + + // pseudo-class names are case-insensitive + // http://www.w3.org/TR/selectors/#pseudo-classes + // Prioritize by case sensitivity in case custom pseudos are added with uppercase letters + // Remember that setFilters inherits from pseudos + var args, + fn = Expr.pseudos[ pseudo ] || Expr.setFilters[ pseudo.toLowerCase() ] || + Sizzle.error( "unsupported pseudo: " + pseudo ); + + // The user may use createPseudo to indicate that + // arguments are needed to create the filter function + // just as Sizzle does + if ( fn[ expando ] ) { + return fn( argument ); + } + + // But maintain support for old signatures + if ( fn.length > 1 ) { + args = [ pseudo, pseudo, "", argument ]; + return Expr.setFilters.hasOwnProperty( pseudo.toLowerCase() ) ? + markFunction( function( seed, matches ) { + var idx, + matched = fn( seed, argument ), + i = matched.length; + while ( i-- ) { + idx = indexOf( seed, matched[ i ] ); + seed[ idx ] = !( matches[ idx ] = matched[ i ] ); + } + } ) : + function( elem ) { + return fn( elem, 0, args ); + }; + } + + return fn; + } + }, + + pseudos: { + + // Potentially complex pseudos + "not": markFunction( function( selector ) { + + // Trim the selector passed to compile + // to avoid treating leading and trailing + // spaces as combinators + var input = [], + results = [], + matcher = compile( selector.replace( rtrim, "$1" ) ); + + return matcher[ expando ] ? + markFunction( function( seed, matches, _context, xml ) { + var elem, + unmatched = matcher( seed, null, xml, [] ), + i = seed.length; + + // Match elements unmatched by `matcher` + while ( i-- ) { + if ( ( elem = unmatched[ i ] ) ) { + seed[ i ] = !( matches[ i ] = elem ); + } + } + } ) : + function( elem, _context, xml ) { + input[ 0 ] = elem; + matcher( input, null, xml, results ); + + // Don't keep the element (issue #299) + input[ 0 ] = null; + return !results.pop(); + }; + } ), + + "has": markFunction( function( selector ) { + return function( elem ) { + return Sizzle( selector, elem ).length > 0; + }; + } ), + + "contains": markFunction( function( text ) { + text = text.replace( runescape, funescape ); + return function( elem ) { + return ( elem.textContent || getText( elem ) ).indexOf( text ) > -1; + }; + } ), + + // "Whether an element is represented by a :lang() selector + // is based solely on the element's language value + // being equal to the identifier C, + // or beginning with the identifier C immediately followed by "-". + // The matching of C against the element's language value is performed case-insensitively. + // The identifier C does not have to be a valid language name." + // http://www.w3.org/TR/selectors/#lang-pseudo + "lang": markFunction( function( lang ) { + + // lang value must be a valid identifier + if ( !ridentifier.test( lang || "" ) ) { + Sizzle.error( "unsupported lang: " + lang ); + } + lang = lang.replace( runescape, funescape ).toLowerCase(); + return function( elem ) { + var elemLang; + do { + if ( ( elemLang = documentIsHTML ? + elem.lang : + elem.getAttribute( "xml:lang" ) || elem.getAttribute( "lang" ) ) ) { + + elemLang = elemLang.toLowerCase(); + return elemLang === lang || elemLang.indexOf( lang + "-" ) === 0; + } + } while ( ( elem = elem.parentNode ) && elem.nodeType === 1 ); + return false; + }; + } ), + + // Miscellaneous + "target": function( elem ) { + var hash = window.location && window.location.hash; + return hash && hash.slice( 1 ) === elem.id; + }, + + "root": function( elem ) { + return elem === docElem; + }, + + "focus": function( elem ) { + return elem === document.activeElement && + ( !document.hasFocus || document.hasFocus() ) && + !!( elem.type || elem.href || ~elem.tabIndex ); + }, + + // Boolean properties + "enabled": createDisabledPseudo( false ), + "disabled": createDisabledPseudo( true ), + + "checked": function( elem ) { + + // In CSS3, :checked should return both checked and selected elements + // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked + var nodeName = elem.nodeName.toLowerCase(); + return ( nodeName === "input" && !!elem.checked ) || + ( nodeName === "option" && !!elem.selected ); + }, + + "selected": function( elem ) { + + // Accessing this property makes selected-by-default + // options in Safari work properly + if ( elem.parentNode ) { + // eslint-disable-next-line no-unused-expressions + elem.parentNode.selectedIndex; + } + + return elem.selected === true; + }, + + // Contents + "empty": function( elem ) { + + // http://www.w3.org/TR/selectors/#empty-pseudo + // :empty is negated by element (1) or content nodes (text: 3; cdata: 4; entity ref: 5), + // but not by others (comment: 8; processing instruction: 7; etc.) + // nodeType < 6 works because attributes (2) do not appear as children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { + if ( elem.nodeType < 6 ) { + return false; + } + } + return true; + }, + + "parent": function( elem ) { + return !Expr.pseudos[ "empty" ]( elem ); + }, + + // Element/input types + "header": function( elem ) { + return rheader.test( elem.nodeName ); + }, + + "input": function( elem ) { + return rinputs.test( elem.nodeName ); + }, + + "button": function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && elem.type === "button" || name === "button"; + }, + + "text": function( elem ) { + var attr; + return elem.nodeName.toLowerCase() === "input" && + elem.type === "text" && + + // Support: IE<8 + // New HTML5 attribute values (e.g., "search") appear with elem.type === "text" + ( ( attr = elem.getAttribute( "type" ) ) == null || + attr.toLowerCase() === "text" ); + }, + + // Position-in-collection + "first": createPositionalPseudo( function() { + return [ 0 ]; + } ), + + "last": createPositionalPseudo( function( _matchIndexes, length ) { + return [ length - 1 ]; + } ), + + "eq": createPositionalPseudo( function( _matchIndexes, length, argument ) { + return [ argument < 0 ? argument + length : argument ]; + } ), + + "even": createPositionalPseudo( function( matchIndexes, length ) { + var i = 0; + for ( ; i < length; i += 2 ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "odd": createPositionalPseudo( function( matchIndexes, length ) { + var i = 1; + for ( ; i < length; i += 2 ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "lt": createPositionalPseudo( function( matchIndexes, length, argument ) { + var i = argument < 0 ? + argument + length : + argument > length ? + length : + argument; + for ( ; --i >= 0; ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "gt": createPositionalPseudo( function( matchIndexes, length, argument ) { + var i = argument < 0 ? argument + length : argument; + for ( ; ++i < length; ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ) + } +}; + +Expr.pseudos[ "nth" ] = Expr.pseudos[ "eq" ]; + +// Add button/input type pseudos +for ( i in { radio: true, checkbox: true, file: true, password: true, image: true } ) { + Expr.pseudos[ i ] = createInputPseudo( i ); +} +for ( i in { submit: true, reset: true } ) { + Expr.pseudos[ i ] = createButtonPseudo( i ); +} + +// Easy API for creating new setFilters +function setFilters() {} +setFilters.prototype = Expr.filters = Expr.pseudos; +Expr.setFilters = new setFilters(); + +tokenize = Sizzle.tokenize = function( selector, parseOnly ) { + var matched, match, tokens, type, + soFar, groups, preFilters, + cached = tokenCache[ selector + " " ]; + + if ( cached ) { + return parseOnly ? 0 : cached.slice( 0 ); + } + + soFar = selector; + groups = []; + preFilters = Expr.preFilter; + + while ( soFar ) { + + // Comma and first run + if ( !matched || ( match = rcomma.exec( soFar ) ) ) { + if ( match ) { + + // Don't consume trailing commas as valid + soFar = soFar.slice( match[ 0 ].length ) || soFar; + } + groups.push( ( tokens = [] ) ); + } + + matched = false; + + // Combinators + if ( ( match = rcombinators.exec( soFar ) ) ) { + matched = match.shift(); + tokens.push( { + value: matched, + + // Cast descendant combinators to space + type: match[ 0 ].replace( rtrim, " " ) + } ); + soFar = soFar.slice( matched.length ); + } + + // Filters + for ( type in Expr.filter ) { + if ( ( match = matchExpr[ type ].exec( soFar ) ) && ( !preFilters[ type ] || + ( match = preFilters[ type ]( match ) ) ) ) { + matched = match.shift(); + tokens.push( { + value: matched, + type: type, + matches: match + } ); + soFar = soFar.slice( matched.length ); + } + } + + if ( !matched ) { + break; + } + } + + // Return the length of the invalid excess + // if we're just parsing + // Otherwise, throw an error or return tokens + return parseOnly ? + soFar.length : + soFar ? + Sizzle.error( selector ) : + + // Cache the tokens + tokenCache( selector, groups ).slice( 0 ); +}; + +function toSelector( tokens ) { + var i = 0, + len = tokens.length, + selector = ""; + for ( ; i < len; i++ ) { + selector += tokens[ i ].value; + } + return selector; +} + +function addCombinator( matcher, combinator, base ) { + var dir = combinator.dir, + skip = combinator.next, + key = skip || dir, + checkNonElements = base && key === "parentNode", + doneName = done++; + + return combinator.first ? + + // Check against closest ancestor/preceding element + function( elem, context, xml ) { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + return matcher( elem, context, xml ); + } + } + return false; + } : + + // Check against all ancestor/preceding elements + function( elem, context, xml ) { + var oldCache, uniqueCache, outerCache, + newCache = [ dirruns, doneName ]; + + // We can't set arbitrary data on XML nodes, so they don't benefit from combinator caching + if ( xml ) { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + if ( matcher( elem, context, xml ) ) { + return true; + } + } + } + } else { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + outerCache = elem[ expando ] || ( elem[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ elem.uniqueID ] || + ( outerCache[ elem.uniqueID ] = {} ); + + if ( skip && skip === elem.nodeName.toLowerCase() ) { + elem = elem[ dir ] || elem; + } else if ( ( oldCache = uniqueCache[ key ] ) && + oldCache[ 0 ] === dirruns && oldCache[ 1 ] === doneName ) { + + // Assign to newCache so results back-propagate to previous elements + return ( newCache[ 2 ] = oldCache[ 2 ] ); + } else { + + // Reuse newcache so results back-propagate to previous elements + uniqueCache[ key ] = newCache; + + // A match means we're done; a fail means we have to keep checking + if ( ( newCache[ 2 ] = matcher( elem, context, xml ) ) ) { + return true; + } + } + } + } + } + return false; + }; +} + +function elementMatcher( matchers ) { + return matchers.length > 1 ? + function( elem, context, xml ) { + var i = matchers.length; + while ( i-- ) { + if ( !matchers[ i ]( elem, context, xml ) ) { + return false; + } + } + return true; + } : + matchers[ 0 ]; +} + +function multipleContexts( selector, contexts, results ) { + var i = 0, + len = contexts.length; + for ( ; i < len; i++ ) { + Sizzle( selector, contexts[ i ], results ); + } + return results; +} + +function condense( unmatched, map, filter, context, xml ) { + var elem, + newUnmatched = [], + i = 0, + len = unmatched.length, + mapped = map != null; + + for ( ; i < len; i++ ) { + if ( ( elem = unmatched[ i ] ) ) { + if ( !filter || filter( elem, context, xml ) ) { + newUnmatched.push( elem ); + if ( mapped ) { + map.push( i ); + } + } + } + } + + return newUnmatched; +} + +function setMatcher( preFilter, selector, matcher, postFilter, postFinder, postSelector ) { + if ( postFilter && !postFilter[ expando ] ) { + postFilter = setMatcher( postFilter ); + } + if ( postFinder && !postFinder[ expando ] ) { + postFinder = setMatcher( postFinder, postSelector ); + } + return markFunction( function( seed, results, context, xml ) { + var temp, i, elem, + preMap = [], + postMap = [], + preexisting = results.length, + + // Get initial elements from seed or context + elems = seed || multipleContexts( + selector || "*", + context.nodeType ? [ context ] : context, + [] + ), + + // Prefilter to get matcher input, preserving a map for seed-results synchronization + matcherIn = preFilter && ( seed || !selector ) ? + condense( elems, preMap, preFilter, context, xml ) : + elems, + + matcherOut = matcher ? + + // If we have a postFinder, or filtered seed, or non-seed postFilter or preexisting results, + postFinder || ( seed ? preFilter : preexisting || postFilter ) ? + + // ...intermediate processing is necessary + [] : + + // ...otherwise use results directly + results : + matcherIn; + + // Find primary matches + if ( matcher ) { + matcher( matcherIn, matcherOut, context, xml ); + } + + // Apply postFilter + if ( postFilter ) { + temp = condense( matcherOut, postMap ); + postFilter( temp, [], context, xml ); + + // Un-match failing elements by moving them back to matcherIn + i = temp.length; + while ( i-- ) { + if ( ( elem = temp[ i ] ) ) { + matcherOut[ postMap[ i ] ] = !( matcherIn[ postMap[ i ] ] = elem ); + } + } + } + + if ( seed ) { + if ( postFinder || preFilter ) { + if ( postFinder ) { + + // Get the final matcherOut by condensing this intermediate into postFinder contexts + temp = []; + i = matcherOut.length; + while ( i-- ) { + if ( ( elem = matcherOut[ i ] ) ) { + + // Restore matcherIn since elem is not yet a final match + temp.push( ( matcherIn[ i ] = elem ) ); + } + } + postFinder( null, ( matcherOut = [] ), temp, xml ); + } + + // Move matched elements from seed to results to keep them synchronized + i = matcherOut.length; + while ( i-- ) { + if ( ( elem = matcherOut[ i ] ) && + ( temp = postFinder ? indexOf( seed, elem ) : preMap[ i ] ) > -1 ) { + + seed[ temp ] = !( results[ temp ] = elem ); + } + } + } + + // Add elements to results, through postFinder if defined + } else { + matcherOut = condense( + matcherOut === results ? + matcherOut.splice( preexisting, matcherOut.length ) : + matcherOut + ); + if ( postFinder ) { + postFinder( null, results, matcherOut, xml ); + } else { + push.apply( results, matcherOut ); + } + } + } ); +} + +function matcherFromTokens( tokens ) { + var checkContext, matcher, j, + len = tokens.length, + leadingRelative = Expr.relative[ tokens[ 0 ].type ], + implicitRelative = leadingRelative || Expr.relative[ " " ], + i = leadingRelative ? 1 : 0, + + // The foundational matcher ensures that elements are reachable from top-level context(s) + matchContext = addCombinator( function( elem ) { + return elem === checkContext; + }, implicitRelative, true ), + matchAnyContext = addCombinator( function( elem ) { + return indexOf( checkContext, elem ) > -1; + }, implicitRelative, true ), + matchers = [ function( elem, context, xml ) { + var ret = ( !leadingRelative && ( xml || context !== outermostContext ) ) || ( + ( checkContext = context ).nodeType ? + matchContext( elem, context, xml ) : + matchAnyContext( elem, context, xml ) ); + + // Avoid hanging onto element (issue #299) + checkContext = null; + return ret; + } ]; + + for ( ; i < len; i++ ) { + if ( ( matcher = Expr.relative[ tokens[ i ].type ] ) ) { + matchers = [ addCombinator( elementMatcher( matchers ), matcher ) ]; + } else { + matcher = Expr.filter[ tokens[ i ].type ].apply( null, tokens[ i ].matches ); + + // Return special upon seeing a positional matcher + if ( matcher[ expando ] ) { + + // Find the next relative operator (if any) for proper handling + j = ++i; + for ( ; j < len; j++ ) { + if ( Expr.relative[ tokens[ j ].type ] ) { + break; + } + } + return setMatcher( + i > 1 && elementMatcher( matchers ), + i > 1 && toSelector( + + // If the preceding token was a descendant combinator, insert an implicit any-element `*` + tokens + .slice( 0, i - 1 ) + .concat( { value: tokens[ i - 2 ].type === " " ? "*" : "" } ) + ).replace( rtrim, "$1" ), + matcher, + i < j && matcherFromTokens( tokens.slice( i, j ) ), + j < len && matcherFromTokens( ( tokens = tokens.slice( j ) ) ), + j < len && toSelector( tokens ) + ); + } + matchers.push( matcher ); + } + } + + return elementMatcher( matchers ); +} + +function matcherFromGroupMatchers( elementMatchers, setMatchers ) { + var bySet = setMatchers.length > 0, + byElement = elementMatchers.length > 0, + superMatcher = function( seed, context, xml, results, outermost ) { + var elem, j, matcher, + matchedCount = 0, + i = "0", + unmatched = seed && [], + setMatched = [], + contextBackup = outermostContext, + + // We must always have either seed elements or outermost context + elems = seed || byElement && Expr.find[ "TAG" ]( "*", outermost ), + + // Use integer dirruns iff this is the outermost matcher + dirrunsUnique = ( dirruns += contextBackup == null ? 1 : Math.random() || 0.1 ), + len = elems.length; + + if ( outermost ) { + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + outermostContext = context == document || context || outermost; + } + + // Add elements passing elementMatchers directly to results + // Support: IE<9, Safari + // Tolerate NodeList properties (IE: "length"; Safari: ) matching elements by id + for ( ; i !== len && ( elem = elems[ i ] ) != null; i++ ) { + if ( byElement && elem ) { + j = 0; + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( !context && elem.ownerDocument != document ) { + setDocument( elem ); + xml = !documentIsHTML; + } + while ( ( matcher = elementMatchers[ j++ ] ) ) { + if ( matcher( elem, context || document, xml ) ) { + results.push( elem ); + break; + } + } + if ( outermost ) { + dirruns = dirrunsUnique; + } + } + + // Track unmatched elements for set filters + if ( bySet ) { + + // They will have gone through all possible matchers + if ( ( elem = !matcher && elem ) ) { + matchedCount--; + } + + // Lengthen the array for every element, matched or not + if ( seed ) { + unmatched.push( elem ); + } + } + } + + // `i` is now the count of elements visited above, and adding it to `matchedCount` + // makes the latter nonnegative. + matchedCount += i; + + // Apply set filters to unmatched elements + // NOTE: This can be skipped if there are no unmatched elements (i.e., `matchedCount` + // equals `i`), unless we didn't visit _any_ elements in the above loop because we have + // no element matchers and no seed. + // Incrementing an initially-string "0" `i` allows `i` to remain a string only in that + // case, which will result in a "00" `matchedCount` that differs from `i` but is also + // numerically zero. + if ( bySet && i !== matchedCount ) { + j = 0; + while ( ( matcher = setMatchers[ j++ ] ) ) { + matcher( unmatched, setMatched, context, xml ); + } + + if ( seed ) { + + // Reintegrate element matches to eliminate the need for sorting + if ( matchedCount > 0 ) { + while ( i-- ) { + if ( !( unmatched[ i ] || setMatched[ i ] ) ) { + setMatched[ i ] = pop.call( results ); + } + } + } + + // Discard index placeholder values to get only actual matches + setMatched = condense( setMatched ); + } + + // Add matches to results + push.apply( results, setMatched ); + + // Seedless set matches succeeding multiple successful matchers stipulate sorting + if ( outermost && !seed && setMatched.length > 0 && + ( matchedCount + setMatchers.length ) > 1 ) { + + Sizzle.uniqueSort( results ); + } + } + + // Override manipulation of globals by nested matchers + if ( outermost ) { + dirruns = dirrunsUnique; + outermostContext = contextBackup; + } + + return unmatched; + }; + + return bySet ? + markFunction( superMatcher ) : + superMatcher; +} + +compile = Sizzle.compile = function( selector, match /* Internal Use Only */ ) { + var i, + setMatchers = [], + elementMatchers = [], + cached = compilerCache[ selector + " " ]; + + if ( !cached ) { + + // Generate a function of recursive functions that can be used to check each element + if ( !match ) { + match = tokenize( selector ); + } + i = match.length; + while ( i-- ) { + cached = matcherFromTokens( match[ i ] ); + if ( cached[ expando ] ) { + setMatchers.push( cached ); + } else { + elementMatchers.push( cached ); + } + } + + // Cache the compiled function + cached = compilerCache( + selector, + matcherFromGroupMatchers( elementMatchers, setMatchers ) + ); + + // Save selector and tokenization + cached.selector = selector; + } + return cached; +}; + +/** + * A low-level selection function that works with Sizzle's compiled + * selector functions + * @param {String|Function} selector A selector or a pre-compiled + * selector function built with Sizzle.compile + * @param {Element} context + * @param {Array} [results] + * @param {Array} [seed] A set of elements to match against + */ +select = Sizzle.select = function( selector, context, results, seed ) { + var i, tokens, token, type, find, + compiled = typeof selector === "function" && selector, + match = !seed && tokenize( ( selector = compiled.selector || selector ) ); + + results = results || []; + + // Try to minimize operations if there is only one selector in the list and no seed + // (the latter of which guarantees us context) + if ( match.length === 1 ) { + + // Reduce context if the leading compound selector is an ID + tokens = match[ 0 ] = match[ 0 ].slice( 0 ); + if ( tokens.length > 2 && ( token = tokens[ 0 ] ).type === "ID" && + context.nodeType === 9 && documentIsHTML && Expr.relative[ tokens[ 1 ].type ] ) { + + context = ( Expr.find[ "ID" ]( token.matches[ 0 ] + .replace( runescape, funescape ), context ) || [] )[ 0 ]; + if ( !context ) { + return results; + + // Precompiled matchers will still verify ancestry, so step up a level + } else if ( compiled ) { + context = context.parentNode; + } + + selector = selector.slice( tokens.shift().value.length ); + } + + // Fetch a seed set for right-to-left matching + i = matchExpr[ "needsContext" ].test( selector ) ? 0 : tokens.length; + while ( i-- ) { + token = tokens[ i ]; + + // Abort if we hit a combinator + if ( Expr.relative[ ( type = token.type ) ] ) { + break; + } + if ( ( find = Expr.find[ type ] ) ) { + + // Search, expanding context for leading sibling combinators + if ( ( seed = find( + token.matches[ 0 ].replace( runescape, funescape ), + rsibling.test( tokens[ 0 ].type ) && testContext( context.parentNode ) || + context + ) ) ) { + + // If seed is empty or no tokens remain, we can return early + tokens.splice( i, 1 ); + selector = seed.length && toSelector( tokens ); + if ( !selector ) { + push.apply( results, seed ); + return results; + } + + break; + } + } + } + } + + // Compile and execute a filtering function if one is not provided + // Provide `match` to avoid retokenization if we modified the selector above + ( compiled || compile( selector, match ) )( + seed, + context, + !documentIsHTML, + results, + !context || rsibling.test( selector ) && testContext( context.parentNode ) || context + ); + return results; +}; + +// One-time assignments + +// Sort stability +support.sortStable = expando.split( "" ).sort( sortOrder ).join( "" ) === expando; + +// Support: Chrome 14-35+ +// Always assume duplicates if they aren't passed to the comparison function +support.detectDuplicates = !!hasDuplicate; + +// Initialize against the default document +setDocument(); + +// Support: Webkit<537.32 - Safari 6.0.3/Chrome 25 (fixed in Chrome 27) +// Detached nodes confoundingly follow *each other* +support.sortDetached = assert( function( el ) { + + // Should return 1, but returns 4 (following) + return el.compareDocumentPosition( document.createElement( "fieldset" ) ) & 1; +} ); + +// Support: IE<8 +// Prevent attribute/property "interpolation" +// https://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx +if ( !assert( function( el ) { + el.innerHTML = ""; + return el.firstChild.getAttribute( "href" ) === "#"; +} ) ) { + addHandle( "type|href|height|width", function( elem, name, isXML ) { + if ( !isXML ) { + return elem.getAttribute( name, name.toLowerCase() === "type" ? 1 : 2 ); + } + } ); +} + +// Support: IE<9 +// Use defaultValue in place of getAttribute("value") +if ( !support.attributes || !assert( function( el ) { + el.innerHTML = ""; + el.firstChild.setAttribute( "value", "" ); + return el.firstChild.getAttribute( "value" ) === ""; +} ) ) { + addHandle( "value", function( elem, _name, isXML ) { + if ( !isXML && elem.nodeName.toLowerCase() === "input" ) { + return elem.defaultValue; + } + } ); +} + +// Support: IE<9 +// Use getAttributeNode to fetch booleans when getAttribute lies +if ( !assert( function( el ) { + return el.getAttribute( "disabled" ) == null; +} ) ) { + addHandle( booleans, function( elem, name, isXML ) { + var val; + if ( !isXML ) { + return elem[ name ] === true ? name.toLowerCase() : + ( val = elem.getAttributeNode( name ) ) && val.specified ? + val.value : + null; + } + } ); +} + +return Sizzle; + +} )( window ); + + + +jQuery.find = Sizzle; +jQuery.expr = Sizzle.selectors; + +// Deprecated +jQuery.expr[ ":" ] = jQuery.expr.pseudos; +jQuery.uniqueSort = jQuery.unique = Sizzle.uniqueSort; +jQuery.text = Sizzle.getText; +jQuery.isXMLDoc = Sizzle.isXML; +jQuery.contains = Sizzle.contains; +jQuery.escapeSelector = Sizzle.escape; + + + + +var dir = function( elem, dir, until ) { + var matched = [], + truncate = until !== undefined; + + while ( ( elem = elem[ dir ] ) && elem.nodeType !== 9 ) { + if ( elem.nodeType === 1 ) { + if ( truncate && jQuery( elem ).is( until ) ) { + break; + } + matched.push( elem ); + } + } + return matched; +}; + + +var siblings = function( n, elem ) { + var matched = []; + + for ( ; n; n = n.nextSibling ) { + if ( n.nodeType === 1 && n !== elem ) { + matched.push( n ); + } + } + + return matched; +}; + + +var rneedsContext = jQuery.expr.match.needsContext; + + + +function nodeName( elem, name ) { + + return elem.nodeName && elem.nodeName.toLowerCase() === name.toLowerCase(); + +} +var rsingleTag = ( /^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i ); + + + +// Implement the identical functionality for filter and not +function winnow( elements, qualifier, not ) { + if ( isFunction( qualifier ) ) { + return jQuery.grep( elements, function( elem, i ) { + return !!qualifier.call( elem, i, elem ) !== not; + } ); + } + + // Single element + if ( qualifier.nodeType ) { + return jQuery.grep( elements, function( elem ) { + return ( elem === qualifier ) !== not; + } ); + } + + // Arraylike of elements (jQuery, arguments, Array) + if ( typeof qualifier !== "string" ) { + return jQuery.grep( elements, function( elem ) { + return ( indexOf.call( qualifier, elem ) > -1 ) !== not; + } ); + } + + // Filtered directly for both simple and complex selectors + return jQuery.filter( qualifier, elements, not ); +} + +jQuery.filter = function( expr, elems, not ) { + var elem = elems[ 0 ]; + + if ( not ) { + expr = ":not(" + expr + ")"; + } + + if ( elems.length === 1 && elem.nodeType === 1 ) { + return jQuery.find.matchesSelector( elem, expr ) ? [ elem ] : []; + } + + return jQuery.find.matches( expr, jQuery.grep( elems, function( elem ) { + return elem.nodeType === 1; + } ) ); +}; + +jQuery.fn.extend( { + find: function( selector ) { + var i, ret, + len = this.length, + self = this; + + if ( typeof selector !== "string" ) { + return this.pushStack( jQuery( selector ).filter( function() { + for ( i = 0; i < len; i++ ) { + if ( jQuery.contains( self[ i ], this ) ) { + return true; + } + } + } ) ); + } + + ret = this.pushStack( [] ); + + for ( i = 0; i < len; i++ ) { + jQuery.find( selector, self[ i ], ret ); + } + + return len > 1 ? jQuery.uniqueSort( ret ) : ret; + }, + filter: function( selector ) { + return this.pushStack( winnow( this, selector || [], false ) ); + }, + not: function( selector ) { + return this.pushStack( winnow( this, selector || [], true ) ); + }, + is: function( selector ) { + return !!winnow( + this, + + // If this is a positional/relative selector, check membership in the returned set + // so $("p:first").is("p:last") won't return true for a doc with two "p". + typeof selector === "string" && rneedsContext.test( selector ) ? + jQuery( selector ) : + selector || [], + false + ).length; + } +} ); + + +// Initialize a jQuery object + + +// A central reference to the root jQuery(document) +var rootjQuery, + + // A simple way to check for HTML strings + // Prioritize #id over to avoid XSS via location.hash (#9521) + // Strict HTML recognition (#11290: must start with <) + // Shortcut simple #id case for speed + rquickExpr = /^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/, + + init = jQuery.fn.init = function( selector, context, root ) { + var match, elem; + + // HANDLE: $(""), $(null), $(undefined), $(false) + if ( !selector ) { + return this; + } + + // Method init() accepts an alternate rootjQuery + // so migrate can support jQuery.sub (gh-2101) + root = root || rootjQuery; + + // Handle HTML strings + if ( typeof selector === "string" ) { + if ( selector[ 0 ] === "<" && + selector[ selector.length - 1 ] === ">" && + selector.length >= 3 ) { + + // Assume that strings that start and end with <> are HTML and skip the regex check + match = [ null, selector, null ]; + + } else { + match = rquickExpr.exec( selector ); + } + + // Match html or make sure no context is specified for #id + if ( match && ( match[ 1 ] || !context ) ) { + + // HANDLE: $(html) -> $(array) + if ( match[ 1 ] ) { + context = context instanceof jQuery ? context[ 0 ] : context; + + // Option to run scripts is true for back-compat + // Intentionally let the error be thrown if parseHTML is not present + jQuery.merge( this, jQuery.parseHTML( + match[ 1 ], + context && context.nodeType ? context.ownerDocument || context : document, + true + ) ); + + // HANDLE: $(html, props) + if ( rsingleTag.test( match[ 1 ] ) && jQuery.isPlainObject( context ) ) { + for ( match in context ) { + + // Properties of context are called as methods if possible + if ( isFunction( this[ match ] ) ) { + this[ match ]( context[ match ] ); + + // ...and otherwise set as attributes + } else { + this.attr( match, context[ match ] ); + } + } + } + + return this; + + // HANDLE: $(#id) + } else { + elem = document.getElementById( match[ 2 ] ); + + if ( elem ) { + + // Inject the element directly into the jQuery object + this[ 0 ] = elem; + this.length = 1; + } + return this; + } + + // HANDLE: $(expr, $(...)) + } else if ( !context || context.jquery ) { + return ( context || root ).find( selector ); + + // HANDLE: $(expr, context) + // (which is just equivalent to: $(context).find(expr) + } else { + return this.constructor( context ).find( selector ); + } + + // HANDLE: $(DOMElement) + } else if ( selector.nodeType ) { + this[ 0 ] = selector; + this.length = 1; + return this; + + // HANDLE: $(function) + // Shortcut for document ready + } else if ( isFunction( selector ) ) { + return root.ready !== undefined ? + root.ready( selector ) : + + // Execute immediately if ready is not present + selector( jQuery ); + } + + return jQuery.makeArray( selector, this ); + }; + +// Give the init function the jQuery prototype for later instantiation +init.prototype = jQuery.fn; + +// Initialize central reference +rootjQuery = jQuery( document ); + + +var rparentsprev = /^(?:parents|prev(?:Until|All))/, + + // Methods guaranteed to produce a unique set when starting from a unique set + guaranteedUnique = { + children: true, + contents: true, + next: true, + prev: true + }; + +jQuery.fn.extend( { + has: function( target ) { + var targets = jQuery( target, this ), + l = targets.length; + + return this.filter( function() { + var i = 0; + for ( ; i < l; i++ ) { + if ( jQuery.contains( this, targets[ i ] ) ) { + return true; + } + } + } ); + }, + + closest: function( selectors, context ) { + var cur, + i = 0, + l = this.length, + matched = [], + targets = typeof selectors !== "string" && jQuery( selectors ); + + // Positional selectors never match, since there's no _selection_ context + if ( !rneedsContext.test( selectors ) ) { + for ( ; i < l; i++ ) { + for ( cur = this[ i ]; cur && cur !== context; cur = cur.parentNode ) { + + // Always skip document fragments + if ( cur.nodeType < 11 && ( targets ? + targets.index( cur ) > -1 : + + // Don't pass non-elements to Sizzle + cur.nodeType === 1 && + jQuery.find.matchesSelector( cur, selectors ) ) ) { + + matched.push( cur ); + break; + } + } + } + } + + return this.pushStack( matched.length > 1 ? jQuery.uniqueSort( matched ) : matched ); + }, + + // Determine the position of an element within the set + index: function( elem ) { + + // No argument, return index in parent + if ( !elem ) { + return ( this[ 0 ] && this[ 0 ].parentNode ) ? this.first().prevAll().length : -1; + } + + // Index in selector + if ( typeof elem === "string" ) { + return indexOf.call( jQuery( elem ), this[ 0 ] ); + } + + // Locate the position of the desired element + return indexOf.call( this, + + // If it receives a jQuery object, the first element is used + elem.jquery ? elem[ 0 ] : elem + ); + }, + + add: function( selector, context ) { + return this.pushStack( + jQuery.uniqueSort( + jQuery.merge( this.get(), jQuery( selector, context ) ) + ) + ); + }, + + addBack: function( selector ) { + return this.add( selector == null ? + this.prevObject : this.prevObject.filter( selector ) + ); + } +} ); + +function sibling( cur, dir ) { + while ( ( cur = cur[ dir ] ) && cur.nodeType !== 1 ) {} + return cur; +} + +jQuery.each( { + parent: function( elem ) { + var parent = elem.parentNode; + return parent && parent.nodeType !== 11 ? parent : null; + }, + parents: function( elem ) { + return dir( elem, "parentNode" ); + }, + parentsUntil: function( elem, _i, until ) { + return dir( elem, "parentNode", until ); + }, + next: function( elem ) { + return sibling( elem, "nextSibling" ); + }, + prev: function( elem ) { + return sibling( elem, "previousSibling" ); + }, + nextAll: function( elem ) { + return dir( elem, "nextSibling" ); + }, + prevAll: function( elem ) { + return dir( elem, "previousSibling" ); + }, + nextUntil: function( elem, _i, until ) { + return dir( elem, "nextSibling", until ); + }, + prevUntil: function( elem, _i, until ) { + return dir( elem, "previousSibling", until ); + }, + siblings: function( elem ) { + return siblings( ( elem.parentNode || {} ).firstChild, elem ); + }, + children: function( elem ) { + return siblings( elem.firstChild ); + }, + contents: function( elem ) { + if ( elem.contentDocument != null && + + // Support: IE 11+ + // elements with no `data` attribute has an object + // `contentDocument` with a `null` prototype. + getProto( elem.contentDocument ) ) { + + return elem.contentDocument; + } + + // Support: IE 9 - 11 only, iOS 7 only, Android Browser <=4.3 only + // Treat the template element as a regular one in browsers that + // don't support it. + if ( nodeName( elem, "template" ) ) { + elem = elem.content || elem; + } + + return jQuery.merge( [], elem.childNodes ); + } +}, function( name, fn ) { + jQuery.fn[ name ] = function( until, selector ) { + var matched = jQuery.map( this, fn, until ); + + if ( name.slice( -5 ) !== "Until" ) { + selector = until; + } + + if ( selector && typeof selector === "string" ) { + matched = jQuery.filter( selector, matched ); + } + + if ( this.length > 1 ) { + + // Remove duplicates + if ( !guaranteedUnique[ name ] ) { + jQuery.uniqueSort( matched ); + } + + // Reverse order for parents* and prev-derivatives + if ( rparentsprev.test( name ) ) { + matched.reverse(); + } + } + + return this.pushStack( matched ); + }; +} ); +var rnothtmlwhite = ( /[^\x20\t\r\n\f]+/g ); + + + +// Convert String-formatted options into Object-formatted ones +function createOptions( options ) { + var object = {}; + jQuery.each( options.match( rnothtmlwhite ) || [], function( _, flag ) { + object[ flag ] = true; + } ); + return object; +} + +/* + * Create a callback list using the following parameters: + * + * options: an optional list of space-separated options that will change how + * the callback list behaves or a more traditional option object + * + * By default a callback list will act like an event callback list and can be + * "fired" multiple times. + * + * Possible options: + * + * once: will ensure the callback list can only be fired once (like a Deferred) + * + * memory: will keep track of previous values and will call any callback added + * after the list has been fired right away with the latest "memorized" + * values (like a Deferred) + * + * unique: will ensure a callback can only be added once (no duplicate in the list) + * + * stopOnFalse: interrupt callings when a callback returns false + * + */ +jQuery.Callbacks = function( options ) { + + // Convert options from String-formatted to Object-formatted if needed + // (we check in cache first) + options = typeof options === "string" ? + createOptions( options ) : + jQuery.extend( {}, options ); + + var // Flag to know if list is currently firing + firing, + + // Last fire value for non-forgettable lists + memory, + + // Flag to know if list was already fired + fired, + + // Flag to prevent firing + locked, + + // Actual callback list + list = [], + + // Queue of execution data for repeatable lists + queue = [], + + // Index of currently firing callback (modified by add/remove as needed) + firingIndex = -1, + + // Fire callbacks + fire = function() { + + // Enforce single-firing + locked = locked || options.once; + + // Execute callbacks for all pending executions, + // respecting firingIndex overrides and runtime changes + fired = firing = true; + for ( ; queue.length; firingIndex = -1 ) { + memory = queue.shift(); + while ( ++firingIndex < list.length ) { + + // Run callback and check for early termination + if ( list[ firingIndex ].apply( memory[ 0 ], memory[ 1 ] ) === false && + options.stopOnFalse ) { + + // Jump to end and forget the data so .add doesn't re-fire + firingIndex = list.length; + memory = false; + } + } + } + + // Forget the data if we're done with it + if ( !options.memory ) { + memory = false; + } + + firing = false; + + // Clean up if we're done firing for good + if ( locked ) { + + // Keep an empty list if we have data for future add calls + if ( memory ) { + list = []; + + // Otherwise, this object is spent + } else { + list = ""; + } + } + }, + + // Actual Callbacks object + self = { + + // Add a callback or a collection of callbacks to the list + add: function() { + if ( list ) { + + // If we have memory from a past run, we should fire after adding + if ( memory && !firing ) { + firingIndex = list.length - 1; + queue.push( memory ); + } + + ( function add( args ) { + jQuery.each( args, function( _, arg ) { + if ( isFunction( arg ) ) { + if ( !options.unique || !self.has( arg ) ) { + list.push( arg ); + } + } else if ( arg && arg.length && toType( arg ) !== "string" ) { + + // Inspect recursively + add( arg ); + } + } ); + } )( arguments ); + + if ( memory && !firing ) { + fire(); + } + } + return this; + }, + + // Remove a callback from the list + remove: function() { + jQuery.each( arguments, function( _, arg ) { + var index; + while ( ( index = jQuery.inArray( arg, list, index ) ) > -1 ) { + list.splice( index, 1 ); + + // Handle firing indexes + if ( index <= firingIndex ) { + firingIndex--; + } + } + } ); + return this; + }, + + // Check if a given callback is in the list. + // If no argument is given, return whether or not list has callbacks attached. + has: function( fn ) { + return fn ? + jQuery.inArray( fn, list ) > -1 : + list.length > 0; + }, + + // Remove all callbacks from the list + empty: function() { + if ( list ) { + list = []; + } + return this; + }, + + // Disable .fire and .add + // Abort any current/pending executions + // Clear all callbacks and values + disable: function() { + locked = queue = []; + list = memory = ""; + return this; + }, + disabled: function() { + return !list; + }, + + // Disable .fire + // Also disable .add unless we have memory (since it would have no effect) + // Abort any pending executions + lock: function() { + locked = queue = []; + if ( !memory && !firing ) { + list = memory = ""; + } + return this; + }, + locked: function() { + return !!locked; + }, + + // Call all callbacks with the given context and arguments + fireWith: function( context, args ) { + if ( !locked ) { + args = args || []; + args = [ context, args.slice ? args.slice() : args ]; + queue.push( args ); + if ( !firing ) { + fire(); + } + } + return this; + }, + + // Call all the callbacks with the given arguments + fire: function() { + self.fireWith( this, arguments ); + return this; + }, + + // To know if the callbacks have already been called at least once + fired: function() { + return !!fired; + } + }; + + return self; +}; + + +function Identity( v ) { + return v; +} +function Thrower( ex ) { + throw ex; +} + +function adoptValue( value, resolve, reject, noValue ) { + var method; + + try { + + // Check for promise aspect first to privilege synchronous behavior + if ( value && isFunction( ( method = value.promise ) ) ) { + method.call( value ).done( resolve ).fail( reject ); + + // Other thenables + } else if ( value && isFunction( ( method = value.then ) ) ) { + method.call( value, resolve, reject ); + + // Other non-thenables + } else { + + // Control `resolve` arguments by letting Array#slice cast boolean `noValue` to integer: + // * false: [ value ].slice( 0 ) => resolve( value ) + // * true: [ value ].slice( 1 ) => resolve() + resolve.apply( undefined, [ value ].slice( noValue ) ); + } + + // For Promises/A+, convert exceptions into rejections + // Since jQuery.when doesn't unwrap thenables, we can skip the extra checks appearing in + // Deferred#then to conditionally suppress rejection. + } catch ( value ) { + + // Support: Android 4.0 only + // Strict mode functions invoked without .call/.apply get global-object context + reject.apply( undefined, [ value ] ); + } +} + +jQuery.extend( { + + Deferred: function( func ) { + var tuples = [ + + // action, add listener, callbacks, + // ... .then handlers, argument index, [final state] + [ "notify", "progress", jQuery.Callbacks( "memory" ), + jQuery.Callbacks( "memory" ), 2 ], + [ "resolve", "done", jQuery.Callbacks( "once memory" ), + jQuery.Callbacks( "once memory" ), 0, "resolved" ], + [ "reject", "fail", jQuery.Callbacks( "once memory" ), + jQuery.Callbacks( "once memory" ), 1, "rejected" ] + ], + state = "pending", + promise = { + state: function() { + return state; + }, + always: function() { + deferred.done( arguments ).fail( arguments ); + return this; + }, + "catch": function( fn ) { + return promise.then( null, fn ); + }, + + // Keep pipe for back-compat + pipe: function( /* fnDone, fnFail, fnProgress */ ) { + var fns = arguments; + + return jQuery.Deferred( function( newDefer ) { + jQuery.each( tuples, function( _i, tuple ) { + + // Map tuples (progress, done, fail) to arguments (done, fail, progress) + var fn = isFunction( fns[ tuple[ 4 ] ] ) && fns[ tuple[ 4 ] ]; + + // deferred.progress(function() { bind to newDefer or newDefer.notify }) + // deferred.done(function() { bind to newDefer or newDefer.resolve }) + // deferred.fail(function() { bind to newDefer or newDefer.reject }) + deferred[ tuple[ 1 ] ]( function() { + var returned = fn && fn.apply( this, arguments ); + if ( returned && isFunction( returned.promise ) ) { + returned.promise() + .progress( newDefer.notify ) + .done( newDefer.resolve ) + .fail( newDefer.reject ); + } else { + newDefer[ tuple[ 0 ] + "With" ]( + this, + fn ? [ returned ] : arguments + ); + } + } ); + } ); + fns = null; + } ).promise(); + }, + then: function( onFulfilled, onRejected, onProgress ) { + var maxDepth = 0; + function resolve( depth, deferred, handler, special ) { + return function() { + var that = this, + args = arguments, + mightThrow = function() { + var returned, then; + + // Support: Promises/A+ section 2.3.3.3.3 + // https://promisesaplus.com/#point-59 + // Ignore double-resolution attempts + if ( depth < maxDepth ) { + return; + } + + returned = handler.apply( that, args ); + + // Support: Promises/A+ section 2.3.1 + // https://promisesaplus.com/#point-48 + if ( returned === deferred.promise() ) { + throw new TypeError( "Thenable self-resolution" ); + } + + // Support: Promises/A+ sections 2.3.3.1, 3.5 + // https://promisesaplus.com/#point-54 + // https://promisesaplus.com/#point-75 + // Retrieve `then` only once + then = returned && + + // Support: Promises/A+ section 2.3.4 + // https://promisesaplus.com/#point-64 + // Only check objects and functions for thenability + ( typeof returned === "object" || + typeof returned === "function" ) && + returned.then; + + // Handle a returned thenable + if ( isFunction( then ) ) { + + // Special processors (notify) just wait for resolution + if ( special ) { + then.call( + returned, + resolve( maxDepth, deferred, Identity, special ), + resolve( maxDepth, deferred, Thrower, special ) + ); + + // Normal processors (resolve) also hook into progress + } else { + + // ...and disregard older resolution values + maxDepth++; + + then.call( + returned, + resolve( maxDepth, deferred, Identity, special ), + resolve( maxDepth, deferred, Thrower, special ), + resolve( maxDepth, deferred, Identity, + deferred.notifyWith ) + ); + } + + // Handle all other returned values + } else { + + // Only substitute handlers pass on context + // and multiple values (non-spec behavior) + if ( handler !== Identity ) { + that = undefined; + args = [ returned ]; + } + + // Process the value(s) + // Default process is resolve + ( special || deferred.resolveWith )( that, args ); + } + }, + + // Only normal processors (resolve) catch and reject exceptions + process = special ? + mightThrow : + function() { + try { + mightThrow(); + } catch ( e ) { + + if ( jQuery.Deferred.exceptionHook ) { + jQuery.Deferred.exceptionHook( e, + process.stackTrace ); + } + + // Support: Promises/A+ section 2.3.3.3.4.1 + // https://promisesaplus.com/#point-61 + // Ignore post-resolution exceptions + if ( depth + 1 >= maxDepth ) { + + // Only substitute handlers pass on context + // and multiple values (non-spec behavior) + if ( handler !== Thrower ) { + that = undefined; + args = [ e ]; + } + + deferred.rejectWith( that, args ); + } + } + }; + + // Support: Promises/A+ section 2.3.3.3.1 + // https://promisesaplus.com/#point-57 + // Re-resolve promises immediately to dodge false rejection from + // subsequent errors + if ( depth ) { + process(); + } else { + + // Call an optional hook to record the stack, in case of exception + // since it's otherwise lost when execution goes async + if ( jQuery.Deferred.getStackHook ) { + process.stackTrace = jQuery.Deferred.getStackHook(); + } + window.setTimeout( process ); + } + }; + } + + return jQuery.Deferred( function( newDefer ) { + + // progress_handlers.add( ... ) + tuples[ 0 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onProgress ) ? + onProgress : + Identity, + newDefer.notifyWith + ) + ); + + // fulfilled_handlers.add( ... ) + tuples[ 1 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onFulfilled ) ? + onFulfilled : + Identity + ) + ); + + // rejected_handlers.add( ... ) + tuples[ 2 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onRejected ) ? + onRejected : + Thrower + ) + ); + } ).promise(); + }, + + // Get a promise for this deferred + // If obj is provided, the promise aspect is added to the object + promise: function( obj ) { + return obj != null ? jQuery.extend( obj, promise ) : promise; + } + }, + deferred = {}; + + // Add list-specific methods + jQuery.each( tuples, function( i, tuple ) { + var list = tuple[ 2 ], + stateString = tuple[ 5 ]; + + // promise.progress = list.add + // promise.done = list.add + // promise.fail = list.add + promise[ tuple[ 1 ] ] = list.add; + + // Handle state + if ( stateString ) { + list.add( + function() { + + // state = "resolved" (i.e., fulfilled) + // state = "rejected" + state = stateString; + }, + + // rejected_callbacks.disable + // fulfilled_callbacks.disable + tuples[ 3 - i ][ 2 ].disable, + + // rejected_handlers.disable + // fulfilled_handlers.disable + tuples[ 3 - i ][ 3 ].disable, + + // progress_callbacks.lock + tuples[ 0 ][ 2 ].lock, + + // progress_handlers.lock + tuples[ 0 ][ 3 ].lock + ); + } + + // progress_handlers.fire + // fulfilled_handlers.fire + // rejected_handlers.fire + list.add( tuple[ 3 ].fire ); + + // deferred.notify = function() { deferred.notifyWith(...) } + // deferred.resolve = function() { deferred.resolveWith(...) } + // deferred.reject = function() { deferred.rejectWith(...) } + deferred[ tuple[ 0 ] ] = function() { + deferred[ tuple[ 0 ] + "With" ]( this === deferred ? undefined : this, arguments ); + return this; + }; + + // deferred.notifyWith = list.fireWith + // deferred.resolveWith = list.fireWith + // deferred.rejectWith = list.fireWith + deferred[ tuple[ 0 ] + "With" ] = list.fireWith; + } ); + + // Make the deferred a promise + promise.promise( deferred ); + + // Call given func if any + if ( func ) { + func.call( deferred, deferred ); + } + + // All done! + return deferred; + }, + + // Deferred helper + when: function( singleValue ) { + var + + // count of uncompleted subordinates + remaining = arguments.length, + + // count of unprocessed arguments + i = remaining, + + // subordinate fulfillment data + resolveContexts = Array( i ), + resolveValues = slice.call( arguments ), + + // the primary Deferred + primary = jQuery.Deferred(), + + // subordinate callback factory + updateFunc = function( i ) { + return function( value ) { + resolveContexts[ i ] = this; + resolveValues[ i ] = arguments.length > 1 ? slice.call( arguments ) : value; + if ( !( --remaining ) ) { + primary.resolveWith( resolveContexts, resolveValues ); + } + }; + }; + + // Single- and empty arguments are adopted like Promise.resolve + if ( remaining <= 1 ) { + adoptValue( singleValue, primary.done( updateFunc( i ) ).resolve, primary.reject, + !remaining ); + + // Use .then() to unwrap secondary thenables (cf. gh-3000) + if ( primary.state() === "pending" || + isFunction( resolveValues[ i ] && resolveValues[ i ].then ) ) { + + return primary.then(); + } + } + + // Multiple arguments are aggregated like Promise.all array elements + while ( i-- ) { + adoptValue( resolveValues[ i ], updateFunc( i ), primary.reject ); + } + + return primary.promise(); + } +} ); + + +// These usually indicate a programmer mistake during development, +// warn about them ASAP rather than swallowing them by default. +var rerrorNames = /^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/; + +jQuery.Deferred.exceptionHook = function( error, stack ) { + + // Support: IE 8 - 9 only + // Console exists when dev tools are open, which can happen at any time + if ( window.console && window.console.warn && error && rerrorNames.test( error.name ) ) { + window.console.warn( "jQuery.Deferred exception: " + error.message, error.stack, stack ); + } +}; + + + + +jQuery.readyException = function( error ) { + window.setTimeout( function() { + throw error; + } ); +}; + + + + +// The deferred used on DOM ready +var readyList = jQuery.Deferred(); + +jQuery.fn.ready = function( fn ) { + + readyList + .then( fn ) + + // Wrap jQuery.readyException in a function so that the lookup + // happens at the time of error handling instead of callback + // registration. + .catch( function( error ) { + jQuery.readyException( error ); + } ); + + return this; +}; + +jQuery.extend( { + + // Is the DOM ready to be used? Set to true once it occurs. + isReady: false, + + // A counter to track how many items to wait for before + // the ready event fires. See #6781 + readyWait: 1, + + // Handle when the DOM is ready + ready: function( wait ) { + + // Abort if there are pending holds or we're already ready + if ( wait === true ? --jQuery.readyWait : jQuery.isReady ) { + return; + } + + // Remember that the DOM is ready + jQuery.isReady = true; + + // If a normal DOM Ready event fired, decrement, and wait if need be + if ( wait !== true && --jQuery.readyWait > 0 ) { + return; + } + + // If there are functions bound, to execute + readyList.resolveWith( document, [ jQuery ] ); + } +} ); + +jQuery.ready.then = readyList.then; + +// The ready event handler and self cleanup method +function completed() { + document.removeEventListener( "DOMContentLoaded", completed ); + window.removeEventListener( "load", completed ); + jQuery.ready(); +} + +// Catch cases where $(document).ready() is called +// after the browser event has already occurred. +// Support: IE <=9 - 10 only +// Older IE sometimes signals "interactive" too soon +if ( document.readyState === "complete" || + ( document.readyState !== "loading" && !document.documentElement.doScroll ) ) { + + // Handle it asynchronously to allow scripts the opportunity to delay ready + window.setTimeout( jQuery.ready ); + +} else { + + // Use the handy event callback + document.addEventListener( "DOMContentLoaded", completed ); + + // A fallback to window.onload, that will always work + window.addEventListener( "load", completed ); +} + + + + +// Multifunctional method to get and set values of a collection +// The value/s can optionally be executed if it's a function +var access = function( elems, fn, key, value, chainable, emptyGet, raw ) { + var i = 0, + len = elems.length, + bulk = key == null; + + // Sets many values + if ( toType( key ) === "object" ) { + chainable = true; + for ( i in key ) { + access( elems, fn, i, key[ i ], true, emptyGet, raw ); + } + + // Sets one value + } else if ( value !== undefined ) { + chainable = true; + + if ( !isFunction( value ) ) { + raw = true; + } + + if ( bulk ) { + + // Bulk operations run against the entire set + if ( raw ) { + fn.call( elems, value ); + fn = null; + + // ...except when executing function values + } else { + bulk = fn; + fn = function( elem, _key, value ) { + return bulk.call( jQuery( elem ), value ); + }; + } + } + + if ( fn ) { + for ( ; i < len; i++ ) { + fn( + elems[ i ], key, raw ? + value : + value.call( elems[ i ], i, fn( elems[ i ], key ) ) + ); + } + } + } + + if ( chainable ) { + return elems; + } + + // Gets + if ( bulk ) { + return fn.call( elems ); + } + + return len ? fn( elems[ 0 ], key ) : emptyGet; +}; + + +// Matches dashed string for camelizing +var rmsPrefix = /^-ms-/, + rdashAlpha = /-([a-z])/g; + +// Used by camelCase as callback to replace() +function fcamelCase( _all, letter ) { + return letter.toUpperCase(); +} + +// Convert dashed to camelCase; used by the css and data modules +// Support: IE <=9 - 11, Edge 12 - 15 +// Microsoft forgot to hump their vendor prefix (#9572) +function camelCase( string ) { + return string.replace( rmsPrefix, "ms-" ).replace( rdashAlpha, fcamelCase ); +} +var acceptData = function( owner ) { + + // Accepts only: + // - Node + // - Node.ELEMENT_NODE + // - Node.DOCUMENT_NODE + // - Object + // - Any + return owner.nodeType === 1 || owner.nodeType === 9 || !( +owner.nodeType ); +}; + + + + +function Data() { + this.expando = jQuery.expando + Data.uid++; +} + +Data.uid = 1; + +Data.prototype = { + + cache: function( owner ) { + + // Check if the owner object already has a cache + var value = owner[ this.expando ]; + + // If not, create one + if ( !value ) { + value = {}; + + // We can accept data for non-element nodes in modern browsers, + // but we should not, see #8335. + // Always return an empty object. + if ( acceptData( owner ) ) { + + // If it is a node unlikely to be stringify-ed or looped over + // use plain assignment + if ( owner.nodeType ) { + owner[ this.expando ] = value; + + // Otherwise secure it in a non-enumerable property + // configurable must be true to allow the property to be + // deleted when data is removed + } else { + Object.defineProperty( owner, this.expando, { + value: value, + configurable: true + } ); + } + } + } + + return value; + }, + set: function( owner, data, value ) { + var prop, + cache = this.cache( owner ); + + // Handle: [ owner, key, value ] args + // Always use camelCase key (gh-2257) + if ( typeof data === "string" ) { + cache[ camelCase( data ) ] = value; + + // Handle: [ owner, { properties } ] args + } else { + + // Copy the properties one-by-one to the cache object + for ( prop in data ) { + cache[ camelCase( prop ) ] = data[ prop ]; + } + } + return cache; + }, + get: function( owner, key ) { + return key === undefined ? + this.cache( owner ) : + + // Always use camelCase key (gh-2257) + owner[ this.expando ] && owner[ this.expando ][ camelCase( key ) ]; + }, + access: function( owner, key, value ) { + + // In cases where either: + // + // 1. No key was specified + // 2. A string key was specified, but no value provided + // + // Take the "read" path and allow the get method to determine + // which value to return, respectively either: + // + // 1. The entire cache object + // 2. The data stored at the key + // + if ( key === undefined || + ( ( key && typeof key === "string" ) && value === undefined ) ) { + + return this.get( owner, key ); + } + + // When the key is not a string, or both a key and value + // are specified, set or extend (existing objects) with either: + // + // 1. An object of properties + // 2. A key and value + // + this.set( owner, key, value ); + + // Since the "set" path can have two possible entry points + // return the expected data based on which path was taken[*] + return value !== undefined ? value : key; + }, + remove: function( owner, key ) { + var i, + cache = owner[ this.expando ]; + + if ( cache === undefined ) { + return; + } + + if ( key !== undefined ) { + + // Support array or space separated string of keys + if ( Array.isArray( key ) ) { + + // If key is an array of keys... + // We always set camelCase keys, so remove that. + key = key.map( camelCase ); + } else { + key = camelCase( key ); + + // If a key with the spaces exists, use it. + // Otherwise, create an array by matching non-whitespace + key = key in cache ? + [ key ] : + ( key.match( rnothtmlwhite ) || [] ); + } + + i = key.length; + + while ( i-- ) { + delete cache[ key[ i ] ]; + } + } + + // Remove the expando if there's no more data + if ( key === undefined || jQuery.isEmptyObject( cache ) ) { + + // Support: Chrome <=35 - 45 + // Webkit & Blink performance suffers when deleting properties + // from DOM nodes, so set to undefined instead + // https://bugs.chromium.org/p/chromium/issues/detail?id=378607 (bug restricted) + if ( owner.nodeType ) { + owner[ this.expando ] = undefined; + } else { + delete owner[ this.expando ]; + } + } + }, + hasData: function( owner ) { + var cache = owner[ this.expando ]; + return cache !== undefined && !jQuery.isEmptyObject( cache ); + } +}; +var dataPriv = new Data(); + +var dataUser = new Data(); + + + +// Implementation Summary +// +// 1. Enforce API surface and semantic compatibility with 1.9.x branch +// 2. Improve the module's maintainability by reducing the storage +// paths to a single mechanism. +// 3. Use the same single mechanism to support "private" and "user" data. +// 4. _Never_ expose "private" data to user code (TODO: Drop _data, _removeData) +// 5. Avoid exposing implementation details on user objects (eg. expando properties) +// 6. Provide a clear path for implementation upgrade to WeakMap in 2014 + +var rbrace = /^(?:\{[\w\W]*\}|\[[\w\W]*\])$/, + rmultiDash = /[A-Z]/g; + +function getData( data ) { + if ( data === "true" ) { + return true; + } + + if ( data === "false" ) { + return false; + } + + if ( data === "null" ) { + return null; + } + + // Only convert to a number if it doesn't change the string + if ( data === +data + "" ) { + return +data; + } + + if ( rbrace.test( data ) ) { + return JSON.parse( data ); + } + + return data; +} + +function dataAttr( elem, key, data ) { + var name; + + // If nothing was found internally, try to fetch any + // data from the HTML5 data-* attribute + if ( data === undefined && elem.nodeType === 1 ) { + name = "data-" + key.replace( rmultiDash, "-$&" ).toLowerCase(); + data = elem.getAttribute( name ); + + if ( typeof data === "string" ) { + try { + data = getData( data ); + } catch ( e ) {} + + // Make sure we set the data so it isn't changed later + dataUser.set( elem, key, data ); + } else { + data = undefined; + } + } + return data; +} + +jQuery.extend( { + hasData: function( elem ) { + return dataUser.hasData( elem ) || dataPriv.hasData( elem ); + }, + + data: function( elem, name, data ) { + return dataUser.access( elem, name, data ); + }, + + removeData: function( elem, name ) { + dataUser.remove( elem, name ); + }, + + // TODO: Now that all calls to _data and _removeData have been replaced + // with direct calls to dataPriv methods, these can be deprecated. + _data: function( elem, name, data ) { + return dataPriv.access( elem, name, data ); + }, + + _removeData: function( elem, name ) { + dataPriv.remove( elem, name ); + } +} ); + +jQuery.fn.extend( { + data: function( key, value ) { + var i, name, data, + elem = this[ 0 ], + attrs = elem && elem.attributes; + + // Gets all values + if ( key === undefined ) { + if ( this.length ) { + data = dataUser.get( elem ); + + if ( elem.nodeType === 1 && !dataPriv.get( elem, "hasDataAttrs" ) ) { + i = attrs.length; + while ( i-- ) { + + // Support: IE 11 only + // The attrs elements can be null (#14894) + if ( attrs[ i ] ) { + name = attrs[ i ].name; + if ( name.indexOf( "data-" ) === 0 ) { + name = camelCase( name.slice( 5 ) ); + dataAttr( elem, name, data[ name ] ); + } + } + } + dataPriv.set( elem, "hasDataAttrs", true ); + } + } + + return data; + } + + // Sets multiple values + if ( typeof key === "object" ) { + return this.each( function() { + dataUser.set( this, key ); + } ); + } + + return access( this, function( value ) { + var data; + + // The calling jQuery object (element matches) is not empty + // (and therefore has an element appears at this[ 0 ]) and the + // `value` parameter was not undefined. An empty jQuery object + // will result in `undefined` for elem = this[ 0 ] which will + // throw an exception if an attempt to read a data cache is made. + if ( elem && value === undefined ) { + + // Attempt to get data from the cache + // The key will always be camelCased in Data + data = dataUser.get( elem, key ); + if ( data !== undefined ) { + return data; + } + + // Attempt to "discover" the data in + // HTML5 custom data-* attrs + data = dataAttr( elem, key ); + if ( data !== undefined ) { + return data; + } + + // We tried really hard, but the data doesn't exist. + return; + } + + // Set the data... + this.each( function() { + + // We always store the camelCased key + dataUser.set( this, key, value ); + } ); + }, null, value, arguments.length > 1, null, true ); + }, + + removeData: function( key ) { + return this.each( function() { + dataUser.remove( this, key ); + } ); + } +} ); + + +jQuery.extend( { + queue: function( elem, type, data ) { + var queue; + + if ( elem ) { + type = ( type || "fx" ) + "queue"; + queue = dataPriv.get( elem, type ); + + // Speed up dequeue by getting out quickly if this is just a lookup + if ( data ) { + if ( !queue || Array.isArray( data ) ) { + queue = dataPriv.access( elem, type, jQuery.makeArray( data ) ); + } else { + queue.push( data ); + } + } + return queue || []; + } + }, + + dequeue: function( elem, type ) { + type = type || "fx"; + + var queue = jQuery.queue( elem, type ), + startLength = queue.length, + fn = queue.shift(), + hooks = jQuery._queueHooks( elem, type ), + next = function() { + jQuery.dequeue( elem, type ); + }; + + // If the fx queue is dequeued, always remove the progress sentinel + if ( fn === "inprogress" ) { + fn = queue.shift(); + startLength--; + } + + if ( fn ) { + + // Add a progress sentinel to prevent the fx queue from being + // automatically dequeued + if ( type === "fx" ) { + queue.unshift( "inprogress" ); + } + + // Clear up the last queue stop function + delete hooks.stop; + fn.call( elem, next, hooks ); + } + + if ( !startLength && hooks ) { + hooks.empty.fire(); + } + }, + + // Not public - generate a queueHooks object, or return the current one + _queueHooks: function( elem, type ) { + var key = type + "queueHooks"; + return dataPriv.get( elem, key ) || dataPriv.access( elem, key, { + empty: jQuery.Callbacks( "once memory" ).add( function() { + dataPriv.remove( elem, [ type + "queue", key ] ); + } ) + } ); + } +} ); + +jQuery.fn.extend( { + queue: function( type, data ) { + var setter = 2; + + if ( typeof type !== "string" ) { + data = type; + type = "fx"; + setter--; + } + + if ( arguments.length < setter ) { + return jQuery.queue( this[ 0 ], type ); + } + + return data === undefined ? + this : + this.each( function() { + var queue = jQuery.queue( this, type, data ); + + // Ensure a hooks for this queue + jQuery._queueHooks( this, type ); + + if ( type === "fx" && queue[ 0 ] !== "inprogress" ) { + jQuery.dequeue( this, type ); + } + } ); + }, + dequeue: function( type ) { + return this.each( function() { + jQuery.dequeue( this, type ); + } ); + }, + clearQueue: function( type ) { + return this.queue( type || "fx", [] ); + }, + + // Get a promise resolved when queues of a certain type + // are emptied (fx is the type by default) + promise: function( type, obj ) { + var tmp, + count = 1, + defer = jQuery.Deferred(), + elements = this, + i = this.length, + resolve = function() { + if ( !( --count ) ) { + defer.resolveWith( elements, [ elements ] ); + } + }; + + if ( typeof type !== "string" ) { + obj = type; + type = undefined; + } + type = type || "fx"; + + while ( i-- ) { + tmp = dataPriv.get( elements[ i ], type + "queueHooks" ); + if ( tmp && tmp.empty ) { + count++; + tmp.empty.add( resolve ); + } + } + resolve(); + return defer.promise( obj ); + } +} ); +var pnum = ( /[+-]?(?:\d*\.|)\d+(?:[eE][+-]?\d+|)/ ).source; + +var rcssNum = new RegExp( "^(?:([+-])=|)(" + pnum + ")([a-z%]*)$", "i" ); + + +var cssExpand = [ "Top", "Right", "Bottom", "Left" ]; + +var documentElement = document.documentElement; + + + + var isAttached = function( elem ) { + return jQuery.contains( elem.ownerDocument, elem ); + }, + composed = { composed: true }; + + // Support: IE 9 - 11+, Edge 12 - 18+, iOS 10.0 - 10.2 only + // Check attachment across shadow DOM boundaries when possible (gh-3504) + // Support: iOS 10.0-10.2 only + // Early iOS 10 versions support `attachShadow` but not `getRootNode`, + // leading to errors. We need to check for `getRootNode`. + if ( documentElement.getRootNode ) { + isAttached = function( elem ) { + return jQuery.contains( elem.ownerDocument, elem ) || + elem.getRootNode( composed ) === elem.ownerDocument; + }; + } +var isHiddenWithinTree = function( elem, el ) { + + // isHiddenWithinTree might be called from jQuery#filter function; + // in that case, element will be second argument + elem = el || elem; + + // Inline style trumps all + return elem.style.display === "none" || + elem.style.display === "" && + + // Otherwise, check computed style + // Support: Firefox <=43 - 45 + // Disconnected elements can have computed display: none, so first confirm that elem is + // in the document. + isAttached( elem ) && + + jQuery.css( elem, "display" ) === "none"; + }; + + + +function adjustCSS( elem, prop, valueParts, tween ) { + var adjusted, scale, + maxIterations = 20, + currentValue = tween ? + function() { + return tween.cur(); + } : + function() { + return jQuery.css( elem, prop, "" ); + }, + initial = currentValue(), + unit = valueParts && valueParts[ 3 ] || ( jQuery.cssNumber[ prop ] ? "" : "px" ), + + // Starting value computation is required for potential unit mismatches + initialInUnit = elem.nodeType && + ( jQuery.cssNumber[ prop ] || unit !== "px" && +initial ) && + rcssNum.exec( jQuery.css( elem, prop ) ); + + if ( initialInUnit && initialInUnit[ 3 ] !== unit ) { + + // Support: Firefox <=54 + // Halve the iteration target value to prevent interference from CSS upper bounds (gh-2144) + initial = initial / 2; + + // Trust units reported by jQuery.css + unit = unit || initialInUnit[ 3 ]; + + // Iteratively approximate from a nonzero starting point + initialInUnit = +initial || 1; + + while ( maxIterations-- ) { + + // Evaluate and update our best guess (doubling guesses that zero out). + // Finish if the scale equals or crosses 1 (making the old*new product non-positive). + jQuery.style( elem, prop, initialInUnit + unit ); + if ( ( 1 - scale ) * ( 1 - ( scale = currentValue() / initial || 0.5 ) ) <= 0 ) { + maxIterations = 0; + } + initialInUnit = initialInUnit / scale; + + } + + initialInUnit = initialInUnit * 2; + jQuery.style( elem, prop, initialInUnit + unit ); + + // Make sure we update the tween properties later on + valueParts = valueParts || []; + } + + if ( valueParts ) { + initialInUnit = +initialInUnit || +initial || 0; + + // Apply relative offset (+=/-=) if specified + adjusted = valueParts[ 1 ] ? + initialInUnit + ( valueParts[ 1 ] + 1 ) * valueParts[ 2 ] : + +valueParts[ 2 ]; + if ( tween ) { + tween.unit = unit; + tween.start = initialInUnit; + tween.end = adjusted; + } + } + return adjusted; +} + + +var defaultDisplayMap = {}; + +function getDefaultDisplay( elem ) { + var temp, + doc = elem.ownerDocument, + nodeName = elem.nodeName, + display = defaultDisplayMap[ nodeName ]; + + if ( display ) { + return display; + } + + temp = doc.body.appendChild( doc.createElement( nodeName ) ); + display = jQuery.css( temp, "display" ); + + temp.parentNode.removeChild( temp ); + + if ( display === "none" ) { + display = "block"; + } + defaultDisplayMap[ nodeName ] = display; + + return display; +} + +function showHide( elements, show ) { + var display, elem, + values = [], + index = 0, + length = elements.length; + + // Determine new display value for elements that need to change + for ( ; index < length; index++ ) { + elem = elements[ index ]; + if ( !elem.style ) { + continue; + } + + display = elem.style.display; + if ( show ) { + + // Since we force visibility upon cascade-hidden elements, an immediate (and slow) + // check is required in this first loop unless we have a nonempty display value (either + // inline or about-to-be-restored) + if ( display === "none" ) { + values[ index ] = dataPriv.get( elem, "display" ) || null; + if ( !values[ index ] ) { + elem.style.display = ""; + } + } + if ( elem.style.display === "" && isHiddenWithinTree( elem ) ) { + values[ index ] = getDefaultDisplay( elem ); + } + } else { + if ( display !== "none" ) { + values[ index ] = "none"; + + // Remember what we're overwriting + dataPriv.set( elem, "display", display ); + } + } + } + + // Set the display of the elements in a second loop to avoid constant reflow + for ( index = 0; index < length; index++ ) { + if ( values[ index ] != null ) { + elements[ index ].style.display = values[ index ]; + } + } + + return elements; +} + +jQuery.fn.extend( { + show: function() { + return showHide( this, true ); + }, + hide: function() { + return showHide( this ); + }, + toggle: function( state ) { + if ( typeof state === "boolean" ) { + return state ? this.show() : this.hide(); + } + + return this.each( function() { + if ( isHiddenWithinTree( this ) ) { + jQuery( this ).show(); + } else { + jQuery( this ).hide(); + } + } ); + } +} ); +var rcheckableType = ( /^(?:checkbox|radio)$/i ); + +var rtagName = ( /<([a-z][^\/\0>\x20\t\r\n\f]*)/i ); + +var rscriptType = ( /^$|^module$|\/(?:java|ecma)script/i ); + + + +( function() { + var fragment = document.createDocumentFragment(), + div = fragment.appendChild( document.createElement( "div" ) ), + input = document.createElement( "input" ); + + // Support: Android 4.0 - 4.3 only + // Check state lost if the name is set (#11217) + // Support: Windows Web Apps (WWA) + // `name` and `type` must use .setAttribute for WWA (#14901) + input.setAttribute( "type", "radio" ); + input.setAttribute( "checked", "checked" ); + input.setAttribute( "name", "t" ); + + div.appendChild( input ); + + // Support: Android <=4.1 only + // Older WebKit doesn't clone checked state correctly in fragments + support.checkClone = div.cloneNode( true ).cloneNode( true ).lastChild.checked; + + // Support: IE <=11 only + // Make sure textarea (and checkbox) defaultValue is properly cloned + div.innerHTML = ""; + support.noCloneChecked = !!div.cloneNode( true ).lastChild.defaultValue; + + // Support: IE <=9 only + // IE <=9 replaces "; + support.option = !!div.lastChild; +} )(); + + +// We have to close these tags to support XHTML (#13200) +var wrapMap = { + + // XHTML parsers do not magically insert elements in the + // same way that tag soup parsers do. So we cannot shorten + // this by omitting or other required elements. + thead: [ 1, "", "
" ], + col: [ 2, "", "
" ], + tr: [ 2, "", "
" ], + td: [ 3, "", "
" ], + + _default: [ 0, "", "" ] +}; + +wrapMap.tbody = wrapMap.tfoot = wrapMap.colgroup = wrapMap.caption = wrapMap.thead; +wrapMap.th = wrapMap.td; + +// Support: IE <=9 only +if ( !support.option ) { + wrapMap.optgroup = wrapMap.option = [ 1, "" ]; +} + + +function getAll( context, tag ) { + + // Support: IE <=9 - 11 only + // Use typeof to avoid zero-argument method invocation on host objects (#15151) + var ret; + + if ( typeof context.getElementsByTagName !== "undefined" ) { + ret = context.getElementsByTagName( tag || "*" ); + + } else if ( typeof context.querySelectorAll !== "undefined" ) { + ret = context.querySelectorAll( tag || "*" ); + + } else { + ret = []; + } + + if ( tag === undefined || tag && nodeName( context, tag ) ) { + return jQuery.merge( [ context ], ret ); + } + + return ret; +} + + +// Mark scripts as having already been evaluated +function setGlobalEval( elems, refElements ) { + var i = 0, + l = elems.length; + + for ( ; i < l; i++ ) { + dataPriv.set( + elems[ i ], + "globalEval", + !refElements || dataPriv.get( refElements[ i ], "globalEval" ) + ); + } +} + + +var rhtml = /<|&#?\w+;/; + +function buildFragment( elems, context, scripts, selection, ignored ) { + var elem, tmp, tag, wrap, attached, j, + fragment = context.createDocumentFragment(), + nodes = [], + i = 0, + l = elems.length; + + for ( ; i < l; i++ ) { + elem = elems[ i ]; + + if ( elem || elem === 0 ) { + + // Add nodes directly + if ( toType( elem ) === "object" ) { + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( nodes, elem.nodeType ? [ elem ] : elem ); + + // Convert non-html into a text node + } else if ( !rhtml.test( elem ) ) { + nodes.push( context.createTextNode( elem ) ); + + // Convert html into DOM nodes + } else { + tmp = tmp || fragment.appendChild( context.createElement( "div" ) ); + + // Deserialize a standard representation + tag = ( rtagName.exec( elem ) || [ "", "" ] )[ 1 ].toLowerCase(); + wrap = wrapMap[ tag ] || wrapMap._default; + tmp.innerHTML = wrap[ 1 ] + jQuery.htmlPrefilter( elem ) + wrap[ 2 ]; + + // Descend through wrappers to the right content + j = wrap[ 0 ]; + while ( j-- ) { + tmp = tmp.lastChild; + } + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( nodes, tmp.childNodes ); + + // Remember the top-level container + tmp = fragment.firstChild; + + // Ensure the created nodes are orphaned (#12392) + tmp.textContent = ""; + } + } + } + + // Remove wrapper from fragment + fragment.textContent = ""; + + i = 0; + while ( ( elem = nodes[ i++ ] ) ) { + + // Skip elements already in the context collection (trac-4087) + if ( selection && jQuery.inArray( elem, selection ) > -1 ) { + if ( ignored ) { + ignored.push( elem ); + } + continue; + } + + attached = isAttached( elem ); + + // Append to fragment + tmp = getAll( fragment.appendChild( elem ), "script" ); + + // Preserve script evaluation history + if ( attached ) { + setGlobalEval( tmp ); + } + + // Capture executables + if ( scripts ) { + j = 0; + while ( ( elem = tmp[ j++ ] ) ) { + if ( rscriptType.test( elem.type || "" ) ) { + scripts.push( elem ); + } + } + } + } + + return fragment; +} + + +var rtypenamespace = /^([^.]*)(?:\.(.+)|)/; + +function returnTrue() { + return true; +} + +function returnFalse() { + return false; +} + +// Support: IE <=9 - 11+ +// focus() and blur() are asynchronous, except when they are no-op. +// So expect focus to be synchronous when the element is already active, +// and blur to be synchronous when the element is not already active. +// (focus and blur are always synchronous in other supported browsers, +// this just defines when we can count on it). +function expectSync( elem, type ) { + return ( elem === safeActiveElement() ) === ( type === "focus" ); +} + +// Support: IE <=9 only +// Accessing document.activeElement can throw unexpectedly +// https://bugs.jquery.com/ticket/13393 +function safeActiveElement() { + try { + return document.activeElement; + } catch ( err ) { } +} + +function on( elem, types, selector, data, fn, one ) { + var origFn, type; + + // Types can be a map of types/handlers + if ( typeof types === "object" ) { + + // ( types-Object, selector, data ) + if ( typeof selector !== "string" ) { + + // ( types-Object, data ) + data = data || selector; + selector = undefined; + } + for ( type in types ) { + on( elem, type, selector, data, types[ type ], one ); + } + return elem; + } + + if ( data == null && fn == null ) { + + // ( types, fn ) + fn = selector; + data = selector = undefined; + } else if ( fn == null ) { + if ( typeof selector === "string" ) { + + // ( types, selector, fn ) + fn = data; + data = undefined; + } else { + + // ( types, data, fn ) + fn = data; + data = selector; + selector = undefined; + } + } + if ( fn === false ) { + fn = returnFalse; + } else if ( !fn ) { + return elem; + } + + if ( one === 1 ) { + origFn = fn; + fn = function( event ) { + + // Can use an empty set, since event contains the info + jQuery().off( event ); + return origFn.apply( this, arguments ); + }; + + // Use same guid so caller can remove using origFn + fn.guid = origFn.guid || ( origFn.guid = jQuery.guid++ ); + } + return elem.each( function() { + jQuery.event.add( this, types, fn, data, selector ); + } ); +} + +/* + * Helper functions for managing events -- not part of the public interface. + * Props to Dean Edwards' addEvent library for many of the ideas. + */ +jQuery.event = { + + global: {}, + + add: function( elem, types, handler, data, selector ) { + + var handleObjIn, eventHandle, tmp, + events, t, handleObj, + special, handlers, type, namespaces, origType, + elemData = dataPriv.get( elem ); + + // Only attach events to objects that accept data + if ( !acceptData( elem ) ) { + return; + } + + // Caller can pass in an object of custom data in lieu of the handler + if ( handler.handler ) { + handleObjIn = handler; + handler = handleObjIn.handler; + selector = handleObjIn.selector; + } + + // Ensure that invalid selectors throw exceptions at attach time + // Evaluate against documentElement in case elem is a non-element node (e.g., document) + if ( selector ) { + jQuery.find.matchesSelector( documentElement, selector ); + } + + // Make sure that the handler has a unique ID, used to find/remove it later + if ( !handler.guid ) { + handler.guid = jQuery.guid++; + } + + // Init the element's event structure and main handler, if this is the first + if ( !( events = elemData.events ) ) { + events = elemData.events = Object.create( null ); + } + if ( !( eventHandle = elemData.handle ) ) { + eventHandle = elemData.handle = function( e ) { + + // Discard the second event of a jQuery.event.trigger() and + // when an event is called after a page has unloaded + return typeof jQuery !== "undefined" && jQuery.event.triggered !== e.type ? + jQuery.event.dispatch.apply( elem, arguments ) : undefined; + }; + } + + // Handle multiple events separated by a space + types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; + t = types.length; + while ( t-- ) { + tmp = rtypenamespace.exec( types[ t ] ) || []; + type = origType = tmp[ 1 ]; + namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); + + // There *must* be a type, no attaching namespace-only handlers + if ( !type ) { + continue; + } + + // If event changes its type, use the special event handlers for the changed type + special = jQuery.event.special[ type ] || {}; + + // If selector defined, determine special event api type, otherwise given type + type = ( selector ? special.delegateType : special.bindType ) || type; + + // Update special based on newly reset type + special = jQuery.event.special[ type ] || {}; + + // handleObj is passed to all event handlers + handleObj = jQuery.extend( { + type: type, + origType: origType, + data: data, + handler: handler, + guid: handler.guid, + selector: selector, + needsContext: selector && jQuery.expr.match.needsContext.test( selector ), + namespace: namespaces.join( "." ) + }, handleObjIn ); + + // Init the event handler queue if we're the first + if ( !( handlers = events[ type ] ) ) { + handlers = events[ type ] = []; + handlers.delegateCount = 0; + + // Only use addEventListener if the special events handler returns false + if ( !special.setup || + special.setup.call( elem, data, namespaces, eventHandle ) === false ) { + + if ( elem.addEventListener ) { + elem.addEventListener( type, eventHandle ); + } + } + } + + if ( special.add ) { + special.add.call( elem, handleObj ); + + if ( !handleObj.handler.guid ) { + handleObj.handler.guid = handler.guid; + } + } + + // Add to the element's handler list, delegates in front + if ( selector ) { + handlers.splice( handlers.delegateCount++, 0, handleObj ); + } else { + handlers.push( handleObj ); + } + + // Keep track of which events have ever been used, for event optimization + jQuery.event.global[ type ] = true; + } + + }, + + // Detach an event or set of events from an element + remove: function( elem, types, handler, selector, mappedTypes ) { + + var j, origCount, tmp, + events, t, handleObj, + special, handlers, type, namespaces, origType, + elemData = dataPriv.hasData( elem ) && dataPriv.get( elem ); + + if ( !elemData || !( events = elemData.events ) ) { + return; + } + + // Once for each type.namespace in types; type may be omitted + types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; + t = types.length; + while ( t-- ) { + tmp = rtypenamespace.exec( types[ t ] ) || []; + type = origType = tmp[ 1 ]; + namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); + + // Unbind all events (on this namespace, if provided) for the element + if ( !type ) { + for ( type in events ) { + jQuery.event.remove( elem, type + types[ t ], handler, selector, true ); + } + continue; + } + + special = jQuery.event.special[ type ] || {}; + type = ( selector ? special.delegateType : special.bindType ) || type; + handlers = events[ type ] || []; + tmp = tmp[ 2 ] && + new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ); + + // Remove matching events + origCount = j = handlers.length; + while ( j-- ) { + handleObj = handlers[ j ]; + + if ( ( mappedTypes || origType === handleObj.origType ) && + ( !handler || handler.guid === handleObj.guid ) && + ( !tmp || tmp.test( handleObj.namespace ) ) && + ( !selector || selector === handleObj.selector || + selector === "**" && handleObj.selector ) ) { + handlers.splice( j, 1 ); + + if ( handleObj.selector ) { + handlers.delegateCount--; + } + if ( special.remove ) { + special.remove.call( elem, handleObj ); + } + } + } + + // Remove generic event handler if we removed something and no more handlers exist + // (avoids potential for endless recursion during removal of special event handlers) + if ( origCount && !handlers.length ) { + if ( !special.teardown || + special.teardown.call( elem, namespaces, elemData.handle ) === false ) { + + jQuery.removeEvent( elem, type, elemData.handle ); + } + + delete events[ type ]; + } + } + + // Remove data and the expando if it's no longer used + if ( jQuery.isEmptyObject( events ) ) { + dataPriv.remove( elem, "handle events" ); + } + }, + + dispatch: function( nativeEvent ) { + + var i, j, ret, matched, handleObj, handlerQueue, + args = new Array( arguments.length ), + + // Make a writable jQuery.Event from the native event object + event = jQuery.event.fix( nativeEvent ), + + handlers = ( + dataPriv.get( this, "events" ) || Object.create( null ) + )[ event.type ] || [], + special = jQuery.event.special[ event.type ] || {}; + + // Use the fix-ed jQuery.Event rather than the (read-only) native event + args[ 0 ] = event; + + for ( i = 1; i < arguments.length; i++ ) { + args[ i ] = arguments[ i ]; + } + + event.delegateTarget = this; + + // Call the preDispatch hook for the mapped type, and let it bail if desired + if ( special.preDispatch && special.preDispatch.call( this, event ) === false ) { + return; + } + + // Determine handlers + handlerQueue = jQuery.event.handlers.call( this, event, handlers ); + + // Run delegates first; they may want to stop propagation beneath us + i = 0; + while ( ( matched = handlerQueue[ i++ ] ) && !event.isPropagationStopped() ) { + event.currentTarget = matched.elem; + + j = 0; + while ( ( handleObj = matched.handlers[ j++ ] ) && + !event.isImmediatePropagationStopped() ) { + + // If the event is namespaced, then each handler is only invoked if it is + // specially universal or its namespaces are a superset of the event's. + if ( !event.rnamespace || handleObj.namespace === false || + event.rnamespace.test( handleObj.namespace ) ) { + + event.handleObj = handleObj; + event.data = handleObj.data; + + ret = ( ( jQuery.event.special[ handleObj.origType ] || {} ).handle || + handleObj.handler ).apply( matched.elem, args ); + + if ( ret !== undefined ) { + if ( ( event.result = ret ) === false ) { + event.preventDefault(); + event.stopPropagation(); + } + } + } + } + } + + // Call the postDispatch hook for the mapped type + if ( special.postDispatch ) { + special.postDispatch.call( this, event ); + } + + return event.result; + }, + + handlers: function( event, handlers ) { + var i, handleObj, sel, matchedHandlers, matchedSelectors, + handlerQueue = [], + delegateCount = handlers.delegateCount, + cur = event.target; + + // Find delegate handlers + if ( delegateCount && + + // Support: IE <=9 + // Black-hole SVG instance trees (trac-13180) + cur.nodeType && + + // Support: Firefox <=42 + // Suppress spec-violating clicks indicating a non-primary pointer button (trac-3861) + // https://www.w3.org/TR/DOM-Level-3-Events/#event-type-click + // Support: IE 11 only + // ...but not arrow key "clicks" of radio inputs, which can have `button` -1 (gh-2343) + !( event.type === "click" && event.button >= 1 ) ) { + + for ( ; cur !== this; cur = cur.parentNode || this ) { + + // Don't check non-elements (#13208) + // Don't process clicks on disabled elements (#6911, #8165, #11382, #11764) + if ( cur.nodeType === 1 && !( event.type === "click" && cur.disabled === true ) ) { + matchedHandlers = []; + matchedSelectors = {}; + for ( i = 0; i < delegateCount; i++ ) { + handleObj = handlers[ i ]; + + // Don't conflict with Object.prototype properties (#13203) + sel = handleObj.selector + " "; + + if ( matchedSelectors[ sel ] === undefined ) { + matchedSelectors[ sel ] = handleObj.needsContext ? + jQuery( sel, this ).index( cur ) > -1 : + jQuery.find( sel, this, null, [ cur ] ).length; + } + if ( matchedSelectors[ sel ] ) { + matchedHandlers.push( handleObj ); + } + } + if ( matchedHandlers.length ) { + handlerQueue.push( { elem: cur, handlers: matchedHandlers } ); + } + } + } + } + + // Add the remaining (directly-bound) handlers + cur = this; + if ( delegateCount < handlers.length ) { + handlerQueue.push( { elem: cur, handlers: handlers.slice( delegateCount ) } ); + } + + return handlerQueue; + }, + + addProp: function( name, hook ) { + Object.defineProperty( jQuery.Event.prototype, name, { + enumerable: true, + configurable: true, + + get: isFunction( hook ) ? + function() { + if ( this.originalEvent ) { + return hook( this.originalEvent ); + } + } : + function() { + if ( this.originalEvent ) { + return this.originalEvent[ name ]; + } + }, + + set: function( value ) { + Object.defineProperty( this, name, { + enumerable: true, + configurable: true, + writable: true, + value: value + } ); + } + } ); + }, + + fix: function( originalEvent ) { + return originalEvent[ jQuery.expando ] ? + originalEvent : + new jQuery.Event( originalEvent ); + }, + + special: { + load: { + + // Prevent triggered image.load events from bubbling to window.load + noBubble: true + }, + click: { + + // Utilize native event to ensure correct state for checkable inputs + setup: function( data ) { + + // For mutual compressibility with _default, replace `this` access with a local var. + // `|| data` is dead code meant only to preserve the variable through minification. + var el = this || data; + + // Claim the first handler + if ( rcheckableType.test( el.type ) && + el.click && nodeName( el, "input" ) ) { + + // dataPriv.set( el, "click", ... ) + leverageNative( el, "click", returnTrue ); + } + + // Return false to allow normal processing in the caller + return false; + }, + trigger: function( data ) { + + // For mutual compressibility with _default, replace `this` access with a local var. + // `|| data` is dead code meant only to preserve the variable through minification. + var el = this || data; + + // Force setup before triggering a click + if ( rcheckableType.test( el.type ) && + el.click && nodeName( el, "input" ) ) { + + leverageNative( el, "click" ); + } + + // Return non-false to allow normal event-path propagation + return true; + }, + + // For cross-browser consistency, suppress native .click() on links + // Also prevent it if we're currently inside a leveraged native-event stack + _default: function( event ) { + var target = event.target; + return rcheckableType.test( target.type ) && + target.click && nodeName( target, "input" ) && + dataPriv.get( target, "click" ) || + nodeName( target, "a" ); + } + }, + + beforeunload: { + postDispatch: function( event ) { + + // Support: Firefox 20+ + // Firefox doesn't alert if the returnValue field is not set. + if ( event.result !== undefined && event.originalEvent ) { + event.originalEvent.returnValue = event.result; + } + } + } + } +}; + +// Ensure the presence of an event listener that handles manually-triggered +// synthetic events by interrupting progress until reinvoked in response to +// *native* events that it fires directly, ensuring that state changes have +// already occurred before other listeners are invoked. +function leverageNative( el, type, expectSync ) { + + // Missing expectSync indicates a trigger call, which must force setup through jQuery.event.add + if ( !expectSync ) { + if ( dataPriv.get( el, type ) === undefined ) { + jQuery.event.add( el, type, returnTrue ); + } + return; + } + + // Register the controller as a special universal handler for all event namespaces + dataPriv.set( el, type, false ); + jQuery.event.add( el, type, { + namespace: false, + handler: function( event ) { + var notAsync, result, + saved = dataPriv.get( this, type ); + + if ( ( event.isTrigger & 1 ) && this[ type ] ) { + + // Interrupt processing of the outer synthetic .trigger()ed event + // Saved data should be false in such cases, but might be a leftover capture object + // from an async native handler (gh-4350) + if ( !saved.length ) { + + // Store arguments for use when handling the inner native event + // There will always be at least one argument (an event object), so this array + // will not be confused with a leftover capture object. + saved = slice.call( arguments ); + dataPriv.set( this, type, saved ); + + // Trigger the native event and capture its result + // Support: IE <=9 - 11+ + // focus() and blur() are asynchronous + notAsync = expectSync( this, type ); + this[ type ](); + result = dataPriv.get( this, type ); + if ( saved !== result || notAsync ) { + dataPriv.set( this, type, false ); + } else { + result = {}; + } + if ( saved !== result ) { + + // Cancel the outer synthetic event + event.stopImmediatePropagation(); + event.preventDefault(); + + // Support: Chrome 86+ + // In Chrome, if an element having a focusout handler is blurred by + // clicking outside of it, it invokes the handler synchronously. If + // that handler calls `.remove()` on the element, the data is cleared, + // leaving `result` undefined. We need to guard against this. + return result && result.value; + } + + // If this is an inner synthetic event for an event with a bubbling surrogate + // (focus or blur), assume that the surrogate already propagated from triggering the + // native event and prevent that from happening again here. + // This technically gets the ordering wrong w.r.t. to `.trigger()` (in which the + // bubbling surrogate propagates *after* the non-bubbling base), but that seems + // less bad than duplication. + } else if ( ( jQuery.event.special[ type ] || {} ).delegateType ) { + event.stopPropagation(); + } + + // If this is a native event triggered above, everything is now in order + // Fire an inner synthetic event with the original arguments + } else if ( saved.length ) { + + // ...and capture the result + dataPriv.set( this, type, { + value: jQuery.event.trigger( + + // Support: IE <=9 - 11+ + // Extend with the prototype to reset the above stopImmediatePropagation() + jQuery.extend( saved[ 0 ], jQuery.Event.prototype ), + saved.slice( 1 ), + this + ) + } ); + + // Abort handling of the native event + event.stopImmediatePropagation(); + } + } + } ); +} + +jQuery.removeEvent = function( elem, type, handle ) { + + // This "if" is needed for plain objects + if ( elem.removeEventListener ) { + elem.removeEventListener( type, handle ); + } +}; + +jQuery.Event = function( src, props ) { + + // Allow instantiation without the 'new' keyword + if ( !( this instanceof jQuery.Event ) ) { + return new jQuery.Event( src, props ); + } + + // Event object + if ( src && src.type ) { + this.originalEvent = src; + this.type = src.type; + + // Events bubbling up the document may have been marked as prevented + // by a handler lower down the tree; reflect the correct value. + this.isDefaultPrevented = src.defaultPrevented || + src.defaultPrevented === undefined && + + // Support: Android <=2.3 only + src.returnValue === false ? + returnTrue : + returnFalse; + + // Create target properties + // Support: Safari <=6 - 7 only + // Target should not be a text node (#504, #13143) + this.target = ( src.target && src.target.nodeType === 3 ) ? + src.target.parentNode : + src.target; + + this.currentTarget = src.currentTarget; + this.relatedTarget = src.relatedTarget; + + // Event type + } else { + this.type = src; + } + + // Put explicitly provided properties onto the event object + if ( props ) { + jQuery.extend( this, props ); + } + + // Create a timestamp if incoming event doesn't have one + this.timeStamp = src && src.timeStamp || Date.now(); + + // Mark it as fixed + this[ jQuery.expando ] = true; +}; + +// jQuery.Event is based on DOM3 Events as specified by the ECMAScript Language Binding +// https://www.w3.org/TR/2003/WD-DOM-Level-3-Events-20030331/ecma-script-binding.html +jQuery.Event.prototype = { + constructor: jQuery.Event, + isDefaultPrevented: returnFalse, + isPropagationStopped: returnFalse, + isImmediatePropagationStopped: returnFalse, + isSimulated: false, + + preventDefault: function() { + var e = this.originalEvent; + + this.isDefaultPrevented = returnTrue; + + if ( e && !this.isSimulated ) { + e.preventDefault(); + } + }, + stopPropagation: function() { + var e = this.originalEvent; + + this.isPropagationStopped = returnTrue; + + if ( e && !this.isSimulated ) { + e.stopPropagation(); + } + }, + stopImmediatePropagation: function() { + var e = this.originalEvent; + + this.isImmediatePropagationStopped = returnTrue; + + if ( e && !this.isSimulated ) { + e.stopImmediatePropagation(); + } + + this.stopPropagation(); + } +}; + +// Includes all common event props including KeyEvent and MouseEvent specific props +jQuery.each( { + altKey: true, + bubbles: true, + cancelable: true, + changedTouches: true, + ctrlKey: true, + detail: true, + eventPhase: true, + metaKey: true, + pageX: true, + pageY: true, + shiftKey: true, + view: true, + "char": true, + code: true, + charCode: true, + key: true, + keyCode: true, + button: true, + buttons: true, + clientX: true, + clientY: true, + offsetX: true, + offsetY: true, + pointerId: true, + pointerType: true, + screenX: true, + screenY: true, + targetTouches: true, + toElement: true, + touches: true, + which: true +}, jQuery.event.addProp ); + +jQuery.each( { focus: "focusin", blur: "focusout" }, function( type, delegateType ) { + jQuery.event.special[ type ] = { + + // Utilize native event if possible so blur/focus sequence is correct + setup: function() { + + // Claim the first handler + // dataPriv.set( this, "focus", ... ) + // dataPriv.set( this, "blur", ... ) + leverageNative( this, type, expectSync ); + + // Return false to allow normal processing in the caller + return false; + }, + trigger: function() { + + // Force setup before trigger + leverageNative( this, type ); + + // Return non-false to allow normal event-path propagation + return true; + }, + + // Suppress native focus or blur as it's already being fired + // in leverageNative. + _default: function() { + return true; + }, + + delegateType: delegateType + }; +} ); + +// Create mouseenter/leave events using mouseover/out and event-time checks +// so that event delegation works in jQuery. +// Do the same for pointerenter/pointerleave and pointerover/pointerout +// +// Support: Safari 7 only +// Safari sends mouseenter too often; see: +// https://bugs.chromium.org/p/chromium/issues/detail?id=470258 +// for the description of the bug (it existed in older Chrome versions as well). +jQuery.each( { + mouseenter: "mouseover", + mouseleave: "mouseout", + pointerenter: "pointerover", + pointerleave: "pointerout" +}, function( orig, fix ) { + jQuery.event.special[ orig ] = { + delegateType: fix, + bindType: fix, + + handle: function( event ) { + var ret, + target = this, + related = event.relatedTarget, + handleObj = event.handleObj; + + // For mouseenter/leave call the handler if related is outside the target. + // NB: No relatedTarget if the mouse left/entered the browser window + if ( !related || ( related !== target && !jQuery.contains( target, related ) ) ) { + event.type = handleObj.origType; + ret = handleObj.handler.apply( this, arguments ); + event.type = fix; + } + return ret; + } + }; +} ); + +jQuery.fn.extend( { + + on: function( types, selector, data, fn ) { + return on( this, types, selector, data, fn ); + }, + one: function( types, selector, data, fn ) { + return on( this, types, selector, data, fn, 1 ); + }, + off: function( types, selector, fn ) { + var handleObj, type; + if ( types && types.preventDefault && types.handleObj ) { + + // ( event ) dispatched jQuery.Event + handleObj = types.handleObj; + jQuery( types.delegateTarget ).off( + handleObj.namespace ? + handleObj.origType + "." + handleObj.namespace : + handleObj.origType, + handleObj.selector, + handleObj.handler + ); + return this; + } + if ( typeof types === "object" ) { + + // ( types-object [, selector] ) + for ( type in types ) { + this.off( type, selector, types[ type ] ); + } + return this; + } + if ( selector === false || typeof selector === "function" ) { + + // ( types [, fn] ) + fn = selector; + selector = undefined; + } + if ( fn === false ) { + fn = returnFalse; + } + return this.each( function() { + jQuery.event.remove( this, types, fn, selector ); + } ); + } +} ); + + +var + + // Support: IE <=10 - 11, Edge 12 - 13 only + // In IE/Edge using regex groups here causes severe slowdowns. + // See https://connect.microsoft.com/IE/feedback/details/1736512/ + rnoInnerhtml = /\s*$/g; + +// Prefer a tbody over its parent table for containing new rows +function manipulationTarget( elem, content ) { + if ( nodeName( elem, "table" ) && + nodeName( content.nodeType !== 11 ? content : content.firstChild, "tr" ) ) { + + return jQuery( elem ).children( "tbody" )[ 0 ] || elem; + } + + return elem; +} + +// Replace/restore the type attribute of script elements for safe DOM manipulation +function disableScript( elem ) { + elem.type = ( elem.getAttribute( "type" ) !== null ) + "/" + elem.type; + return elem; +} +function restoreScript( elem ) { + if ( ( elem.type || "" ).slice( 0, 5 ) === "true/" ) { + elem.type = elem.type.slice( 5 ); + } else { + elem.removeAttribute( "type" ); + } + + return elem; +} + +function cloneCopyEvent( src, dest ) { + var i, l, type, pdataOld, udataOld, udataCur, events; + + if ( dest.nodeType !== 1 ) { + return; + } + + // 1. Copy private data: events, handlers, etc. + if ( dataPriv.hasData( src ) ) { + pdataOld = dataPriv.get( src ); + events = pdataOld.events; + + if ( events ) { + dataPriv.remove( dest, "handle events" ); + + for ( type in events ) { + for ( i = 0, l = events[ type ].length; i < l; i++ ) { + jQuery.event.add( dest, type, events[ type ][ i ] ); + } + } + } + } + + // 2. Copy user data + if ( dataUser.hasData( src ) ) { + udataOld = dataUser.access( src ); + udataCur = jQuery.extend( {}, udataOld ); + + dataUser.set( dest, udataCur ); + } +} + +// Fix IE bugs, see support tests +function fixInput( src, dest ) { + var nodeName = dest.nodeName.toLowerCase(); + + // Fails to persist the checked state of a cloned checkbox or radio button. + if ( nodeName === "input" && rcheckableType.test( src.type ) ) { + dest.checked = src.checked; + + // Fails to return the selected option to the default selected state when cloning options + } else if ( nodeName === "input" || nodeName === "textarea" ) { + dest.defaultValue = src.defaultValue; + } +} + +function domManip( collection, args, callback, ignored ) { + + // Flatten any nested arrays + args = flat( args ); + + var fragment, first, scripts, hasScripts, node, doc, + i = 0, + l = collection.length, + iNoClone = l - 1, + value = args[ 0 ], + valueIsFunction = isFunction( value ); + + // We can't cloneNode fragments that contain checked, in WebKit + if ( valueIsFunction || + ( l > 1 && typeof value === "string" && + !support.checkClone && rchecked.test( value ) ) ) { + return collection.each( function( index ) { + var self = collection.eq( index ); + if ( valueIsFunction ) { + args[ 0 ] = value.call( this, index, self.html() ); + } + domManip( self, args, callback, ignored ); + } ); + } + + if ( l ) { + fragment = buildFragment( args, collection[ 0 ].ownerDocument, false, collection, ignored ); + first = fragment.firstChild; + + if ( fragment.childNodes.length === 1 ) { + fragment = first; + } + + // Require either new content or an interest in ignored elements to invoke the callback + if ( first || ignored ) { + scripts = jQuery.map( getAll( fragment, "script" ), disableScript ); + hasScripts = scripts.length; + + // Use the original fragment for the last item + // instead of the first because it can end up + // being emptied incorrectly in certain situations (#8070). + for ( ; i < l; i++ ) { + node = fragment; + + if ( i !== iNoClone ) { + node = jQuery.clone( node, true, true ); + + // Keep references to cloned scripts for later restoration + if ( hasScripts ) { + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( scripts, getAll( node, "script" ) ); + } + } + + callback.call( collection[ i ], node, i ); + } + + if ( hasScripts ) { + doc = scripts[ scripts.length - 1 ].ownerDocument; + + // Reenable scripts + jQuery.map( scripts, restoreScript ); + + // Evaluate executable scripts on first document insertion + for ( i = 0; i < hasScripts; i++ ) { + node = scripts[ i ]; + if ( rscriptType.test( node.type || "" ) && + !dataPriv.access( node, "globalEval" ) && + jQuery.contains( doc, node ) ) { + + if ( node.src && ( node.type || "" ).toLowerCase() !== "module" ) { + + // Optional AJAX dependency, but won't run scripts if not present + if ( jQuery._evalUrl && !node.noModule ) { + jQuery._evalUrl( node.src, { + nonce: node.nonce || node.getAttribute( "nonce" ) + }, doc ); + } + } else { + DOMEval( node.textContent.replace( rcleanScript, "" ), node, doc ); + } + } + } + } + } + } + + return collection; +} + +function remove( elem, selector, keepData ) { + var node, + nodes = selector ? jQuery.filter( selector, elem ) : elem, + i = 0; + + for ( ; ( node = nodes[ i ] ) != null; i++ ) { + if ( !keepData && node.nodeType === 1 ) { + jQuery.cleanData( getAll( node ) ); + } + + if ( node.parentNode ) { + if ( keepData && isAttached( node ) ) { + setGlobalEval( getAll( node, "script" ) ); + } + node.parentNode.removeChild( node ); + } + } + + return elem; +} + +jQuery.extend( { + htmlPrefilter: function( html ) { + return html; + }, + + clone: function( elem, dataAndEvents, deepDataAndEvents ) { + var i, l, srcElements, destElements, + clone = elem.cloneNode( true ), + inPage = isAttached( elem ); + + // Fix IE cloning issues + if ( !support.noCloneChecked && ( elem.nodeType === 1 || elem.nodeType === 11 ) && + !jQuery.isXMLDoc( elem ) ) { + + // We eschew Sizzle here for performance reasons: https://jsperf.com/getall-vs-sizzle/2 + destElements = getAll( clone ); + srcElements = getAll( elem ); + + for ( i = 0, l = srcElements.length; i < l; i++ ) { + fixInput( srcElements[ i ], destElements[ i ] ); + } + } + + // Copy the events from the original to the clone + if ( dataAndEvents ) { + if ( deepDataAndEvents ) { + srcElements = srcElements || getAll( elem ); + destElements = destElements || getAll( clone ); + + for ( i = 0, l = srcElements.length; i < l; i++ ) { + cloneCopyEvent( srcElements[ i ], destElements[ i ] ); + } + } else { + cloneCopyEvent( elem, clone ); + } + } + + // Preserve script evaluation history + destElements = getAll( clone, "script" ); + if ( destElements.length > 0 ) { + setGlobalEval( destElements, !inPage && getAll( elem, "script" ) ); + } + + // Return the cloned set + return clone; + }, + + cleanData: function( elems ) { + var data, elem, type, + special = jQuery.event.special, + i = 0; + + for ( ; ( elem = elems[ i ] ) !== undefined; i++ ) { + if ( acceptData( elem ) ) { + if ( ( data = elem[ dataPriv.expando ] ) ) { + if ( data.events ) { + for ( type in data.events ) { + if ( special[ type ] ) { + jQuery.event.remove( elem, type ); + + // This is a shortcut to avoid jQuery.event.remove's overhead + } else { + jQuery.removeEvent( elem, type, data.handle ); + } + } + } + + // Support: Chrome <=35 - 45+ + // Assign undefined instead of using delete, see Data#remove + elem[ dataPriv.expando ] = undefined; + } + if ( elem[ dataUser.expando ] ) { + + // Support: Chrome <=35 - 45+ + // Assign undefined instead of using delete, see Data#remove + elem[ dataUser.expando ] = undefined; + } + } + } + } +} ); + +jQuery.fn.extend( { + detach: function( selector ) { + return remove( this, selector, true ); + }, + + remove: function( selector ) { + return remove( this, selector ); + }, + + text: function( value ) { + return access( this, function( value ) { + return value === undefined ? + jQuery.text( this ) : + this.empty().each( function() { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + this.textContent = value; + } + } ); + }, null, value, arguments.length ); + }, + + append: function() { + return domManip( this, arguments, function( elem ) { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + var target = manipulationTarget( this, elem ); + target.appendChild( elem ); + } + } ); + }, + + prepend: function() { + return domManip( this, arguments, function( elem ) { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + var target = manipulationTarget( this, elem ); + target.insertBefore( elem, target.firstChild ); + } + } ); + }, + + before: function() { + return domManip( this, arguments, function( elem ) { + if ( this.parentNode ) { + this.parentNode.insertBefore( elem, this ); + } + } ); + }, + + after: function() { + return domManip( this, arguments, function( elem ) { + if ( this.parentNode ) { + this.parentNode.insertBefore( elem, this.nextSibling ); + } + } ); + }, + + empty: function() { + var elem, + i = 0; + + for ( ; ( elem = this[ i ] ) != null; i++ ) { + if ( elem.nodeType === 1 ) { + + // Prevent memory leaks + jQuery.cleanData( getAll( elem, false ) ); + + // Remove any remaining nodes + elem.textContent = ""; + } + } + + return this; + }, + + clone: function( dataAndEvents, deepDataAndEvents ) { + dataAndEvents = dataAndEvents == null ? false : dataAndEvents; + deepDataAndEvents = deepDataAndEvents == null ? dataAndEvents : deepDataAndEvents; + + return this.map( function() { + return jQuery.clone( this, dataAndEvents, deepDataAndEvents ); + } ); + }, + + html: function( value ) { + return access( this, function( value ) { + var elem = this[ 0 ] || {}, + i = 0, + l = this.length; + + if ( value === undefined && elem.nodeType === 1 ) { + return elem.innerHTML; + } + + // See if we can take a shortcut and just use innerHTML + if ( typeof value === "string" && !rnoInnerhtml.test( value ) && + !wrapMap[ ( rtagName.exec( value ) || [ "", "" ] )[ 1 ].toLowerCase() ] ) { + + value = jQuery.htmlPrefilter( value ); + + try { + for ( ; i < l; i++ ) { + elem = this[ i ] || {}; + + // Remove element nodes and prevent memory leaks + if ( elem.nodeType === 1 ) { + jQuery.cleanData( getAll( elem, false ) ); + elem.innerHTML = value; + } + } + + elem = 0; + + // If using innerHTML throws an exception, use the fallback method + } catch ( e ) {} + } + + if ( elem ) { + this.empty().append( value ); + } + }, null, value, arguments.length ); + }, + + replaceWith: function() { + var ignored = []; + + // Make the changes, replacing each non-ignored context element with the new content + return domManip( this, arguments, function( elem ) { + var parent = this.parentNode; + + if ( jQuery.inArray( this, ignored ) < 0 ) { + jQuery.cleanData( getAll( this ) ); + if ( parent ) { + parent.replaceChild( elem, this ); + } + } + + // Force callback invocation + }, ignored ); + } +} ); + +jQuery.each( { + appendTo: "append", + prependTo: "prepend", + insertBefore: "before", + insertAfter: "after", + replaceAll: "replaceWith" +}, function( name, original ) { + jQuery.fn[ name ] = function( selector ) { + var elems, + ret = [], + insert = jQuery( selector ), + last = insert.length - 1, + i = 0; + + for ( ; i <= last; i++ ) { + elems = i === last ? this : this.clone( true ); + jQuery( insert[ i ] )[ original ]( elems ); + + // Support: Android <=4.0 only, PhantomJS 1 only + // .get() because push.apply(_, arraylike) throws on ancient WebKit + push.apply( ret, elems.get() ); + } + + return this.pushStack( ret ); + }; +} ); +var rnumnonpx = new RegExp( "^(" + pnum + ")(?!px)[a-z%]+$", "i" ); + +var getStyles = function( elem ) { + + // Support: IE <=11 only, Firefox <=30 (#15098, #14150) + // IE throws on elements created in popups + // FF meanwhile throws on frame elements through "defaultView.getComputedStyle" + var view = elem.ownerDocument.defaultView; + + if ( !view || !view.opener ) { + view = window; + } + + return view.getComputedStyle( elem ); + }; + +var swap = function( elem, options, callback ) { + var ret, name, + old = {}; + + // Remember the old values, and insert the new ones + for ( name in options ) { + old[ name ] = elem.style[ name ]; + elem.style[ name ] = options[ name ]; + } + + ret = callback.call( elem ); + + // Revert the old values + for ( name in options ) { + elem.style[ name ] = old[ name ]; + } + + return ret; +}; + + +var rboxStyle = new RegExp( cssExpand.join( "|" ), "i" ); + + + +( function() { + + // Executing both pixelPosition & boxSizingReliable tests require only one layout + // so they're executed at the same time to save the second computation. + function computeStyleTests() { + + // This is a singleton, we need to execute it only once + if ( !div ) { + return; + } + + container.style.cssText = "position:absolute;left:-11111px;width:60px;" + + "margin-top:1px;padding:0;border:0"; + div.style.cssText = + "position:relative;display:block;box-sizing:border-box;overflow:scroll;" + + "margin:auto;border:1px;padding:1px;" + + "width:60%;top:1%"; + documentElement.appendChild( container ).appendChild( div ); + + var divStyle = window.getComputedStyle( div ); + pixelPositionVal = divStyle.top !== "1%"; + + // Support: Android 4.0 - 4.3 only, Firefox <=3 - 44 + reliableMarginLeftVal = roundPixelMeasures( divStyle.marginLeft ) === 12; + + // Support: Android 4.0 - 4.3 only, Safari <=9.1 - 10.1, iOS <=7.0 - 9.3 + // Some styles come back with percentage values, even though they shouldn't + div.style.right = "60%"; + pixelBoxStylesVal = roundPixelMeasures( divStyle.right ) === 36; + + // Support: IE 9 - 11 only + // Detect misreporting of content dimensions for box-sizing:border-box elements + boxSizingReliableVal = roundPixelMeasures( divStyle.width ) === 36; + + // Support: IE 9 only + // Detect overflow:scroll screwiness (gh-3699) + // Support: Chrome <=64 + // Don't get tricked when zoom affects offsetWidth (gh-4029) + div.style.position = "absolute"; + scrollboxSizeVal = roundPixelMeasures( div.offsetWidth / 3 ) === 12; + + documentElement.removeChild( container ); + + // Nullify the div so it wouldn't be stored in the memory and + // it will also be a sign that checks already performed + div = null; + } + + function roundPixelMeasures( measure ) { + return Math.round( parseFloat( measure ) ); + } + + var pixelPositionVal, boxSizingReliableVal, scrollboxSizeVal, pixelBoxStylesVal, + reliableTrDimensionsVal, reliableMarginLeftVal, + container = document.createElement( "div" ), + div = document.createElement( "div" ); + + // Finish early in limited (non-browser) environments + if ( !div.style ) { + return; + } + + // Support: IE <=9 - 11 only + // Style of cloned element affects source element cloned (#8908) + div.style.backgroundClip = "content-box"; + div.cloneNode( true ).style.backgroundClip = ""; + support.clearCloneStyle = div.style.backgroundClip === "content-box"; + + jQuery.extend( support, { + boxSizingReliable: function() { + computeStyleTests(); + return boxSizingReliableVal; + }, + pixelBoxStyles: function() { + computeStyleTests(); + return pixelBoxStylesVal; + }, + pixelPosition: function() { + computeStyleTests(); + return pixelPositionVal; + }, + reliableMarginLeft: function() { + computeStyleTests(); + return reliableMarginLeftVal; + }, + scrollboxSize: function() { + computeStyleTests(); + return scrollboxSizeVal; + }, + + // Support: IE 9 - 11+, Edge 15 - 18+ + // IE/Edge misreport `getComputedStyle` of table rows with width/height + // set in CSS while `offset*` properties report correct values. + // Behavior in IE 9 is more subtle than in newer versions & it passes + // some versions of this test; make sure not to make it pass there! + // + // Support: Firefox 70+ + // Only Firefox includes border widths + // in computed dimensions. (gh-4529) + reliableTrDimensions: function() { + var table, tr, trChild, trStyle; + if ( reliableTrDimensionsVal == null ) { + table = document.createElement( "table" ); + tr = document.createElement( "tr" ); + trChild = document.createElement( "div" ); + + table.style.cssText = "position:absolute;left:-11111px;border-collapse:separate"; + tr.style.cssText = "border:1px solid"; + + // Support: Chrome 86+ + // Height set through cssText does not get applied. + // Computed height then comes back as 0. + tr.style.height = "1px"; + trChild.style.height = "9px"; + + // Support: Android 8 Chrome 86+ + // In our bodyBackground.html iframe, + // display for all div elements is set to "inline", + // which causes a problem only in Android 8 Chrome 86. + // Ensuring the div is display: block + // gets around this issue. + trChild.style.display = "block"; + + documentElement + .appendChild( table ) + .appendChild( tr ) + .appendChild( trChild ); + + trStyle = window.getComputedStyle( tr ); + reliableTrDimensionsVal = ( parseInt( trStyle.height, 10 ) + + parseInt( trStyle.borderTopWidth, 10 ) + + parseInt( trStyle.borderBottomWidth, 10 ) ) === tr.offsetHeight; + + documentElement.removeChild( table ); + } + return reliableTrDimensionsVal; + } + } ); +} )(); + + +function curCSS( elem, name, computed ) { + var width, minWidth, maxWidth, ret, + + // Support: Firefox 51+ + // Retrieving style before computed somehow + // fixes an issue with getting wrong values + // on detached elements + style = elem.style; + + computed = computed || getStyles( elem ); + + // getPropertyValue is needed for: + // .css('filter') (IE 9 only, #12537) + // .css('--customProperty) (#3144) + if ( computed ) { + ret = computed.getPropertyValue( name ) || computed[ name ]; + + if ( ret === "" && !isAttached( elem ) ) { + ret = jQuery.style( elem, name ); + } + + // A tribute to the "awesome hack by Dean Edwards" + // Android Browser returns percentage for some values, + // but width seems to be reliably pixels. + // This is against the CSSOM draft spec: + // https://drafts.csswg.org/cssom/#resolved-values + if ( !support.pixelBoxStyles() && rnumnonpx.test( ret ) && rboxStyle.test( name ) ) { + + // Remember the original values + width = style.width; + minWidth = style.minWidth; + maxWidth = style.maxWidth; + + // Put in the new values to get a computed value out + style.minWidth = style.maxWidth = style.width = ret; + ret = computed.width; + + // Revert the changed values + style.width = width; + style.minWidth = minWidth; + style.maxWidth = maxWidth; + } + } + + return ret !== undefined ? + + // Support: IE <=9 - 11 only + // IE returns zIndex value as an integer. + ret + "" : + ret; +} + + +function addGetHookIf( conditionFn, hookFn ) { + + // Define the hook, we'll check on the first run if it's really needed. + return { + get: function() { + if ( conditionFn() ) { + + // Hook not needed (or it's not possible to use it due + // to missing dependency), remove it. + delete this.get; + return; + } + + // Hook needed; redefine it so that the support test is not executed again. + return ( this.get = hookFn ).apply( this, arguments ); + } + }; +} + + +var cssPrefixes = [ "Webkit", "Moz", "ms" ], + emptyStyle = document.createElement( "div" ).style, + vendorProps = {}; + +// Return a vendor-prefixed property or undefined +function vendorPropName( name ) { + + // Check for vendor prefixed names + var capName = name[ 0 ].toUpperCase() + name.slice( 1 ), + i = cssPrefixes.length; + + while ( i-- ) { + name = cssPrefixes[ i ] + capName; + if ( name in emptyStyle ) { + return name; + } + } +} + +// Return a potentially-mapped jQuery.cssProps or vendor prefixed property +function finalPropName( name ) { + var final = jQuery.cssProps[ name ] || vendorProps[ name ]; + + if ( final ) { + return final; + } + if ( name in emptyStyle ) { + return name; + } + return vendorProps[ name ] = vendorPropName( name ) || name; +} + + +var + + // Swappable if display is none or starts with table + // except "table", "table-cell", or "table-caption" + // See here for display values: https://developer.mozilla.org/en-US/docs/CSS/display + rdisplayswap = /^(none|table(?!-c[ea]).+)/, + rcustomProp = /^--/, + cssShow = { position: "absolute", visibility: "hidden", display: "block" }, + cssNormalTransform = { + letterSpacing: "0", + fontWeight: "400" + }; + +function setPositiveNumber( _elem, value, subtract ) { + + // Any relative (+/-) values have already been + // normalized at this point + var matches = rcssNum.exec( value ); + return matches ? + + // Guard against undefined "subtract", e.g., when used as in cssHooks + Math.max( 0, matches[ 2 ] - ( subtract || 0 ) ) + ( matches[ 3 ] || "px" ) : + value; +} + +function boxModelAdjustment( elem, dimension, box, isBorderBox, styles, computedVal ) { + var i = dimension === "width" ? 1 : 0, + extra = 0, + delta = 0; + + // Adjustment may not be necessary + if ( box === ( isBorderBox ? "border" : "content" ) ) { + return 0; + } + + for ( ; i < 4; i += 2 ) { + + // Both box models exclude margin + if ( box === "margin" ) { + delta += jQuery.css( elem, box + cssExpand[ i ], true, styles ); + } + + // If we get here with a content-box, we're seeking "padding" or "border" or "margin" + if ( !isBorderBox ) { + + // Add padding + delta += jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); + + // For "border" or "margin", add border + if ( box !== "padding" ) { + delta += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + + // But still keep track of it otherwise + } else { + extra += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + } + + // If we get here with a border-box (content + padding + border), we're seeking "content" or + // "padding" or "margin" + } else { + + // For "content", subtract padding + if ( box === "content" ) { + delta -= jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); + } + + // For "content" or "padding", subtract border + if ( box !== "margin" ) { + delta -= jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + } + } + } + + // Account for positive content-box scroll gutter when requested by providing computedVal + if ( !isBorderBox && computedVal >= 0 ) { + + // offsetWidth/offsetHeight is a rounded sum of content, padding, scroll gutter, and border + // Assuming integer scroll gutter, subtract the rest and round down + delta += Math.max( 0, Math.ceil( + elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - + computedVal - + delta - + extra - + 0.5 + + // If offsetWidth/offsetHeight is unknown, then we can't determine content-box scroll gutter + // Use an explicit zero to avoid NaN (gh-3964) + ) ) || 0; + } + + return delta; +} + +function getWidthOrHeight( elem, dimension, extra ) { + + // Start with computed style + var styles = getStyles( elem ), + + // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-4322). + // Fake content-box until we know it's needed to know the true value. + boxSizingNeeded = !support.boxSizingReliable() || extra, + isBorderBox = boxSizingNeeded && + jQuery.css( elem, "boxSizing", false, styles ) === "border-box", + valueIsBorderBox = isBorderBox, + + val = curCSS( elem, dimension, styles ), + offsetProp = "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ); + + // Support: Firefox <=54 + // Return a confounding non-pixel value or feign ignorance, as appropriate. + if ( rnumnonpx.test( val ) ) { + if ( !extra ) { + return val; + } + val = "auto"; + } + + + // Support: IE 9 - 11 only + // Use offsetWidth/offsetHeight for when box sizing is unreliable. + // In those cases, the computed value can be trusted to be border-box. + if ( ( !support.boxSizingReliable() && isBorderBox || + + // Support: IE 10 - 11+, Edge 15 - 18+ + // IE/Edge misreport `getComputedStyle` of table rows with width/height + // set in CSS while `offset*` properties report correct values. + // Interestingly, in some cases IE 9 doesn't suffer from this issue. + !support.reliableTrDimensions() && nodeName( elem, "tr" ) || + + // Fall back to offsetWidth/offsetHeight when value is "auto" + // This happens for inline elements with no explicit setting (gh-3571) + val === "auto" || + + // Support: Android <=4.1 - 4.3 only + // Also use offsetWidth/offsetHeight for misreported inline dimensions (gh-3602) + !parseFloat( val ) && jQuery.css( elem, "display", false, styles ) === "inline" ) && + + // Make sure the element is visible & connected + elem.getClientRects().length ) { + + isBorderBox = jQuery.css( elem, "boxSizing", false, styles ) === "border-box"; + + // Where available, offsetWidth/offsetHeight approximate border box dimensions. + // Where not available (e.g., SVG), assume unreliable box-sizing and interpret the + // retrieved value as a content box dimension. + valueIsBorderBox = offsetProp in elem; + if ( valueIsBorderBox ) { + val = elem[ offsetProp ]; + } + } + + // Normalize "" and auto + val = parseFloat( val ) || 0; + + // Adjust for the element's box model + return ( val + + boxModelAdjustment( + elem, + dimension, + extra || ( isBorderBox ? "border" : "content" ), + valueIsBorderBox, + styles, + + // Provide the current computed size to request scroll gutter calculation (gh-3589) + val + ) + ) + "px"; +} + +jQuery.extend( { + + // Add in style property hooks for overriding the default + // behavior of getting and setting a style property + cssHooks: { + opacity: { + get: function( elem, computed ) { + if ( computed ) { + + // We should always get a number back from opacity + var ret = curCSS( elem, "opacity" ); + return ret === "" ? "1" : ret; + } + } + } + }, + + // Don't automatically add "px" to these possibly-unitless properties + cssNumber: { + "animationIterationCount": true, + "columnCount": true, + "fillOpacity": true, + "flexGrow": true, + "flexShrink": true, + "fontWeight": true, + "gridArea": true, + "gridColumn": true, + "gridColumnEnd": true, + "gridColumnStart": true, + "gridRow": true, + "gridRowEnd": true, + "gridRowStart": true, + "lineHeight": true, + "opacity": true, + "order": true, + "orphans": true, + "widows": true, + "zIndex": true, + "zoom": true + }, + + // Add in properties whose names you wish to fix before + // setting or getting the value + cssProps: {}, + + // Get and set the style property on a DOM Node + style: function( elem, name, value, extra ) { + + // Don't set styles on text and comment nodes + if ( !elem || elem.nodeType === 3 || elem.nodeType === 8 || !elem.style ) { + return; + } + + // Make sure that we're working with the right name + var ret, type, hooks, + origName = camelCase( name ), + isCustomProp = rcustomProp.test( name ), + style = elem.style; + + // Make sure that we're working with the right name. We don't + // want to query the value if it is a CSS custom property + // since they are user-defined. + if ( !isCustomProp ) { + name = finalPropName( origName ); + } + + // Gets hook for the prefixed version, then unprefixed version + hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; + + // Check if we're setting a value + if ( value !== undefined ) { + type = typeof value; + + // Convert "+=" or "-=" to relative numbers (#7345) + if ( type === "string" && ( ret = rcssNum.exec( value ) ) && ret[ 1 ] ) { + value = adjustCSS( elem, name, ret ); + + // Fixes bug #9237 + type = "number"; + } + + // Make sure that null and NaN values aren't set (#7116) + if ( value == null || value !== value ) { + return; + } + + // If a number was passed in, add the unit (except for certain CSS properties) + // The isCustomProp check can be removed in jQuery 4.0 when we only auto-append + // "px" to a few hardcoded values. + if ( type === "number" && !isCustomProp ) { + value += ret && ret[ 3 ] || ( jQuery.cssNumber[ origName ] ? "" : "px" ); + } + + // background-* props affect original clone's values + if ( !support.clearCloneStyle && value === "" && name.indexOf( "background" ) === 0 ) { + style[ name ] = "inherit"; + } + + // If a hook was provided, use that value, otherwise just set the specified value + if ( !hooks || !( "set" in hooks ) || + ( value = hooks.set( elem, value, extra ) ) !== undefined ) { + + if ( isCustomProp ) { + style.setProperty( name, value ); + } else { + style[ name ] = value; + } + } + + } else { + + // If a hook was provided get the non-computed value from there + if ( hooks && "get" in hooks && + ( ret = hooks.get( elem, false, extra ) ) !== undefined ) { + + return ret; + } + + // Otherwise just get the value from the style object + return style[ name ]; + } + }, + + css: function( elem, name, extra, styles ) { + var val, num, hooks, + origName = camelCase( name ), + isCustomProp = rcustomProp.test( name ); + + // Make sure that we're working with the right name. We don't + // want to modify the value if it is a CSS custom property + // since they are user-defined. + if ( !isCustomProp ) { + name = finalPropName( origName ); + } + + // Try prefixed name followed by the unprefixed name + hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; + + // If a hook was provided get the computed value from there + if ( hooks && "get" in hooks ) { + val = hooks.get( elem, true, extra ); + } + + // Otherwise, if a way to get the computed value exists, use that + if ( val === undefined ) { + val = curCSS( elem, name, styles ); + } + + // Convert "normal" to computed value + if ( val === "normal" && name in cssNormalTransform ) { + val = cssNormalTransform[ name ]; + } + + // Make numeric if forced or a qualifier was provided and val looks numeric + if ( extra === "" || extra ) { + num = parseFloat( val ); + return extra === true || isFinite( num ) ? num || 0 : val; + } + + return val; + } +} ); + +jQuery.each( [ "height", "width" ], function( _i, dimension ) { + jQuery.cssHooks[ dimension ] = { + get: function( elem, computed, extra ) { + if ( computed ) { + + // Certain elements can have dimension info if we invisibly show them + // but it must have a current display style that would benefit + return rdisplayswap.test( jQuery.css( elem, "display" ) ) && + + // Support: Safari 8+ + // Table columns in Safari have non-zero offsetWidth & zero + // getBoundingClientRect().width unless display is changed. + // Support: IE <=11 only + // Running getBoundingClientRect on a disconnected node + // in IE throws an error. + ( !elem.getClientRects().length || !elem.getBoundingClientRect().width ) ? + swap( elem, cssShow, function() { + return getWidthOrHeight( elem, dimension, extra ); + } ) : + getWidthOrHeight( elem, dimension, extra ); + } + }, + + set: function( elem, value, extra ) { + var matches, + styles = getStyles( elem ), + + // Only read styles.position if the test has a chance to fail + // to avoid forcing a reflow. + scrollboxSizeBuggy = !support.scrollboxSize() && + styles.position === "absolute", + + // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-3991) + boxSizingNeeded = scrollboxSizeBuggy || extra, + isBorderBox = boxSizingNeeded && + jQuery.css( elem, "boxSizing", false, styles ) === "border-box", + subtract = extra ? + boxModelAdjustment( + elem, + dimension, + extra, + isBorderBox, + styles + ) : + 0; + + // Account for unreliable border-box dimensions by comparing offset* to computed and + // faking a content-box to get border and padding (gh-3699) + if ( isBorderBox && scrollboxSizeBuggy ) { + subtract -= Math.ceil( + elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - + parseFloat( styles[ dimension ] ) - + boxModelAdjustment( elem, dimension, "border", false, styles ) - + 0.5 + ); + } + + // Convert to pixels if value adjustment is needed + if ( subtract && ( matches = rcssNum.exec( value ) ) && + ( matches[ 3 ] || "px" ) !== "px" ) { + + elem.style[ dimension ] = value; + value = jQuery.css( elem, dimension ); + } + + return setPositiveNumber( elem, value, subtract ); + } + }; +} ); + +jQuery.cssHooks.marginLeft = addGetHookIf( support.reliableMarginLeft, + function( elem, computed ) { + if ( computed ) { + return ( parseFloat( curCSS( elem, "marginLeft" ) ) || + elem.getBoundingClientRect().left - + swap( elem, { marginLeft: 0 }, function() { + return elem.getBoundingClientRect().left; + } ) + ) + "px"; + } + } +); + +// These hooks are used by animate to expand properties +jQuery.each( { + margin: "", + padding: "", + border: "Width" +}, function( prefix, suffix ) { + jQuery.cssHooks[ prefix + suffix ] = { + expand: function( value ) { + var i = 0, + expanded = {}, + + // Assumes a single number if not a string + parts = typeof value === "string" ? value.split( " " ) : [ value ]; + + for ( ; i < 4; i++ ) { + expanded[ prefix + cssExpand[ i ] + suffix ] = + parts[ i ] || parts[ i - 2 ] || parts[ 0 ]; + } + + return expanded; + } + }; + + if ( prefix !== "margin" ) { + jQuery.cssHooks[ prefix + suffix ].set = setPositiveNumber; + } +} ); + +jQuery.fn.extend( { + css: function( name, value ) { + return access( this, function( elem, name, value ) { + var styles, len, + map = {}, + i = 0; + + if ( Array.isArray( name ) ) { + styles = getStyles( elem ); + len = name.length; + + for ( ; i < len; i++ ) { + map[ name[ i ] ] = jQuery.css( elem, name[ i ], false, styles ); + } + + return map; + } + + return value !== undefined ? + jQuery.style( elem, name, value ) : + jQuery.css( elem, name ); + }, name, value, arguments.length > 1 ); + } +} ); + + +function Tween( elem, options, prop, end, easing ) { + return new Tween.prototype.init( elem, options, prop, end, easing ); +} +jQuery.Tween = Tween; + +Tween.prototype = { + constructor: Tween, + init: function( elem, options, prop, end, easing, unit ) { + this.elem = elem; + this.prop = prop; + this.easing = easing || jQuery.easing._default; + this.options = options; + this.start = this.now = this.cur(); + this.end = end; + this.unit = unit || ( jQuery.cssNumber[ prop ] ? "" : "px" ); + }, + cur: function() { + var hooks = Tween.propHooks[ this.prop ]; + + return hooks && hooks.get ? + hooks.get( this ) : + Tween.propHooks._default.get( this ); + }, + run: function( percent ) { + var eased, + hooks = Tween.propHooks[ this.prop ]; + + if ( this.options.duration ) { + this.pos = eased = jQuery.easing[ this.easing ]( + percent, this.options.duration * percent, 0, 1, this.options.duration + ); + } else { + this.pos = eased = percent; + } + this.now = ( this.end - this.start ) * eased + this.start; + + if ( this.options.step ) { + this.options.step.call( this.elem, this.now, this ); + } + + if ( hooks && hooks.set ) { + hooks.set( this ); + } else { + Tween.propHooks._default.set( this ); + } + return this; + } +}; + +Tween.prototype.init.prototype = Tween.prototype; + +Tween.propHooks = { + _default: { + get: function( tween ) { + var result; + + // Use a property on the element directly when it is not a DOM element, + // or when there is no matching style property that exists. + if ( tween.elem.nodeType !== 1 || + tween.elem[ tween.prop ] != null && tween.elem.style[ tween.prop ] == null ) { + return tween.elem[ tween.prop ]; + } + + // Passing an empty string as a 3rd parameter to .css will automatically + // attempt a parseFloat and fallback to a string if the parse fails. + // Simple values such as "10px" are parsed to Float; + // complex values such as "rotate(1rad)" are returned as-is. + result = jQuery.css( tween.elem, tween.prop, "" ); + + // Empty strings, null, undefined and "auto" are converted to 0. + return !result || result === "auto" ? 0 : result; + }, + set: function( tween ) { + + // Use step hook for back compat. + // Use cssHook if its there. + // Use .style if available and use plain properties where available. + if ( jQuery.fx.step[ tween.prop ] ) { + jQuery.fx.step[ tween.prop ]( tween ); + } else if ( tween.elem.nodeType === 1 && ( + jQuery.cssHooks[ tween.prop ] || + tween.elem.style[ finalPropName( tween.prop ) ] != null ) ) { + jQuery.style( tween.elem, tween.prop, tween.now + tween.unit ); + } else { + tween.elem[ tween.prop ] = tween.now; + } + } + } +}; + +// Support: IE <=9 only +// Panic based approach to setting things on disconnected nodes +Tween.propHooks.scrollTop = Tween.propHooks.scrollLeft = { + set: function( tween ) { + if ( tween.elem.nodeType && tween.elem.parentNode ) { + tween.elem[ tween.prop ] = tween.now; + } + } +}; + +jQuery.easing = { + linear: function( p ) { + return p; + }, + swing: function( p ) { + return 0.5 - Math.cos( p * Math.PI ) / 2; + }, + _default: "swing" +}; + +jQuery.fx = Tween.prototype.init; + +// Back compat <1.8 extension point +jQuery.fx.step = {}; + + + + +var + fxNow, inProgress, + rfxtypes = /^(?:toggle|show|hide)$/, + rrun = /queueHooks$/; + +function schedule() { + if ( inProgress ) { + if ( document.hidden === false && window.requestAnimationFrame ) { + window.requestAnimationFrame( schedule ); + } else { + window.setTimeout( schedule, jQuery.fx.interval ); + } + + jQuery.fx.tick(); + } +} + +// Animations created synchronously will run synchronously +function createFxNow() { + window.setTimeout( function() { + fxNow = undefined; + } ); + return ( fxNow = Date.now() ); +} + +// Generate parameters to create a standard animation +function genFx( type, includeWidth ) { + var which, + i = 0, + attrs = { height: type }; + + // If we include width, step value is 1 to do all cssExpand values, + // otherwise step value is 2 to skip over Left and Right + includeWidth = includeWidth ? 1 : 0; + for ( ; i < 4; i += 2 - includeWidth ) { + which = cssExpand[ i ]; + attrs[ "margin" + which ] = attrs[ "padding" + which ] = type; + } + + if ( includeWidth ) { + attrs.opacity = attrs.width = type; + } + + return attrs; +} + +function createTween( value, prop, animation ) { + var tween, + collection = ( Animation.tweeners[ prop ] || [] ).concat( Animation.tweeners[ "*" ] ), + index = 0, + length = collection.length; + for ( ; index < length; index++ ) { + if ( ( tween = collection[ index ].call( animation, prop, value ) ) ) { + + // We're done with this property + return tween; + } + } +} + +function defaultPrefilter( elem, props, opts ) { + var prop, value, toggle, hooks, oldfire, propTween, restoreDisplay, display, + isBox = "width" in props || "height" in props, + anim = this, + orig = {}, + style = elem.style, + hidden = elem.nodeType && isHiddenWithinTree( elem ), + dataShow = dataPriv.get( elem, "fxshow" ); + + // Queue-skipping animations hijack the fx hooks + if ( !opts.queue ) { + hooks = jQuery._queueHooks( elem, "fx" ); + if ( hooks.unqueued == null ) { + hooks.unqueued = 0; + oldfire = hooks.empty.fire; + hooks.empty.fire = function() { + if ( !hooks.unqueued ) { + oldfire(); + } + }; + } + hooks.unqueued++; + + anim.always( function() { + + // Ensure the complete handler is called before this completes + anim.always( function() { + hooks.unqueued--; + if ( !jQuery.queue( elem, "fx" ).length ) { + hooks.empty.fire(); + } + } ); + } ); + } + + // Detect show/hide animations + for ( prop in props ) { + value = props[ prop ]; + if ( rfxtypes.test( value ) ) { + delete props[ prop ]; + toggle = toggle || value === "toggle"; + if ( value === ( hidden ? "hide" : "show" ) ) { + + // Pretend to be hidden if this is a "show" and + // there is still data from a stopped show/hide + if ( value === "show" && dataShow && dataShow[ prop ] !== undefined ) { + hidden = true; + + // Ignore all other no-op show/hide data + } else { + continue; + } + } + orig[ prop ] = dataShow && dataShow[ prop ] || jQuery.style( elem, prop ); + } + } + + // Bail out if this is a no-op like .hide().hide() + propTween = !jQuery.isEmptyObject( props ); + if ( !propTween && jQuery.isEmptyObject( orig ) ) { + return; + } + + // Restrict "overflow" and "display" styles during box animations + if ( isBox && elem.nodeType === 1 ) { + + // Support: IE <=9 - 11, Edge 12 - 15 + // Record all 3 overflow attributes because IE does not infer the shorthand + // from identically-valued overflowX and overflowY and Edge just mirrors + // the overflowX value there. + opts.overflow = [ style.overflow, style.overflowX, style.overflowY ]; + + // Identify a display type, preferring old show/hide data over the CSS cascade + restoreDisplay = dataShow && dataShow.display; + if ( restoreDisplay == null ) { + restoreDisplay = dataPriv.get( elem, "display" ); + } + display = jQuery.css( elem, "display" ); + if ( display === "none" ) { + if ( restoreDisplay ) { + display = restoreDisplay; + } else { + + // Get nonempty value(s) by temporarily forcing visibility + showHide( [ elem ], true ); + restoreDisplay = elem.style.display || restoreDisplay; + display = jQuery.css( elem, "display" ); + showHide( [ elem ] ); + } + } + + // Animate inline elements as inline-block + if ( display === "inline" || display === "inline-block" && restoreDisplay != null ) { + if ( jQuery.css( elem, "float" ) === "none" ) { + + // Restore the original display value at the end of pure show/hide animations + if ( !propTween ) { + anim.done( function() { + style.display = restoreDisplay; + } ); + if ( restoreDisplay == null ) { + display = style.display; + restoreDisplay = display === "none" ? "" : display; + } + } + style.display = "inline-block"; + } + } + } + + if ( opts.overflow ) { + style.overflow = "hidden"; + anim.always( function() { + style.overflow = opts.overflow[ 0 ]; + style.overflowX = opts.overflow[ 1 ]; + style.overflowY = opts.overflow[ 2 ]; + } ); + } + + // Implement show/hide animations + propTween = false; + for ( prop in orig ) { + + // General show/hide setup for this element animation + if ( !propTween ) { + if ( dataShow ) { + if ( "hidden" in dataShow ) { + hidden = dataShow.hidden; + } + } else { + dataShow = dataPriv.access( elem, "fxshow", { display: restoreDisplay } ); + } + + // Store hidden/visible for toggle so `.stop().toggle()` "reverses" + if ( toggle ) { + dataShow.hidden = !hidden; + } + + // Show elements before animating them + if ( hidden ) { + showHide( [ elem ], true ); + } + + /* eslint-disable no-loop-func */ + + anim.done( function() { + + /* eslint-enable no-loop-func */ + + // The final step of a "hide" animation is actually hiding the element + if ( !hidden ) { + showHide( [ elem ] ); + } + dataPriv.remove( elem, "fxshow" ); + for ( prop in orig ) { + jQuery.style( elem, prop, orig[ prop ] ); + } + } ); + } + + // Per-property setup + propTween = createTween( hidden ? dataShow[ prop ] : 0, prop, anim ); + if ( !( prop in dataShow ) ) { + dataShow[ prop ] = propTween.start; + if ( hidden ) { + propTween.end = propTween.start; + propTween.start = 0; + } + } + } +} + +function propFilter( props, specialEasing ) { + var index, name, easing, value, hooks; + + // camelCase, specialEasing and expand cssHook pass + for ( index in props ) { + name = camelCase( index ); + easing = specialEasing[ name ]; + value = props[ index ]; + if ( Array.isArray( value ) ) { + easing = value[ 1 ]; + value = props[ index ] = value[ 0 ]; + } + + if ( index !== name ) { + props[ name ] = value; + delete props[ index ]; + } + + hooks = jQuery.cssHooks[ name ]; + if ( hooks && "expand" in hooks ) { + value = hooks.expand( value ); + delete props[ name ]; + + // Not quite $.extend, this won't overwrite existing keys. + // Reusing 'index' because we have the correct "name" + for ( index in value ) { + if ( !( index in props ) ) { + props[ index ] = value[ index ]; + specialEasing[ index ] = easing; + } + } + } else { + specialEasing[ name ] = easing; + } + } +} + +function Animation( elem, properties, options ) { + var result, + stopped, + index = 0, + length = Animation.prefilters.length, + deferred = jQuery.Deferred().always( function() { + + // Don't match elem in the :animated selector + delete tick.elem; + } ), + tick = function() { + if ( stopped ) { + return false; + } + var currentTime = fxNow || createFxNow(), + remaining = Math.max( 0, animation.startTime + animation.duration - currentTime ), + + // Support: Android 2.3 only + // Archaic crash bug won't allow us to use `1 - ( 0.5 || 0 )` (#12497) + temp = remaining / animation.duration || 0, + percent = 1 - temp, + index = 0, + length = animation.tweens.length; + + for ( ; index < length; index++ ) { + animation.tweens[ index ].run( percent ); + } + + deferred.notifyWith( elem, [ animation, percent, remaining ] ); + + // If there's more to do, yield + if ( percent < 1 && length ) { + return remaining; + } + + // If this was an empty animation, synthesize a final progress notification + if ( !length ) { + deferred.notifyWith( elem, [ animation, 1, 0 ] ); + } + + // Resolve the animation and report its conclusion + deferred.resolveWith( elem, [ animation ] ); + return false; + }, + animation = deferred.promise( { + elem: elem, + props: jQuery.extend( {}, properties ), + opts: jQuery.extend( true, { + specialEasing: {}, + easing: jQuery.easing._default + }, options ), + originalProperties: properties, + originalOptions: options, + startTime: fxNow || createFxNow(), + duration: options.duration, + tweens: [], + createTween: function( prop, end ) { + var tween = jQuery.Tween( elem, animation.opts, prop, end, + animation.opts.specialEasing[ prop ] || animation.opts.easing ); + animation.tweens.push( tween ); + return tween; + }, + stop: function( gotoEnd ) { + var index = 0, + + // If we are going to the end, we want to run all the tweens + // otherwise we skip this part + length = gotoEnd ? animation.tweens.length : 0; + if ( stopped ) { + return this; + } + stopped = true; + for ( ; index < length; index++ ) { + animation.tweens[ index ].run( 1 ); + } + + // Resolve when we played the last frame; otherwise, reject + if ( gotoEnd ) { + deferred.notifyWith( elem, [ animation, 1, 0 ] ); + deferred.resolveWith( elem, [ animation, gotoEnd ] ); + } else { + deferred.rejectWith( elem, [ animation, gotoEnd ] ); + } + return this; + } + } ), + props = animation.props; + + propFilter( props, animation.opts.specialEasing ); + + for ( ; index < length; index++ ) { + result = Animation.prefilters[ index ].call( animation, elem, props, animation.opts ); + if ( result ) { + if ( isFunction( result.stop ) ) { + jQuery._queueHooks( animation.elem, animation.opts.queue ).stop = + result.stop.bind( result ); + } + return result; + } + } + + jQuery.map( props, createTween, animation ); + + if ( isFunction( animation.opts.start ) ) { + animation.opts.start.call( elem, animation ); + } + + // Attach callbacks from options + animation + .progress( animation.opts.progress ) + .done( animation.opts.done, animation.opts.complete ) + .fail( animation.opts.fail ) + .always( animation.opts.always ); + + jQuery.fx.timer( + jQuery.extend( tick, { + elem: elem, + anim: animation, + queue: animation.opts.queue + } ) + ); + + return animation; +} + +jQuery.Animation = jQuery.extend( Animation, { + + tweeners: { + "*": [ function( prop, value ) { + var tween = this.createTween( prop, value ); + adjustCSS( tween.elem, prop, rcssNum.exec( value ), tween ); + return tween; + } ] + }, + + tweener: function( props, callback ) { + if ( isFunction( props ) ) { + callback = props; + props = [ "*" ]; + } else { + props = props.match( rnothtmlwhite ); + } + + var prop, + index = 0, + length = props.length; + + for ( ; index < length; index++ ) { + prop = props[ index ]; + Animation.tweeners[ prop ] = Animation.tweeners[ prop ] || []; + Animation.tweeners[ prop ].unshift( callback ); + } + }, + + prefilters: [ defaultPrefilter ], + + prefilter: function( callback, prepend ) { + if ( prepend ) { + Animation.prefilters.unshift( callback ); + } else { + Animation.prefilters.push( callback ); + } + } +} ); + +jQuery.speed = function( speed, easing, fn ) { + var opt = speed && typeof speed === "object" ? jQuery.extend( {}, speed ) : { + complete: fn || !fn && easing || + isFunction( speed ) && speed, + duration: speed, + easing: fn && easing || easing && !isFunction( easing ) && easing + }; + + // Go to the end state if fx are off + if ( jQuery.fx.off ) { + opt.duration = 0; + + } else { + if ( typeof opt.duration !== "number" ) { + if ( opt.duration in jQuery.fx.speeds ) { + opt.duration = jQuery.fx.speeds[ opt.duration ]; + + } else { + opt.duration = jQuery.fx.speeds._default; + } + } + } + + // Normalize opt.queue - true/undefined/null -> "fx" + if ( opt.queue == null || opt.queue === true ) { + opt.queue = "fx"; + } + + // Queueing + opt.old = opt.complete; + + opt.complete = function() { + if ( isFunction( opt.old ) ) { + opt.old.call( this ); + } + + if ( opt.queue ) { + jQuery.dequeue( this, opt.queue ); + } + }; + + return opt; +}; + +jQuery.fn.extend( { + fadeTo: function( speed, to, easing, callback ) { + + // Show any hidden elements after setting opacity to 0 + return this.filter( isHiddenWithinTree ).css( "opacity", 0 ).show() + + // Animate to the value specified + .end().animate( { opacity: to }, speed, easing, callback ); + }, + animate: function( prop, speed, easing, callback ) { + var empty = jQuery.isEmptyObject( prop ), + optall = jQuery.speed( speed, easing, callback ), + doAnimation = function() { + + // Operate on a copy of prop so per-property easing won't be lost + var anim = Animation( this, jQuery.extend( {}, prop ), optall ); + + // Empty animations, or finishing resolves immediately + if ( empty || dataPriv.get( this, "finish" ) ) { + anim.stop( true ); + } + }; + + doAnimation.finish = doAnimation; + + return empty || optall.queue === false ? + this.each( doAnimation ) : + this.queue( optall.queue, doAnimation ); + }, + stop: function( type, clearQueue, gotoEnd ) { + var stopQueue = function( hooks ) { + var stop = hooks.stop; + delete hooks.stop; + stop( gotoEnd ); + }; + + if ( typeof type !== "string" ) { + gotoEnd = clearQueue; + clearQueue = type; + type = undefined; + } + if ( clearQueue ) { + this.queue( type || "fx", [] ); + } + + return this.each( function() { + var dequeue = true, + index = type != null && type + "queueHooks", + timers = jQuery.timers, + data = dataPriv.get( this ); + + if ( index ) { + if ( data[ index ] && data[ index ].stop ) { + stopQueue( data[ index ] ); + } + } else { + for ( index in data ) { + if ( data[ index ] && data[ index ].stop && rrun.test( index ) ) { + stopQueue( data[ index ] ); + } + } + } + + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && + ( type == null || timers[ index ].queue === type ) ) { + + timers[ index ].anim.stop( gotoEnd ); + dequeue = false; + timers.splice( index, 1 ); + } + } + + // Start the next in the queue if the last step wasn't forced. + // Timers currently will call their complete callbacks, which + // will dequeue but only if they were gotoEnd. + if ( dequeue || !gotoEnd ) { + jQuery.dequeue( this, type ); + } + } ); + }, + finish: function( type ) { + if ( type !== false ) { + type = type || "fx"; + } + return this.each( function() { + var index, + data = dataPriv.get( this ), + queue = data[ type + "queue" ], + hooks = data[ type + "queueHooks" ], + timers = jQuery.timers, + length = queue ? queue.length : 0; + + // Enable finishing flag on private data + data.finish = true; + + // Empty the queue first + jQuery.queue( this, type, [] ); + + if ( hooks && hooks.stop ) { + hooks.stop.call( this, true ); + } + + // Look for any active animations, and finish them + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && timers[ index ].queue === type ) { + timers[ index ].anim.stop( true ); + timers.splice( index, 1 ); + } + } + + // Look for any animations in the old queue and finish them + for ( index = 0; index < length; index++ ) { + if ( queue[ index ] && queue[ index ].finish ) { + queue[ index ].finish.call( this ); + } + } + + // Turn off finishing flag + delete data.finish; + } ); + } +} ); + +jQuery.each( [ "toggle", "show", "hide" ], function( _i, name ) { + var cssFn = jQuery.fn[ name ]; + jQuery.fn[ name ] = function( speed, easing, callback ) { + return speed == null || typeof speed === "boolean" ? + cssFn.apply( this, arguments ) : + this.animate( genFx( name, true ), speed, easing, callback ); + }; +} ); + +// Generate shortcuts for custom animations +jQuery.each( { + slideDown: genFx( "show" ), + slideUp: genFx( "hide" ), + slideToggle: genFx( "toggle" ), + fadeIn: { opacity: "show" }, + fadeOut: { opacity: "hide" }, + fadeToggle: { opacity: "toggle" } +}, function( name, props ) { + jQuery.fn[ name ] = function( speed, easing, callback ) { + return this.animate( props, speed, easing, callback ); + }; +} ); + +jQuery.timers = []; +jQuery.fx.tick = function() { + var timer, + i = 0, + timers = jQuery.timers; + + fxNow = Date.now(); + + for ( ; i < timers.length; i++ ) { + timer = timers[ i ]; + + // Run the timer and safely remove it when done (allowing for external removal) + if ( !timer() && timers[ i ] === timer ) { + timers.splice( i--, 1 ); + } + } + + if ( !timers.length ) { + jQuery.fx.stop(); + } + fxNow = undefined; +}; + +jQuery.fx.timer = function( timer ) { + jQuery.timers.push( timer ); + jQuery.fx.start(); +}; + +jQuery.fx.interval = 13; +jQuery.fx.start = function() { + if ( inProgress ) { + return; + } + + inProgress = true; + schedule(); +}; + +jQuery.fx.stop = function() { + inProgress = null; +}; + +jQuery.fx.speeds = { + slow: 600, + fast: 200, + + // Default speed + _default: 400 +}; + + +// Based off of the plugin by Clint Helfers, with permission. +// https://web.archive.org/web/20100324014747/http://blindsignals.com/index.php/2009/07/jquery-delay/ +jQuery.fn.delay = function( time, type ) { + time = jQuery.fx ? jQuery.fx.speeds[ time ] || time : time; + type = type || "fx"; + + return this.queue( type, function( next, hooks ) { + var timeout = window.setTimeout( next, time ); + hooks.stop = function() { + window.clearTimeout( timeout ); + }; + } ); +}; + + +( function() { + var input = document.createElement( "input" ), + select = document.createElement( "select" ), + opt = select.appendChild( document.createElement( "option" ) ); + + input.type = "checkbox"; + + // Support: Android <=4.3 only + // Default value for a checkbox should be "on" + support.checkOn = input.value !== ""; + + // Support: IE <=11 only + // Must access selectedIndex to make default options select + support.optSelected = opt.selected; + + // Support: IE <=11 only + // An input loses its value after becoming a radio + input = document.createElement( "input" ); + input.value = "t"; + input.type = "radio"; + support.radioValue = input.value === "t"; +} )(); + + +var boolHook, + attrHandle = jQuery.expr.attrHandle; + +jQuery.fn.extend( { + attr: function( name, value ) { + return access( this, jQuery.attr, name, value, arguments.length > 1 ); + }, + + removeAttr: function( name ) { + return this.each( function() { + jQuery.removeAttr( this, name ); + } ); + } +} ); + +jQuery.extend( { + attr: function( elem, name, value ) { + var ret, hooks, + nType = elem.nodeType; + + // Don't get/set attributes on text, comment and attribute nodes + if ( nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + // Fallback to prop when attributes are not supported + if ( typeof elem.getAttribute === "undefined" ) { + return jQuery.prop( elem, name, value ); + } + + // Attribute hooks are determined by the lowercase version + // Grab necessary hook if one is defined + if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { + hooks = jQuery.attrHooks[ name.toLowerCase() ] || + ( jQuery.expr.match.bool.test( name ) ? boolHook : undefined ); + } + + if ( value !== undefined ) { + if ( value === null ) { + jQuery.removeAttr( elem, name ); + return; + } + + if ( hooks && "set" in hooks && + ( ret = hooks.set( elem, value, name ) ) !== undefined ) { + return ret; + } + + elem.setAttribute( name, value + "" ); + return value; + } + + if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { + return ret; + } + + ret = jQuery.find.attr( elem, name ); + + // Non-existent attributes return null, we normalize to undefined + return ret == null ? undefined : ret; + }, + + attrHooks: { + type: { + set: function( elem, value ) { + if ( !support.radioValue && value === "radio" && + nodeName( elem, "input" ) ) { + var val = elem.value; + elem.setAttribute( "type", value ); + if ( val ) { + elem.value = val; + } + return value; + } + } + } + }, + + removeAttr: function( elem, value ) { + var name, + i = 0, + + // Attribute names can contain non-HTML whitespace characters + // https://html.spec.whatwg.org/multipage/syntax.html#attributes-2 + attrNames = value && value.match( rnothtmlwhite ); + + if ( attrNames && elem.nodeType === 1 ) { + while ( ( name = attrNames[ i++ ] ) ) { + elem.removeAttribute( name ); + } + } + } +} ); + +// Hooks for boolean attributes +boolHook = { + set: function( elem, value, name ) { + if ( value === false ) { + + // Remove boolean attributes when set to false + jQuery.removeAttr( elem, name ); + } else { + elem.setAttribute( name, name ); + } + return name; + } +}; + +jQuery.each( jQuery.expr.match.bool.source.match( /\w+/g ), function( _i, name ) { + var getter = attrHandle[ name ] || jQuery.find.attr; + + attrHandle[ name ] = function( elem, name, isXML ) { + var ret, handle, + lowercaseName = name.toLowerCase(); + + if ( !isXML ) { + + // Avoid an infinite loop by temporarily removing this function from the getter + handle = attrHandle[ lowercaseName ]; + attrHandle[ lowercaseName ] = ret; + ret = getter( elem, name, isXML ) != null ? + lowercaseName : + null; + attrHandle[ lowercaseName ] = handle; + } + return ret; + }; +} ); + + + + +var rfocusable = /^(?:input|select|textarea|button)$/i, + rclickable = /^(?:a|area)$/i; + +jQuery.fn.extend( { + prop: function( name, value ) { + return access( this, jQuery.prop, name, value, arguments.length > 1 ); + }, + + removeProp: function( name ) { + return this.each( function() { + delete this[ jQuery.propFix[ name ] || name ]; + } ); + } +} ); + +jQuery.extend( { + prop: function( elem, name, value ) { + var ret, hooks, + nType = elem.nodeType; + + // Don't get/set properties on text, comment and attribute nodes + if ( nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { + + // Fix name and attach hooks + name = jQuery.propFix[ name ] || name; + hooks = jQuery.propHooks[ name ]; + } + + if ( value !== undefined ) { + if ( hooks && "set" in hooks && + ( ret = hooks.set( elem, value, name ) ) !== undefined ) { + return ret; + } + + return ( elem[ name ] = value ); + } + + if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { + return ret; + } + + return elem[ name ]; + }, + + propHooks: { + tabIndex: { + get: function( elem ) { + + // Support: IE <=9 - 11 only + // elem.tabIndex doesn't always return the + // correct value when it hasn't been explicitly set + // https://web.archive.org/web/20141116233347/http://fluidproject.org/blog/2008/01/09/getting-setting-and-removing-tabindex-values-with-javascript/ + // Use proper attribute retrieval(#12072) + var tabindex = jQuery.find.attr( elem, "tabindex" ); + + if ( tabindex ) { + return parseInt( tabindex, 10 ); + } + + if ( + rfocusable.test( elem.nodeName ) || + rclickable.test( elem.nodeName ) && + elem.href + ) { + return 0; + } + + return -1; + } + } + }, + + propFix: { + "for": "htmlFor", + "class": "className" + } +} ); + +// Support: IE <=11 only +// Accessing the selectedIndex property +// forces the browser to respect setting selected +// on the option +// The getter ensures a default option is selected +// when in an optgroup +// eslint rule "no-unused-expressions" is disabled for this code +// since it considers such accessions noop +if ( !support.optSelected ) { + jQuery.propHooks.selected = { + get: function( elem ) { + + /* eslint no-unused-expressions: "off" */ + + var parent = elem.parentNode; + if ( parent && parent.parentNode ) { + parent.parentNode.selectedIndex; + } + return null; + }, + set: function( elem ) { + + /* eslint no-unused-expressions: "off" */ + + var parent = elem.parentNode; + if ( parent ) { + parent.selectedIndex; + + if ( parent.parentNode ) { + parent.parentNode.selectedIndex; + } + } + } + }; +} + +jQuery.each( [ + "tabIndex", + "readOnly", + "maxLength", + "cellSpacing", + "cellPadding", + "rowSpan", + "colSpan", + "useMap", + "frameBorder", + "contentEditable" +], function() { + jQuery.propFix[ this.toLowerCase() ] = this; +} ); + + + + + // Strip and collapse whitespace according to HTML spec + // https://infra.spec.whatwg.org/#strip-and-collapse-ascii-whitespace + function stripAndCollapse( value ) { + var tokens = value.match( rnothtmlwhite ) || []; + return tokens.join( " " ); + } + + +function getClass( elem ) { + return elem.getAttribute && elem.getAttribute( "class" ) || ""; +} + +function classesToArray( value ) { + if ( Array.isArray( value ) ) { + return value; + } + if ( typeof value === "string" ) { + return value.match( rnothtmlwhite ) || []; + } + return []; +} + +jQuery.fn.extend( { + addClass: function( value ) { + var classes, elem, cur, curValue, clazz, j, finalValue, + i = 0; + + if ( isFunction( value ) ) { + return this.each( function( j ) { + jQuery( this ).addClass( value.call( this, j, getClass( this ) ) ); + } ); + } + + classes = classesToArray( value ); + + if ( classes.length ) { + while ( ( elem = this[ i++ ] ) ) { + curValue = getClass( elem ); + cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); + + if ( cur ) { + j = 0; + while ( ( clazz = classes[ j++ ] ) ) { + if ( cur.indexOf( " " + clazz + " " ) < 0 ) { + cur += clazz + " "; + } + } + + // Only assign if different to avoid unneeded rendering. + finalValue = stripAndCollapse( cur ); + if ( curValue !== finalValue ) { + elem.setAttribute( "class", finalValue ); + } + } + } + } + + return this; + }, + + removeClass: function( value ) { + var classes, elem, cur, curValue, clazz, j, finalValue, + i = 0; + + if ( isFunction( value ) ) { + return this.each( function( j ) { + jQuery( this ).removeClass( value.call( this, j, getClass( this ) ) ); + } ); + } + + if ( !arguments.length ) { + return this.attr( "class", "" ); + } + + classes = classesToArray( value ); + + if ( classes.length ) { + while ( ( elem = this[ i++ ] ) ) { + curValue = getClass( elem ); + + // This expression is here for better compressibility (see addClass) + cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); + + if ( cur ) { + j = 0; + while ( ( clazz = classes[ j++ ] ) ) { + + // Remove *all* instances + while ( cur.indexOf( " " + clazz + " " ) > -1 ) { + cur = cur.replace( " " + clazz + " ", " " ); + } + } + + // Only assign if different to avoid unneeded rendering. + finalValue = stripAndCollapse( cur ); + if ( curValue !== finalValue ) { + elem.setAttribute( "class", finalValue ); + } + } + } + } + + return this; + }, + + toggleClass: function( value, stateVal ) { + var type = typeof value, + isValidValue = type === "string" || Array.isArray( value ); + + if ( typeof stateVal === "boolean" && isValidValue ) { + return stateVal ? this.addClass( value ) : this.removeClass( value ); + } + + if ( isFunction( value ) ) { + return this.each( function( i ) { + jQuery( this ).toggleClass( + value.call( this, i, getClass( this ), stateVal ), + stateVal + ); + } ); + } + + return this.each( function() { + var className, i, self, classNames; + + if ( isValidValue ) { + + // Toggle individual class names + i = 0; + self = jQuery( this ); + classNames = classesToArray( value ); + + while ( ( className = classNames[ i++ ] ) ) { + + // Check each className given, space separated list + if ( self.hasClass( className ) ) { + self.removeClass( className ); + } else { + self.addClass( className ); + } + } + + // Toggle whole class name + } else if ( value === undefined || type === "boolean" ) { + className = getClass( this ); + if ( className ) { + + // Store className if set + dataPriv.set( this, "__className__", className ); + } + + // If the element has a class name or if we're passed `false`, + // then remove the whole classname (if there was one, the above saved it). + // Otherwise bring back whatever was previously saved (if anything), + // falling back to the empty string if nothing was stored. + if ( this.setAttribute ) { + this.setAttribute( "class", + className || value === false ? + "" : + dataPriv.get( this, "__className__" ) || "" + ); + } + } + } ); + }, + + hasClass: function( selector ) { + var className, elem, + i = 0; + + className = " " + selector + " "; + while ( ( elem = this[ i++ ] ) ) { + if ( elem.nodeType === 1 && + ( " " + stripAndCollapse( getClass( elem ) ) + " " ).indexOf( className ) > -1 ) { + return true; + } + } + + return false; + } +} ); + + + + +var rreturn = /\r/g; + +jQuery.fn.extend( { + val: function( value ) { + var hooks, ret, valueIsFunction, + elem = this[ 0 ]; + + if ( !arguments.length ) { + if ( elem ) { + hooks = jQuery.valHooks[ elem.type ] || + jQuery.valHooks[ elem.nodeName.toLowerCase() ]; + + if ( hooks && + "get" in hooks && + ( ret = hooks.get( elem, "value" ) ) !== undefined + ) { + return ret; + } + + ret = elem.value; + + // Handle most common string cases + if ( typeof ret === "string" ) { + return ret.replace( rreturn, "" ); + } + + // Handle cases where value is null/undef or number + return ret == null ? "" : ret; + } + + return; + } + + valueIsFunction = isFunction( value ); + + return this.each( function( i ) { + var val; + + if ( this.nodeType !== 1 ) { + return; + } + + if ( valueIsFunction ) { + val = value.call( this, i, jQuery( this ).val() ); + } else { + val = value; + } + + // Treat null/undefined as ""; convert numbers to string + if ( val == null ) { + val = ""; + + } else if ( typeof val === "number" ) { + val += ""; + + } else if ( Array.isArray( val ) ) { + val = jQuery.map( val, function( value ) { + return value == null ? "" : value + ""; + } ); + } + + hooks = jQuery.valHooks[ this.type ] || jQuery.valHooks[ this.nodeName.toLowerCase() ]; + + // If set returns undefined, fall back to normal setting + if ( !hooks || !( "set" in hooks ) || hooks.set( this, val, "value" ) === undefined ) { + this.value = val; + } + } ); + } +} ); + +jQuery.extend( { + valHooks: { + option: { + get: function( elem ) { + + var val = jQuery.find.attr( elem, "value" ); + return val != null ? + val : + + // Support: IE <=10 - 11 only + // option.text throws exceptions (#14686, #14858) + // Strip and collapse whitespace + // https://html.spec.whatwg.org/#strip-and-collapse-whitespace + stripAndCollapse( jQuery.text( elem ) ); + } + }, + select: { + get: function( elem ) { + var value, option, i, + options = elem.options, + index = elem.selectedIndex, + one = elem.type === "select-one", + values = one ? null : [], + max = one ? index + 1 : options.length; + + if ( index < 0 ) { + i = max; + + } else { + i = one ? index : 0; + } + + // Loop through all the selected options + for ( ; i < max; i++ ) { + option = options[ i ]; + + // Support: IE <=9 only + // IE8-9 doesn't update selected after form reset (#2551) + if ( ( option.selected || i === index ) && + + // Don't return options that are disabled or in a disabled optgroup + !option.disabled && + ( !option.parentNode.disabled || + !nodeName( option.parentNode, "optgroup" ) ) ) { + + // Get the specific value for the option + value = jQuery( option ).val(); + + // We don't need an array for one selects + if ( one ) { + return value; + } + + // Multi-Selects return an array + values.push( value ); + } + } + + return values; + }, + + set: function( elem, value ) { + var optionSet, option, + options = elem.options, + values = jQuery.makeArray( value ), + i = options.length; + + while ( i-- ) { + option = options[ i ]; + + /* eslint-disable no-cond-assign */ + + if ( option.selected = + jQuery.inArray( jQuery.valHooks.option.get( option ), values ) > -1 + ) { + optionSet = true; + } + + /* eslint-enable no-cond-assign */ + } + + // Force browsers to behave consistently when non-matching value is set + if ( !optionSet ) { + elem.selectedIndex = -1; + } + return values; + } + } + } +} ); + +// Radios and checkboxes getter/setter +jQuery.each( [ "radio", "checkbox" ], function() { + jQuery.valHooks[ this ] = { + set: function( elem, value ) { + if ( Array.isArray( value ) ) { + return ( elem.checked = jQuery.inArray( jQuery( elem ).val(), value ) > -1 ); + } + } + }; + if ( !support.checkOn ) { + jQuery.valHooks[ this ].get = function( elem ) { + return elem.getAttribute( "value" ) === null ? "on" : elem.value; + }; + } +} ); + + + + +// Return jQuery for attributes-only inclusion + + +support.focusin = "onfocusin" in window; + + +var rfocusMorph = /^(?:focusinfocus|focusoutblur)$/, + stopPropagationCallback = function( e ) { + e.stopPropagation(); + }; + +jQuery.extend( jQuery.event, { + + trigger: function( event, data, elem, onlyHandlers ) { + + var i, cur, tmp, bubbleType, ontype, handle, special, lastElement, + eventPath = [ elem || document ], + type = hasOwn.call( event, "type" ) ? event.type : event, + namespaces = hasOwn.call( event, "namespace" ) ? event.namespace.split( "." ) : []; + + cur = lastElement = tmp = elem = elem || document; + + // Don't do events on text and comment nodes + if ( elem.nodeType === 3 || elem.nodeType === 8 ) { + return; + } + + // focus/blur morphs to focusin/out; ensure we're not firing them right now + if ( rfocusMorph.test( type + jQuery.event.triggered ) ) { + return; + } + + if ( type.indexOf( "." ) > -1 ) { + + // Namespaced trigger; create a regexp to match event type in handle() + namespaces = type.split( "." ); + type = namespaces.shift(); + namespaces.sort(); + } + ontype = type.indexOf( ":" ) < 0 && "on" + type; + + // Caller can pass in a jQuery.Event object, Object, or just an event type string + event = event[ jQuery.expando ] ? + event : + new jQuery.Event( type, typeof event === "object" && event ); + + // Trigger bitmask: & 1 for native handlers; & 2 for jQuery (always true) + event.isTrigger = onlyHandlers ? 2 : 3; + event.namespace = namespaces.join( "." ); + event.rnamespace = event.namespace ? + new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ) : + null; + + // Clean up the event in case it is being reused + event.result = undefined; + if ( !event.target ) { + event.target = elem; + } + + // Clone any incoming data and prepend the event, creating the handler arg list + data = data == null ? + [ event ] : + jQuery.makeArray( data, [ event ] ); + + // Allow special events to draw outside the lines + special = jQuery.event.special[ type ] || {}; + if ( !onlyHandlers && special.trigger && special.trigger.apply( elem, data ) === false ) { + return; + } + + // Determine event propagation path in advance, per W3C events spec (#9951) + // Bubble up to document, then to window; watch for a global ownerDocument var (#9724) + if ( !onlyHandlers && !special.noBubble && !isWindow( elem ) ) { + + bubbleType = special.delegateType || type; + if ( !rfocusMorph.test( bubbleType + type ) ) { + cur = cur.parentNode; + } + for ( ; cur; cur = cur.parentNode ) { + eventPath.push( cur ); + tmp = cur; + } + + // Only add window if we got to document (e.g., not plain obj or detached DOM) + if ( tmp === ( elem.ownerDocument || document ) ) { + eventPath.push( tmp.defaultView || tmp.parentWindow || window ); + } + } + + // Fire handlers on the event path + i = 0; + while ( ( cur = eventPath[ i++ ] ) && !event.isPropagationStopped() ) { + lastElement = cur; + event.type = i > 1 ? + bubbleType : + special.bindType || type; + + // jQuery handler + handle = ( dataPriv.get( cur, "events" ) || Object.create( null ) )[ event.type ] && + dataPriv.get( cur, "handle" ); + if ( handle ) { + handle.apply( cur, data ); + } + + // Native handler + handle = ontype && cur[ ontype ]; + if ( handle && handle.apply && acceptData( cur ) ) { + event.result = handle.apply( cur, data ); + if ( event.result === false ) { + event.preventDefault(); + } + } + } + event.type = type; + + // If nobody prevented the default action, do it now + if ( !onlyHandlers && !event.isDefaultPrevented() ) { + + if ( ( !special._default || + special._default.apply( eventPath.pop(), data ) === false ) && + acceptData( elem ) ) { + + // Call a native DOM method on the target with the same name as the event. + // Don't do default actions on window, that's where global variables be (#6170) + if ( ontype && isFunction( elem[ type ] ) && !isWindow( elem ) ) { + + // Don't re-trigger an onFOO event when we call its FOO() method + tmp = elem[ ontype ]; + + if ( tmp ) { + elem[ ontype ] = null; + } + + // Prevent re-triggering of the same event, since we already bubbled it above + jQuery.event.triggered = type; + + if ( event.isPropagationStopped() ) { + lastElement.addEventListener( type, stopPropagationCallback ); + } + + elem[ type ](); + + if ( event.isPropagationStopped() ) { + lastElement.removeEventListener( type, stopPropagationCallback ); + } + + jQuery.event.triggered = undefined; + + if ( tmp ) { + elem[ ontype ] = tmp; + } + } + } + } + + return event.result; + }, + + // Piggyback on a donor event to simulate a different one + // Used only for `focus(in | out)` events + simulate: function( type, elem, event ) { + var e = jQuery.extend( + new jQuery.Event(), + event, + { + type: type, + isSimulated: true + } + ); + + jQuery.event.trigger( e, null, elem ); + } + +} ); + +jQuery.fn.extend( { + + trigger: function( type, data ) { + return this.each( function() { + jQuery.event.trigger( type, data, this ); + } ); + }, + triggerHandler: function( type, data ) { + var elem = this[ 0 ]; + if ( elem ) { + return jQuery.event.trigger( type, data, elem, true ); + } + } +} ); + + +// Support: Firefox <=44 +// Firefox doesn't have focus(in | out) events +// Related ticket - https://bugzilla.mozilla.org/show_bug.cgi?id=687787 +// +// Support: Chrome <=48 - 49, Safari <=9.0 - 9.1 +// focus(in | out) events fire after focus & blur events, +// which is spec violation - http://www.w3.org/TR/DOM-Level-3-Events/#events-focusevent-event-order +// Related ticket - https://bugs.chromium.org/p/chromium/issues/detail?id=449857 +if ( !support.focusin ) { + jQuery.each( { focus: "focusin", blur: "focusout" }, function( orig, fix ) { + + // Attach a single capturing handler on the document while someone wants focusin/focusout + var handler = function( event ) { + jQuery.event.simulate( fix, event.target, jQuery.event.fix( event ) ); + }; + + jQuery.event.special[ fix ] = { + setup: function() { + + // Handle: regular nodes (via `this.ownerDocument`), window + // (via `this.document`) & document (via `this`). + var doc = this.ownerDocument || this.document || this, + attaches = dataPriv.access( doc, fix ); + + if ( !attaches ) { + doc.addEventListener( orig, handler, true ); + } + dataPriv.access( doc, fix, ( attaches || 0 ) + 1 ); + }, + teardown: function() { + var doc = this.ownerDocument || this.document || this, + attaches = dataPriv.access( doc, fix ) - 1; + + if ( !attaches ) { + doc.removeEventListener( orig, handler, true ); + dataPriv.remove( doc, fix ); + + } else { + dataPriv.access( doc, fix, attaches ); + } + } + }; + } ); +} +var location = window.location; + +var nonce = { guid: Date.now() }; + +var rquery = ( /\?/ ); + + + +// Cross-browser xml parsing +jQuery.parseXML = function( data ) { + var xml, parserErrorElem; + if ( !data || typeof data !== "string" ) { + return null; + } + + // Support: IE 9 - 11 only + // IE throws on parseFromString with invalid input. + try { + xml = ( new window.DOMParser() ).parseFromString( data, "text/xml" ); + } catch ( e ) {} + + parserErrorElem = xml && xml.getElementsByTagName( "parsererror" )[ 0 ]; + if ( !xml || parserErrorElem ) { + jQuery.error( "Invalid XML: " + ( + parserErrorElem ? + jQuery.map( parserErrorElem.childNodes, function( el ) { + return el.textContent; + } ).join( "\n" ) : + data + ) ); + } + return xml; +}; + + +var + rbracket = /\[\]$/, + rCRLF = /\r?\n/g, + rsubmitterTypes = /^(?:submit|button|image|reset|file)$/i, + rsubmittable = /^(?:input|select|textarea|keygen)/i; + +function buildParams( prefix, obj, traditional, add ) { + var name; + + if ( Array.isArray( obj ) ) { + + // Serialize array item. + jQuery.each( obj, function( i, v ) { + if ( traditional || rbracket.test( prefix ) ) { + + // Treat each array item as a scalar. + add( prefix, v ); + + } else { + + // Item is non-scalar (array or object), encode its numeric index. + buildParams( + prefix + "[" + ( typeof v === "object" && v != null ? i : "" ) + "]", + v, + traditional, + add + ); + } + } ); + + } else if ( !traditional && toType( obj ) === "object" ) { + + // Serialize object item. + for ( name in obj ) { + buildParams( prefix + "[" + name + "]", obj[ name ], traditional, add ); + } + + } else { + + // Serialize scalar item. + add( prefix, obj ); + } +} + +// Serialize an array of form elements or a set of +// key/values into a query string +jQuery.param = function( a, traditional ) { + var prefix, + s = [], + add = function( key, valueOrFunction ) { + + // If value is a function, invoke it and use its return value + var value = isFunction( valueOrFunction ) ? + valueOrFunction() : + valueOrFunction; + + s[ s.length ] = encodeURIComponent( key ) + "=" + + encodeURIComponent( value == null ? "" : value ); + }; + + if ( a == null ) { + return ""; + } + + // If an array was passed in, assume that it is an array of form elements. + if ( Array.isArray( a ) || ( a.jquery && !jQuery.isPlainObject( a ) ) ) { + + // Serialize the form elements + jQuery.each( a, function() { + add( this.name, this.value ); + } ); + + } else { + + // If traditional, encode the "old" way (the way 1.3.2 or older + // did it), otherwise encode params recursively. + for ( prefix in a ) { + buildParams( prefix, a[ prefix ], traditional, add ); + } + } + + // Return the resulting serialization + return s.join( "&" ); +}; + +jQuery.fn.extend( { + serialize: function() { + return jQuery.param( this.serializeArray() ); + }, + serializeArray: function() { + return this.map( function() { + + // Can add propHook for "elements" to filter or add form elements + var elements = jQuery.prop( this, "elements" ); + return elements ? jQuery.makeArray( elements ) : this; + } ).filter( function() { + var type = this.type; + + // Use .is( ":disabled" ) so that fieldset[disabled] works + return this.name && !jQuery( this ).is( ":disabled" ) && + rsubmittable.test( this.nodeName ) && !rsubmitterTypes.test( type ) && + ( this.checked || !rcheckableType.test( type ) ); + } ).map( function( _i, elem ) { + var val = jQuery( this ).val(); + + if ( val == null ) { + return null; + } + + if ( Array.isArray( val ) ) { + return jQuery.map( val, function( val ) { + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + } ); + } + + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + } ).get(); + } +} ); + + +var + r20 = /%20/g, + rhash = /#.*$/, + rantiCache = /([?&])_=[^&]*/, + rheaders = /^(.*?):[ \t]*([^\r\n]*)$/mg, + + // #7653, #8125, #8152: local protocol detection + rlocalProtocol = /^(?:about|app|app-storage|.+-extension|file|res|widget):$/, + rnoContent = /^(?:GET|HEAD)$/, + rprotocol = /^\/\//, + + /* Prefilters + * 1) They are useful to introduce custom dataTypes (see ajax/jsonp.js for an example) + * 2) These are called: + * - BEFORE asking for a transport + * - AFTER param serialization (s.data is a string if s.processData is true) + * 3) key is the dataType + * 4) the catchall symbol "*" can be used + * 5) execution will start with transport dataType and THEN continue down to "*" if needed + */ + prefilters = {}, + + /* Transports bindings + * 1) key is the dataType + * 2) the catchall symbol "*" can be used + * 3) selection will start with transport dataType and THEN go to "*" if needed + */ + transports = {}, + + // Avoid comment-prolog char sequence (#10098); must appease lint and evade compression + allTypes = "*/".concat( "*" ), + + // Anchor tag for parsing the document origin + originAnchor = document.createElement( "a" ); + +originAnchor.href = location.href; + +// Base "constructor" for jQuery.ajaxPrefilter and jQuery.ajaxTransport +function addToPrefiltersOrTransports( structure ) { + + // dataTypeExpression is optional and defaults to "*" + return function( dataTypeExpression, func ) { + + if ( typeof dataTypeExpression !== "string" ) { + func = dataTypeExpression; + dataTypeExpression = "*"; + } + + var dataType, + i = 0, + dataTypes = dataTypeExpression.toLowerCase().match( rnothtmlwhite ) || []; + + if ( isFunction( func ) ) { + + // For each dataType in the dataTypeExpression + while ( ( dataType = dataTypes[ i++ ] ) ) { + + // Prepend if requested + if ( dataType[ 0 ] === "+" ) { + dataType = dataType.slice( 1 ) || "*"; + ( structure[ dataType ] = structure[ dataType ] || [] ).unshift( func ); + + // Otherwise append + } else { + ( structure[ dataType ] = structure[ dataType ] || [] ).push( func ); + } + } + } + }; +} + +// Base inspection function for prefilters and transports +function inspectPrefiltersOrTransports( structure, options, originalOptions, jqXHR ) { + + var inspected = {}, + seekingTransport = ( structure === transports ); + + function inspect( dataType ) { + var selected; + inspected[ dataType ] = true; + jQuery.each( structure[ dataType ] || [], function( _, prefilterOrFactory ) { + var dataTypeOrTransport = prefilterOrFactory( options, originalOptions, jqXHR ); + if ( typeof dataTypeOrTransport === "string" && + !seekingTransport && !inspected[ dataTypeOrTransport ] ) { + + options.dataTypes.unshift( dataTypeOrTransport ); + inspect( dataTypeOrTransport ); + return false; + } else if ( seekingTransport ) { + return !( selected = dataTypeOrTransport ); + } + } ); + return selected; + } + + return inspect( options.dataTypes[ 0 ] ) || !inspected[ "*" ] && inspect( "*" ); +} + +// A special extend for ajax options +// that takes "flat" options (not to be deep extended) +// Fixes #9887 +function ajaxExtend( target, src ) { + var key, deep, + flatOptions = jQuery.ajaxSettings.flatOptions || {}; + + for ( key in src ) { + if ( src[ key ] !== undefined ) { + ( flatOptions[ key ] ? target : ( deep || ( deep = {} ) ) )[ key ] = src[ key ]; + } + } + if ( deep ) { + jQuery.extend( true, target, deep ); + } + + return target; +} + +/* Handles responses to an ajax request: + * - finds the right dataType (mediates between content-type and expected dataType) + * - returns the corresponding response + */ +function ajaxHandleResponses( s, jqXHR, responses ) { + + var ct, type, finalDataType, firstDataType, + contents = s.contents, + dataTypes = s.dataTypes; + + // Remove auto dataType and get content-type in the process + while ( dataTypes[ 0 ] === "*" ) { + dataTypes.shift(); + if ( ct === undefined ) { + ct = s.mimeType || jqXHR.getResponseHeader( "Content-Type" ); + } + } + + // Check if we're dealing with a known content-type + if ( ct ) { + for ( type in contents ) { + if ( contents[ type ] && contents[ type ].test( ct ) ) { + dataTypes.unshift( type ); + break; + } + } + } + + // Check to see if we have a response for the expected dataType + if ( dataTypes[ 0 ] in responses ) { + finalDataType = dataTypes[ 0 ]; + } else { + + // Try convertible dataTypes + for ( type in responses ) { + if ( !dataTypes[ 0 ] || s.converters[ type + " " + dataTypes[ 0 ] ] ) { + finalDataType = type; + break; + } + if ( !firstDataType ) { + firstDataType = type; + } + } + + // Or just use first one + finalDataType = finalDataType || firstDataType; + } + + // If we found a dataType + // We add the dataType to the list if needed + // and return the corresponding response + if ( finalDataType ) { + if ( finalDataType !== dataTypes[ 0 ] ) { + dataTypes.unshift( finalDataType ); + } + return responses[ finalDataType ]; + } +} + +/* Chain conversions given the request and the original response + * Also sets the responseXXX fields on the jqXHR instance + */ +function ajaxConvert( s, response, jqXHR, isSuccess ) { + var conv2, current, conv, tmp, prev, + converters = {}, + + // Work with a copy of dataTypes in case we need to modify it for conversion + dataTypes = s.dataTypes.slice(); + + // Create converters map with lowercased keys + if ( dataTypes[ 1 ] ) { + for ( conv in s.converters ) { + converters[ conv.toLowerCase() ] = s.converters[ conv ]; + } + } + + current = dataTypes.shift(); + + // Convert to each sequential dataType + while ( current ) { + + if ( s.responseFields[ current ] ) { + jqXHR[ s.responseFields[ current ] ] = response; + } + + // Apply the dataFilter if provided + if ( !prev && isSuccess && s.dataFilter ) { + response = s.dataFilter( response, s.dataType ); + } + + prev = current; + current = dataTypes.shift(); + + if ( current ) { + + // There's only work to do if current dataType is non-auto + if ( current === "*" ) { + + current = prev; + + // Convert response if prev dataType is non-auto and differs from current + } else if ( prev !== "*" && prev !== current ) { + + // Seek a direct converter + conv = converters[ prev + " " + current ] || converters[ "* " + current ]; + + // If none found, seek a pair + if ( !conv ) { + for ( conv2 in converters ) { + + // If conv2 outputs current + tmp = conv2.split( " " ); + if ( tmp[ 1 ] === current ) { + + // If prev can be converted to accepted input + conv = converters[ prev + " " + tmp[ 0 ] ] || + converters[ "* " + tmp[ 0 ] ]; + if ( conv ) { + + // Condense equivalence converters + if ( conv === true ) { + conv = converters[ conv2 ]; + + // Otherwise, insert the intermediate dataType + } else if ( converters[ conv2 ] !== true ) { + current = tmp[ 0 ]; + dataTypes.unshift( tmp[ 1 ] ); + } + break; + } + } + } + } + + // Apply converter (if not an equivalence) + if ( conv !== true ) { + + // Unless errors are allowed to bubble, catch and return them + if ( conv && s.throws ) { + response = conv( response ); + } else { + try { + response = conv( response ); + } catch ( e ) { + return { + state: "parsererror", + error: conv ? e : "No conversion from " + prev + " to " + current + }; + } + } + } + } + } + } + + return { state: "success", data: response }; +} + +jQuery.extend( { + + // Counter for holding the number of active queries + active: 0, + + // Last-Modified header cache for next request + lastModified: {}, + etag: {}, + + ajaxSettings: { + url: location.href, + type: "GET", + isLocal: rlocalProtocol.test( location.protocol ), + global: true, + processData: true, + async: true, + contentType: "application/x-www-form-urlencoded; charset=UTF-8", + + /* + timeout: 0, + data: null, + dataType: null, + username: null, + password: null, + cache: null, + throws: false, + traditional: false, + headers: {}, + */ + + accepts: { + "*": allTypes, + text: "text/plain", + html: "text/html", + xml: "application/xml, text/xml", + json: "application/json, text/javascript" + }, + + contents: { + xml: /\bxml\b/, + html: /\bhtml/, + json: /\bjson\b/ + }, + + responseFields: { + xml: "responseXML", + text: "responseText", + json: "responseJSON" + }, + + // Data converters + // Keys separate source (or catchall "*") and destination types with a single space + converters: { + + // Convert anything to text + "* text": String, + + // Text to html (true = no transformation) + "text html": true, + + // Evaluate text as a json expression + "text json": JSON.parse, + + // Parse text as xml + "text xml": jQuery.parseXML + }, + + // For options that shouldn't be deep extended: + // you can add your own custom options here if + // and when you create one that shouldn't be + // deep extended (see ajaxExtend) + flatOptions: { + url: true, + context: true + } + }, + + // Creates a full fledged settings object into target + // with both ajaxSettings and settings fields. + // If target is omitted, writes into ajaxSettings. + ajaxSetup: function( target, settings ) { + return settings ? + + // Building a settings object + ajaxExtend( ajaxExtend( target, jQuery.ajaxSettings ), settings ) : + + // Extending ajaxSettings + ajaxExtend( jQuery.ajaxSettings, target ); + }, + + ajaxPrefilter: addToPrefiltersOrTransports( prefilters ), + ajaxTransport: addToPrefiltersOrTransports( transports ), + + // Main method + ajax: function( url, options ) { + + // If url is an object, simulate pre-1.5 signature + if ( typeof url === "object" ) { + options = url; + url = undefined; + } + + // Force options to be an object + options = options || {}; + + var transport, + + // URL without anti-cache param + cacheURL, + + // Response headers + responseHeadersString, + responseHeaders, + + // timeout handle + timeoutTimer, + + // Url cleanup var + urlAnchor, + + // Request state (becomes false upon send and true upon completion) + completed, + + // To know if global events are to be dispatched + fireGlobals, + + // Loop variable + i, + + // uncached part of the url + uncached, + + // Create the final options object + s = jQuery.ajaxSetup( {}, options ), + + // Callbacks context + callbackContext = s.context || s, + + // Context for global events is callbackContext if it is a DOM node or jQuery collection + globalEventContext = s.context && + ( callbackContext.nodeType || callbackContext.jquery ) ? + jQuery( callbackContext ) : + jQuery.event, + + // Deferreds + deferred = jQuery.Deferred(), + completeDeferred = jQuery.Callbacks( "once memory" ), + + // Status-dependent callbacks + statusCode = s.statusCode || {}, + + // Headers (they are sent all at once) + requestHeaders = {}, + requestHeadersNames = {}, + + // Default abort message + strAbort = "canceled", + + // Fake xhr + jqXHR = { + readyState: 0, + + // Builds headers hashtable if needed + getResponseHeader: function( key ) { + var match; + if ( completed ) { + if ( !responseHeaders ) { + responseHeaders = {}; + while ( ( match = rheaders.exec( responseHeadersString ) ) ) { + responseHeaders[ match[ 1 ].toLowerCase() + " " ] = + ( responseHeaders[ match[ 1 ].toLowerCase() + " " ] || [] ) + .concat( match[ 2 ] ); + } + } + match = responseHeaders[ key.toLowerCase() + " " ]; + } + return match == null ? null : match.join( ", " ); + }, + + // Raw string + getAllResponseHeaders: function() { + return completed ? responseHeadersString : null; + }, + + // Caches the header + setRequestHeader: function( name, value ) { + if ( completed == null ) { + name = requestHeadersNames[ name.toLowerCase() ] = + requestHeadersNames[ name.toLowerCase() ] || name; + requestHeaders[ name ] = value; + } + return this; + }, + + // Overrides response content-type header + overrideMimeType: function( type ) { + if ( completed == null ) { + s.mimeType = type; + } + return this; + }, + + // Status-dependent callbacks + statusCode: function( map ) { + var code; + if ( map ) { + if ( completed ) { + + // Execute the appropriate callbacks + jqXHR.always( map[ jqXHR.status ] ); + } else { + + // Lazy-add the new callbacks in a way that preserves old ones + for ( code in map ) { + statusCode[ code ] = [ statusCode[ code ], map[ code ] ]; + } + } + } + return this; + }, + + // Cancel the request + abort: function( statusText ) { + var finalText = statusText || strAbort; + if ( transport ) { + transport.abort( finalText ); + } + done( 0, finalText ); + return this; + } + }; + + // Attach deferreds + deferred.promise( jqXHR ); + + // Add protocol if not provided (prefilters might expect it) + // Handle falsy url in the settings object (#10093: consistency with old signature) + // We also use the url parameter if available + s.url = ( ( url || s.url || location.href ) + "" ) + .replace( rprotocol, location.protocol + "//" ); + + // Alias method option to type as per ticket #12004 + s.type = options.method || options.type || s.method || s.type; + + // Extract dataTypes list + s.dataTypes = ( s.dataType || "*" ).toLowerCase().match( rnothtmlwhite ) || [ "" ]; + + // A cross-domain request is in order when the origin doesn't match the current origin. + if ( s.crossDomain == null ) { + urlAnchor = document.createElement( "a" ); + + // Support: IE <=8 - 11, Edge 12 - 15 + // IE throws exception on accessing the href property if url is malformed, + // e.g. http://example.com:80x/ + try { + urlAnchor.href = s.url; + + // Support: IE <=8 - 11 only + // Anchor's host property isn't correctly set when s.url is relative + urlAnchor.href = urlAnchor.href; + s.crossDomain = originAnchor.protocol + "//" + originAnchor.host !== + urlAnchor.protocol + "//" + urlAnchor.host; + } catch ( e ) { + + // If there is an error parsing the URL, assume it is crossDomain, + // it can be rejected by the transport if it is invalid + s.crossDomain = true; + } + } + + // Convert data if not already a string + if ( s.data && s.processData && typeof s.data !== "string" ) { + s.data = jQuery.param( s.data, s.traditional ); + } + + // Apply prefilters + inspectPrefiltersOrTransports( prefilters, s, options, jqXHR ); + + // If request was aborted inside a prefilter, stop there + if ( completed ) { + return jqXHR; + } + + // We can fire global events as of now if asked to + // Don't fire events if jQuery.event is undefined in an AMD-usage scenario (#15118) + fireGlobals = jQuery.event && s.global; + + // Watch for a new set of requests + if ( fireGlobals && jQuery.active++ === 0 ) { + jQuery.event.trigger( "ajaxStart" ); + } + + // Uppercase the type + s.type = s.type.toUpperCase(); + + // Determine if request has content + s.hasContent = !rnoContent.test( s.type ); + + // Save the URL in case we're toying with the If-Modified-Since + // and/or If-None-Match header later on + // Remove hash to simplify url manipulation + cacheURL = s.url.replace( rhash, "" ); + + // More options handling for requests with no content + if ( !s.hasContent ) { + + // Remember the hash so we can put it back + uncached = s.url.slice( cacheURL.length ); + + // If data is available and should be processed, append data to url + if ( s.data && ( s.processData || typeof s.data === "string" ) ) { + cacheURL += ( rquery.test( cacheURL ) ? "&" : "?" ) + s.data; + + // #9682: remove data so that it's not used in an eventual retry + delete s.data; + } + + // Add or update anti-cache param if needed + if ( s.cache === false ) { + cacheURL = cacheURL.replace( rantiCache, "$1" ); + uncached = ( rquery.test( cacheURL ) ? "&" : "?" ) + "_=" + ( nonce.guid++ ) + + uncached; + } + + // Put hash and anti-cache on the URL that will be requested (gh-1732) + s.url = cacheURL + uncached; + + // Change '%20' to '+' if this is encoded form body content (gh-2658) + } else if ( s.data && s.processData && + ( s.contentType || "" ).indexOf( "application/x-www-form-urlencoded" ) === 0 ) { + s.data = s.data.replace( r20, "+" ); + } + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + if ( jQuery.lastModified[ cacheURL ] ) { + jqXHR.setRequestHeader( "If-Modified-Since", jQuery.lastModified[ cacheURL ] ); + } + if ( jQuery.etag[ cacheURL ] ) { + jqXHR.setRequestHeader( "If-None-Match", jQuery.etag[ cacheURL ] ); + } + } + + // Set the correct header, if data is being sent + if ( s.data && s.hasContent && s.contentType !== false || options.contentType ) { + jqXHR.setRequestHeader( "Content-Type", s.contentType ); + } + + // Set the Accepts header for the server, depending on the dataType + jqXHR.setRequestHeader( + "Accept", + s.dataTypes[ 0 ] && s.accepts[ s.dataTypes[ 0 ] ] ? + s.accepts[ s.dataTypes[ 0 ] ] + + ( s.dataTypes[ 0 ] !== "*" ? ", " + allTypes + "; q=0.01" : "" ) : + s.accepts[ "*" ] + ); + + // Check for headers option + for ( i in s.headers ) { + jqXHR.setRequestHeader( i, s.headers[ i ] ); + } + + // Allow custom headers/mimetypes and early abort + if ( s.beforeSend && + ( s.beforeSend.call( callbackContext, jqXHR, s ) === false || completed ) ) { + + // Abort if not done already and return + return jqXHR.abort(); + } + + // Aborting is no longer a cancellation + strAbort = "abort"; + + // Install callbacks on deferreds + completeDeferred.add( s.complete ); + jqXHR.done( s.success ); + jqXHR.fail( s.error ); + + // Get transport + transport = inspectPrefiltersOrTransports( transports, s, options, jqXHR ); + + // If no transport, we auto-abort + if ( !transport ) { + done( -1, "No Transport" ); + } else { + jqXHR.readyState = 1; + + // Send global event + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxSend", [ jqXHR, s ] ); + } + + // If request was aborted inside ajaxSend, stop there + if ( completed ) { + return jqXHR; + } + + // Timeout + if ( s.async && s.timeout > 0 ) { + timeoutTimer = window.setTimeout( function() { + jqXHR.abort( "timeout" ); + }, s.timeout ); + } + + try { + completed = false; + transport.send( requestHeaders, done ); + } catch ( e ) { + + // Rethrow post-completion exceptions + if ( completed ) { + throw e; + } + + // Propagate others as results + done( -1, e ); + } + } + + // Callback for when everything is done + function done( status, nativeStatusText, responses, headers ) { + var isSuccess, success, error, response, modified, + statusText = nativeStatusText; + + // Ignore repeat invocations + if ( completed ) { + return; + } + + completed = true; + + // Clear timeout if it exists + if ( timeoutTimer ) { + window.clearTimeout( timeoutTimer ); + } + + // Dereference transport for early garbage collection + // (no matter how long the jqXHR object will be used) + transport = undefined; + + // Cache response headers + responseHeadersString = headers || ""; + + // Set readyState + jqXHR.readyState = status > 0 ? 4 : 0; + + // Determine if successful + isSuccess = status >= 200 && status < 300 || status === 304; + + // Get response data + if ( responses ) { + response = ajaxHandleResponses( s, jqXHR, responses ); + } + + // Use a noop converter for missing script but not if jsonp + if ( !isSuccess && + jQuery.inArray( "script", s.dataTypes ) > -1 && + jQuery.inArray( "json", s.dataTypes ) < 0 ) { + s.converters[ "text script" ] = function() {}; + } + + // Convert no matter what (that way responseXXX fields are always set) + response = ajaxConvert( s, response, jqXHR, isSuccess ); + + // If successful, handle type chaining + if ( isSuccess ) { + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + modified = jqXHR.getResponseHeader( "Last-Modified" ); + if ( modified ) { + jQuery.lastModified[ cacheURL ] = modified; + } + modified = jqXHR.getResponseHeader( "etag" ); + if ( modified ) { + jQuery.etag[ cacheURL ] = modified; + } + } + + // if no content + if ( status === 204 || s.type === "HEAD" ) { + statusText = "nocontent"; + + // if not modified + } else if ( status === 304 ) { + statusText = "notmodified"; + + // If we have data, let's convert it + } else { + statusText = response.state; + success = response.data; + error = response.error; + isSuccess = !error; + } + } else { + + // Extract error from statusText and normalize for non-aborts + error = statusText; + if ( status || !statusText ) { + statusText = "error"; + if ( status < 0 ) { + status = 0; + } + } + } + + // Set data for the fake xhr object + jqXHR.status = status; + jqXHR.statusText = ( nativeStatusText || statusText ) + ""; + + // Success/Error + if ( isSuccess ) { + deferred.resolveWith( callbackContext, [ success, statusText, jqXHR ] ); + } else { + deferred.rejectWith( callbackContext, [ jqXHR, statusText, error ] ); + } + + // Status-dependent callbacks + jqXHR.statusCode( statusCode ); + statusCode = undefined; + + if ( fireGlobals ) { + globalEventContext.trigger( isSuccess ? "ajaxSuccess" : "ajaxError", + [ jqXHR, s, isSuccess ? success : error ] ); + } + + // Complete + completeDeferred.fireWith( callbackContext, [ jqXHR, statusText ] ); + + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxComplete", [ jqXHR, s ] ); + + // Handle the global AJAX counter + if ( !( --jQuery.active ) ) { + jQuery.event.trigger( "ajaxStop" ); + } + } + } + + return jqXHR; + }, + + getJSON: function( url, data, callback ) { + return jQuery.get( url, data, callback, "json" ); + }, + + getScript: function( url, callback ) { + return jQuery.get( url, undefined, callback, "script" ); + } +} ); + +jQuery.each( [ "get", "post" ], function( _i, method ) { + jQuery[ method ] = function( url, data, callback, type ) { + + // Shift arguments if data argument was omitted + if ( isFunction( data ) ) { + type = type || callback; + callback = data; + data = undefined; + } + + // The url can be an options object (which then must have .url) + return jQuery.ajax( jQuery.extend( { + url: url, + type: method, + dataType: type, + data: data, + success: callback + }, jQuery.isPlainObject( url ) && url ) ); + }; +} ); + +jQuery.ajaxPrefilter( function( s ) { + var i; + for ( i in s.headers ) { + if ( i.toLowerCase() === "content-type" ) { + s.contentType = s.headers[ i ] || ""; + } + } +} ); + + +jQuery._evalUrl = function( url, options, doc ) { + return jQuery.ajax( { + url: url, + + // Make this explicit, since user can override this through ajaxSetup (#11264) + type: "GET", + dataType: "script", + cache: true, + async: false, + global: false, + + // Only evaluate the response if it is successful (gh-4126) + // dataFilter is not invoked for failure responses, so using it instead + // of the default converter is kludgy but it works. + converters: { + "text script": function() {} + }, + dataFilter: function( response ) { + jQuery.globalEval( response, options, doc ); + } + } ); +}; + + +jQuery.fn.extend( { + wrapAll: function( html ) { + var wrap; + + if ( this[ 0 ] ) { + if ( isFunction( html ) ) { + html = html.call( this[ 0 ] ); + } + + // The elements to wrap the target around + wrap = jQuery( html, this[ 0 ].ownerDocument ).eq( 0 ).clone( true ); + + if ( this[ 0 ].parentNode ) { + wrap.insertBefore( this[ 0 ] ); + } + + wrap.map( function() { + var elem = this; + + while ( elem.firstElementChild ) { + elem = elem.firstElementChild; + } + + return elem; + } ).append( this ); + } + + return this; + }, + + wrapInner: function( html ) { + if ( isFunction( html ) ) { + return this.each( function( i ) { + jQuery( this ).wrapInner( html.call( this, i ) ); + } ); + } + + return this.each( function() { + var self = jQuery( this ), + contents = self.contents(); + + if ( contents.length ) { + contents.wrapAll( html ); + + } else { + self.append( html ); + } + } ); + }, + + wrap: function( html ) { + var htmlIsFunction = isFunction( html ); + + return this.each( function( i ) { + jQuery( this ).wrapAll( htmlIsFunction ? html.call( this, i ) : html ); + } ); + }, + + unwrap: function( selector ) { + this.parent( selector ).not( "body" ).each( function() { + jQuery( this ).replaceWith( this.childNodes ); + } ); + return this; + } +} ); + + +jQuery.expr.pseudos.hidden = function( elem ) { + return !jQuery.expr.pseudos.visible( elem ); +}; +jQuery.expr.pseudos.visible = function( elem ) { + return !!( elem.offsetWidth || elem.offsetHeight || elem.getClientRects().length ); +}; + + + + +jQuery.ajaxSettings.xhr = function() { + try { + return new window.XMLHttpRequest(); + } catch ( e ) {} +}; + +var xhrSuccessStatus = { + + // File protocol always yields status code 0, assume 200 + 0: 200, + + // Support: IE <=9 only + // #1450: sometimes IE returns 1223 when it should be 204 + 1223: 204 + }, + xhrSupported = jQuery.ajaxSettings.xhr(); + +support.cors = !!xhrSupported && ( "withCredentials" in xhrSupported ); +support.ajax = xhrSupported = !!xhrSupported; + +jQuery.ajaxTransport( function( options ) { + var callback, errorCallback; + + // Cross domain only allowed if supported through XMLHttpRequest + if ( support.cors || xhrSupported && !options.crossDomain ) { + return { + send: function( headers, complete ) { + var i, + xhr = options.xhr(); + + xhr.open( + options.type, + options.url, + options.async, + options.username, + options.password + ); + + // Apply custom fields if provided + if ( options.xhrFields ) { + for ( i in options.xhrFields ) { + xhr[ i ] = options.xhrFields[ i ]; + } + } + + // Override mime type if needed + if ( options.mimeType && xhr.overrideMimeType ) { + xhr.overrideMimeType( options.mimeType ); + } + + // X-Requested-With header + // For cross-domain requests, seeing as conditions for a preflight are + // akin to a jigsaw puzzle, we simply never set it to be sure. + // (it can always be set on a per-request basis or even using ajaxSetup) + // For same-domain requests, won't change header if already provided. + if ( !options.crossDomain && !headers[ "X-Requested-With" ] ) { + headers[ "X-Requested-With" ] = "XMLHttpRequest"; + } + + // Set headers + for ( i in headers ) { + xhr.setRequestHeader( i, headers[ i ] ); + } + + // Callback + callback = function( type ) { + return function() { + if ( callback ) { + callback = errorCallback = xhr.onload = + xhr.onerror = xhr.onabort = xhr.ontimeout = + xhr.onreadystatechange = null; + + if ( type === "abort" ) { + xhr.abort(); + } else if ( type === "error" ) { + + // Support: IE <=9 only + // On a manual native abort, IE9 throws + // errors on any property access that is not readyState + if ( typeof xhr.status !== "number" ) { + complete( 0, "error" ); + } else { + complete( + + // File: protocol always yields status 0; see #8605, #14207 + xhr.status, + xhr.statusText + ); + } + } else { + complete( + xhrSuccessStatus[ xhr.status ] || xhr.status, + xhr.statusText, + + // Support: IE <=9 only + // IE9 has no XHR2 but throws on binary (trac-11426) + // For XHR2 non-text, let the caller handle it (gh-2498) + ( xhr.responseType || "text" ) !== "text" || + typeof xhr.responseText !== "string" ? + { binary: xhr.response } : + { text: xhr.responseText }, + xhr.getAllResponseHeaders() + ); + } + } + }; + }; + + // Listen to events + xhr.onload = callback(); + errorCallback = xhr.onerror = xhr.ontimeout = callback( "error" ); + + // Support: IE 9 only + // Use onreadystatechange to replace onabort + // to handle uncaught aborts + if ( xhr.onabort !== undefined ) { + xhr.onabort = errorCallback; + } else { + xhr.onreadystatechange = function() { + + // Check readyState before timeout as it changes + if ( xhr.readyState === 4 ) { + + // Allow onerror to be called first, + // but that will not handle a native abort + // Also, save errorCallback to a variable + // as xhr.onerror cannot be accessed + window.setTimeout( function() { + if ( callback ) { + errorCallback(); + } + } ); + } + }; + } + + // Create the abort callback + callback = callback( "abort" ); + + try { + + // Do send the request (this may raise an exception) + xhr.send( options.hasContent && options.data || null ); + } catch ( e ) { + + // #14683: Only rethrow if this hasn't been notified as an error yet + if ( callback ) { + throw e; + } + } + }, + + abort: function() { + if ( callback ) { + callback(); + } + } + }; + } +} ); + + + + +// Prevent auto-execution of scripts when no explicit dataType was provided (See gh-2432) +jQuery.ajaxPrefilter( function( s ) { + if ( s.crossDomain ) { + s.contents.script = false; + } +} ); + +// Install script dataType +jQuery.ajaxSetup( { + accepts: { + script: "text/javascript, application/javascript, " + + "application/ecmascript, application/x-ecmascript" + }, + contents: { + script: /\b(?:java|ecma)script\b/ + }, + converters: { + "text script": function( text ) { + jQuery.globalEval( text ); + return text; + } + } +} ); + +// Handle cache's special case and crossDomain +jQuery.ajaxPrefilter( "script", function( s ) { + if ( s.cache === undefined ) { + s.cache = false; + } + if ( s.crossDomain ) { + s.type = "GET"; + } +} ); + +// Bind script tag hack transport +jQuery.ajaxTransport( "script", function( s ) { + + // This transport only deals with cross domain or forced-by-attrs requests + if ( s.crossDomain || s.scriptAttrs ) { + var script, callback; + return { + send: function( _, complete ) { + script = jQuery( " + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel DLB device plugin for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

This Intel DLB device plugin provides support for Intel DLB devices under Kubernetes.

+
+

DLB2 driver configuration for PFs

+

The DLB device plugin requires a Linux Kernel DLB driver to be installed and enabled to operate. Get DLB software release, build and load the dlb2 driver module following the instruction of ‘DLB_Driver_User_Guide.pdf’ in the directory ‘dlb/docs’.

+

After successfully loading the module, available dlb device nodes are visible in devfs.

+
$ ls -1 /dev/dlb*
+/dev/dlb0  /dev/dlb1  /dev/dlb2 ...
+
+
+
+
+

VF configuration using a DPDK tool (but with dlb2 driver)

+

If you configure SR-IOV/VF (virtual functions), continue the following configurations. This instruction uses DPDK tool to check eventdev devices, unbind a VF device, and bind dlb2 driver to a VF device.

+

Patch dpdk sources to work with DLB:

+
$ wget -q https://fast.dpdk.org/rel/dpdk-21.11.tar.xz -O- | tar -Jx
+$ wget -q https://downloadmirror.intel.com/734482/dlb_linux_src_release_7.7.0_2022_06_17.txz -O- | tar -Jx
+$ cd ./dpdk-*/ && patch -p1 < ../dlb/dpdk/dpdk_dlb_*_diff.patch
+$ sed -i 's/270b,2710,2714/270b,2710,2711,2714/g' ./usertools/dpdk-devbind.py
+
+
+

List eventdev devices:

+
$ ./usertools/dpdk-devbind.py -s | grep -A10 ^Eventdev
+Eventdev devices using kernel driver
+====================================
+0000:6d:00.0 'Device 2710' drv=dlb2 unused=
+0000:72:00.0 'Device 2710' drv=dlb2 unused=
+...
+
+
+

Enable virtual functions:

+
$ echo 4 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/sriov_numvfs
+
+
+
+

Note:: If it fails saying “No such file or directory,” it may be bound to vfio-pci driver. Bind the device to dlb2 driver.

+
+

Check if new dlb device nodes appear:

+
$ ls -1 /dev/dlb*
+/dev/dlb0  /dev/dlb1  /dev/dlb10 /dev/dlb11 ... /dev/dlb8  /dev/dlb9
+
+
+

Check that new eventdev devices appear:

+
$ ./usertools/dpdk-devbind.py -s | grep -A14 ^Eventdev
+Eventdev devices using kernel driver
+====================================
+0000:6d:00.0 'Device 2710' drv=dlb2 unused=
+0000:6d:00.1 'Device 2711' drv=dlb2 unused=
+0000:6d:00.2 'Device 2711' drv=dlb2 unused=
+0000:6d:00.3 'Device 2711' drv=dlb2 unused=
+0000:6d:00.4 'Device 2711' drv=dlb2 unused=
+0000:72:00.0 'Device 2710' drv=dlb2 unused=
+...
+
+
+

Assign PF resources to VF:

+
+

Note:: The process below is only for the first vf resource among 4 resources. Repeat for other vfN_resources in /sys/bus/pci/devices/0000:6d:00.0/, and then bind dlb2 driver to 0000:6d:00.M that corresponds to vfN_resources.

+
+
    +

  • Unbind driver from the VF device before configuring it.

    • +
+
$ sudo ./usertools/dpdk-devbind.py --unbind 0000:6d:00.1
+
+
+
    +

  • Assign PF resources to VF:

    • +
+
$ echo 2048 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_atomic_inflights &&
+  echo 2048 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_dir_credits &&
+  echo 64 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_dir_ports &&
+  echo 2048 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_hist_list_entries &&
+  echo 8192 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_ldb_credits &&
+  echo 64 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_ldb_ports &&
+  echo 32 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_ldb_queues &&
+  echo 32 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_sched_domains &&
+  echo 2 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_sn0_slots &&
+  echo 2 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_sn1_slots
+
+
+
    +

  • Bind driver back to the VF device:

    • +
+
$ sudo ./usertools/dpdk-devbind.py --bind dlb2 0000:6d:00.1
+
+
+
+
+

Verification of well-configured devices:

+

Run libdlb example app:

+
+

Note:: Alternative way is to use this Dockerfile for running tests.

+
+
$ ls
+dlb dpdk-21.11
+$ cd ./dlb/libdlb/ && make && sudo LD_LIBRARY_PATH=$PWD ./examples/dir_traffic -n 128 -d 1
+# For running test for /dev/dlbN, replace 1 with N.
+
+
+

Run dpdk example app:

+
+

Note:: Alternative way is to use this Dockerfile for patching and building DPDK and running tests.

+
+
    +

  • Install build dependencies and build dpdk:

    • +
+
$ sudo apt-get update && sudo apt-get install build-essential meson python3-pyelftools libnuma-dev python3-pip && sudo pip install ninja
+# This configuration is based on Ubuntu/Debian distribution. For other distributions that do not use apt, install the dependencies using another way.
+$ ls
+dlb dpdk-21.11
+$ cd ./dpdk-* && meson setup --prefix $(pwd)/installdir builddir && ninja -C builddir install
+
+
+
    +

  • Run eventdev test

    • +
+
sudo ./builddir/app/dpdk-test-eventdev --no-huge --vdev='dlb2_event,dev_id=1' -- --test=order_queue --nb_flows 64 --nb_pkts 512 --plcores 1 --wlcores 2-7
+# For running test for /dev/dlbN, replace 1 with N.
+
+
+
+
+
+

Installation

+

The following sections detail how to obtain, build, deploy and test the DLB device plugin.

+

Examples are provided showing how to deploy the plugin either using a DaemonSet or by hand on a per-node basis.

+
+

Pre-built Images

+

Pre-built images +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository.

+

Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format x.y.z, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/dlb_plugin?ref=<RELEASE_VERSION>'
+daemonset.apps/intel-dlb-plugin created
+
+
+

Where <RELEASE_VERSION> needs to be substituted with the desired release tag or main to get devel images.

+

Nothing else is needed. See the development guide for details if you want to deploy a customized version of the plugin.

+
+
+

Verify Plugin Registration

+

You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes:

+
$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{"  "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\(  dlb\)'
+master
+  dlb.intel.com/pf: 7
+  dlb.intel.com/vf: 4
+
+
+
+
+
+

Testing and Demos

+

We can test the plugin is working by deploying the provided example test images (dlb-libdlb-demo and dlb-dpdk-demo).

+
    +

  1. Build a Docker image and create a pod running unit tests off the local Docker image:

    +
    $ make dlb-libdlb-demo
+...
+Successfully tagged intel/dlb-libdlb-demo:devel
+
+$ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/dlb-libdlb-demo-pod.yaml
+pod/dlb-libdlb-demo-pod created
+
    +
    +
    $ make dlb-dpdk-demo
+...
+Successfully tagged intel/dlb-dpdk-demo:devel
+
+$ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/dlb-dpdk-demo-pod.yaml
+pod/dlb-dpdk-demo-pod created
+
    +
    +
    2. +

  2. Wait until pod is completed:

    +
    $ kubectl get pods | grep dlb-.*-demo
+NAME              READY   STATUS      RESTARTS   AGE
+dlb-dpdk-demo     0/2     Completed   0          79m
+dlb-libdlb-demo   0/2     Completed   0          18h
+
    +
    +
    3. +

  3. Review the job’s logs:

    +
    $ kubectl logs dlb-libdlb-demo <pf/vf>
+<log output>
+
    +
    +
    $ kubectl logs dlb-dpdk-demo <pf/vf>
+<log output>
+
    +
    +

    If the pod did not successfully launch, possibly because it could not obtain the DLB +resource, it will be stuck in the Pending status:

    +
    $ kubectl get pods
+NAME              READY   STATUS      RESTARTS   AGE
+dlb-dpdk-demo     0/2     Pending     0          3s
+dlb-libdlb-demo   0/2     Pending     0          10s
+
    +
    +

    This can be verified by checking the Events of the pod:

    +
    $ kubectl describe pod dlb-libdlb-demo | grep -A3 Events:
+Events:
+Type     Reason            Age   From               Message
+----     ------            ----  ----               -------
+Warning  FailedScheduling  85s   default-scheduler  0/1 nodes are available: 1 Insufficient dlb.intel.com/pf, 1 Insufficient dlb.intel.com/vf.
+
    +
    +
    4. +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/dsa_plugin/README.html b/0.26/cmd/dsa_plugin/README.html new file mode 100644 index 000000000..2d5f76649 --- /dev/null +++ b/0.26/cmd/dsa_plugin/README.html @@ -0,0 +1,258 @@ + + + + + + + Intel DSA device plugin for Kubernetes — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel DSA device plugin for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

The DSA device plugin for Kubernetes supports acceleration using the Intel Data Streaming accelerator(DSA).

+

The DSA plugin discovers DSA work queues and presents them as a node resources.

+

The DSA plugin and operator optionally support provisioning of DSA devices and workqueues with the help of accel-config utility through initcontainer.

+
+
+

Installation

+

The following sections detail how to use the DSA device plugin.

+
+

Pre-built Images

+

Pre-built images +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository.

+

Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format x.y.z, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/dsa_plugin?ref=<RELEASE_VERSION>'
+daemonset.apps/intel-dsa-plugin created
+
+
+

Where <RELEASE_VERSION> needs to be substituted with the desired release tag or main to get devel images.

+

Nothing else is needed. See the development guide for details if you want to deploy a customized version of the plugin.

+
+

Automatic Provisioning

+

There’s a sample idxd initcontainer included that provisions DSA devices and workqueues (1 engine / 1 group / 1 wq (user/dedicated)), to deploy:

+
$ kubectl apply -k deployments/dsa_plugin/overlays/dsa_initcontainer/
+
+
+

The provisioning script and template are available for customization.

+

The provisioning config can be optionally stored in the ProvisioningConfig configMap which is then passed to initcontainer through the volume mount.

+

There’s also a possibility for a node specific congfiguration through passing a nodename via NODE_NAME into initcontainer’s environment and passing a node specific profile via configMap volume mount.

+

To create a custom provisioning config:

+
$ kubectl create configmap --namespace=inteldeviceplugins-system intel-dsa-config --from-file=demo/dsa.conf
+
+
+
+
+
+

Verify Plugin Registration

+

You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes:

+
$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{"  "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\(  dsa\)'
+master
+  dsa.intel.com/wq-user-dedicated: 2
+  dsa.intel.com/wq-user-shared: 8
+node1
+ dsa.intel.com/wq-user-dedicated: 4
+ dsa.intel.com/wq-user-shared: 20
+
+
+
+
+
+

Testing and Demos

+

We can test the plugin is working by deploying the provided example accel-config test image.

+
    +

  1. Build a Docker image with an accel-config tests:

    +
    $ make accel-config-demo
+...
+Successfully tagged accel-config-demo:devel
+
    +
    +
    2. +

  2. Create a pod running unit tests off the local Docker image:

    +
    $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/dsa-accel-config-demo-pod.yaml
+pod/dsa-accel-config-demo created
+
    +
    +
    3. +

  3. Wait until pod is completed:

    +
    $ kubectl get pods  |grep dsa-accel-config-demo
+dsa-accel-config-demo    0/1     Completed   0          31m
+
    +
    +
    4. +

  4. Review the job’s logs:

    +
    $ kubectl logs dsa-accel-config-demo | tail
+[debug] PF in sub-task[6], consider as passed
+[debug] PF in sub-task[7], consider as passed
+[debug] PF in sub-task[8], consider as passed
+[debug] PF in sub-task[9], consider as passed
+[debug] PF in sub-task[10], consider as passed
+[debug] PF in sub-task[11], consider as passed
+[debug] PF in sub-task[12], consider as passed
+[debug] PF in sub-task[13], consider as passed
+[debug] PF in sub-task[14], consider as passed
+[debug] PF in sub-task[15], consider as passed
+
    +
    +

    If the pod did not successfully launch, possibly because it could not obtain the DSA +resource, it will be stuck in the Pending status:

    +
    $ kubectl get pods
+NAME                      READY   STATUS    RESTARTS   AGE
+dsa-accel-config-demo     0/1     Pending   0          7s
+
    +
    +

    This can be verified by checking the Events of the pod:

    +
    $ kubectl describe pod dsa-accel-config-demo | grep -A3 Events:
+Events:
+  Type     Reason            Age    From               Message
+  ----     ------            ----   ----               -------
+  Warning  FailedScheduling  2m26s  default-scheduler  0/1 nodes are available: 1 Insufficient dsa.intel.com/wq-user-dedicated, 1 Insufficient dsa.intel.com/wq-user-shared.
+
    +
    +
    5. +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/fpga_admissionwebhook/README.html b/0.26/cmd/fpga_admissionwebhook/README.html new file mode 100644 index 000000000..358267a03 --- /dev/null +++ b/0.26/cmd/fpga_admissionwebhook/README.html @@ -0,0 +1,313 @@ + + + + + + + Intel FPGA admission controller for Kubernetes — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel FPGA admission controller for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

The FPGA admission controller is one of the components used to add support for Intel FPGA +devices to Kubernetes.

+
+

NOTE: Installation of the FPGA admission controller can be skipped if the +FPGA device plugin is operated with the Intel Device Plugins Operator +since it integrates the controller’s functionality.

+
+

The FPGA admission controller webhook is responsible for performing mapping from user-friendly +function IDs to the Interface ID and Bitstream ID that are required for FPGA programming by +the FPGA CRI-O hook.

+

Mappings are stored in namespaced custom resource definition (CRD) objects, therefore the admission +controller also performs access control, determining which bitstream can be used for which namespace. +More details can be found in the Mappings section.

+

The admission controller also keeps the user from bypassing namespaced mapping restrictions, +by denying admission of any pods that are trying to use internal knowledge of InterfaceID or +Bitstream ID environment variables used by the prestart hook.

+
+
+

Dependencies

+

This component is one of a set of components that work together. You may also want to +install the following:

+ +

All components have the same basic dependencies as the +generic plugin framework dependencies

+
+
+

Installation

+

The following sections detail how to obtain, build and deploy the admission +controller webhook plugin.

+
+

Pre-requisites

+

The default webhook deployment depends on having cert-manager +installed. See its installation instructions here.

+

Also if your cluster operates behind a corporate proxy make sure that the API +server is configured not to send requests to cluster services through the +proxy. You can check that with the following command:

+
$ kubectl describe pod kube-apiserver --namespace kube-system | grep -i no_proxy | grep "\.svc"
+
+
+

In case there’s no output and your cluster was deployed with kubeadm open +/etc/kubernetes/manifests/kube-apiserver.yaml at the control plane nodes and +append .svc and .svc.cluster.local to the no_proxy environment variable:

+
apiVersion: v1
+kind: Pod
+metadata:
+  ...
+spec:
+  containers:
+  - command:
+    - kube-apiserver
+    - --advertise-address=10.237.71.99
+    ...
+    env:
+    - name: http_proxy
+      value: http://proxy.host:8080
+    - name: https_proxy
+      value: http://proxy.host:8433
+    - name: no_proxy
+      value: 127.0.0.1,localhost,.example.com,10.0.0.0/8,.svc,.svc.cluster.local
+    ...
+
+
+

Note: To build clusters using kubeadm with the right no_proxy settings from the very beginning, +set the cluster service names to $no_proxy before kubeadm init:

+
$ export no_proxy=$no_proxy,.svc,.svc.cluster.local
+
+
+
+
+
+

Mappings

+

Mappings is a an essential part of the setup that gives a flexible instrument to a cluster +administrator to manage FPGA bitstreams and to control access to them. Being a set of +custom resource definitions they are used to configure the way FPGA resource requests get +translated into actual resources provided by the cluster.

+

For the following mapping

+
apiVersion: fpga.intel.com/v2
+kind: AcceleratorFunction
+metadata:
+  name: arria10.dcp1.2-nlb0-preprogrammed
+spec:
+  afuId: d8424dc4a4a3c413f89e433683f9040b
+  interfaceId: 69528db6eb31577a8c3668f9faa081f6
+  mode: af
+
+
+

requested FPGA resources are translated to AF resources. For example, +fpga.intel.com/arria10.dcp1.2-nlb0-preprogrammed is translated to +fpga.intel.com/af-695.d84.aVKNtusxV3qMNmj5-qCB9thCTcSko8QT-J5DNoP5BAs where the af- +prefix indicates the plugin’s mode (af), 695 is the first three characters of +the region interface ID, d84 is the first three characters of the accelerator function ID +and the last part aVKNtusxV3qMNmj5-qCB9thCTcSko8QT-J5DNoP5BAs is a base64-encoded concatenation +of the full region interface ID and accelerator function ID. +The format of resource names (e.g. arria10.dcp1.2-nlb0-preprogrammed) can be any and is up +to a cluster administrator.

+

The same mapping, but with its mode field set to region, would translate +fpga.intel.com/arria10.dcp1.2-nlb0-preprogrammed to fpga.intel.com/region-69528db6eb31577a8c3668f9faa081f6, +and the corresponding AF IDs are set in environment variables for the container. +Though in this case the cluster administrator would probably want to rename +the mapping arria10.dcp1.2-nlb0-preprogrammed to something like arria10.dcp1.2-nlb0-orchestrated +to reflect its mode. The FPGA CRI-O hook then loads the requested +bitstream to a region before the container is started.

+

Mappings of resource names are configured with objects of AcceleratorFunction and +FpgaRegion custom resource definitions found respectively in +./deployment/fpga_admissionwebhook/crd/bases/fpga.intel.com_af.yaml +and ./deployment/fpga_admissionwebhook/crd/bases/fpga.intel.com_region.yaml.

+

Example mappings between ‘names’ and ‘ID’s are controlled by the admission controller mappings collection file found in +./deployments/fpga_admissionwebhook/mappings-collection.yaml.

+
+

Deployment

+
+

Webhook deployment

+

To deploy the webhook, run

+
$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/fpga_admissionwebhook/default?ref=main
+namespace/intelfpgawebhook-system created
+customresourcedefinition.apiextensions.k8s.io/acceleratorfunctions.fpga.intel.com created
+customresourcedefinition.apiextensions.k8s.io/fpgaregions.fpga.intel.com created
+mutatingwebhookconfiguration.admissionregistration.k8s.io/intelfpgawebhook-mutating-webhook-configuration created
+clusterrole.rbac.authorization.k8s.io/intelfpgawebhook-manager-role created
+clusterrolebinding.rbac.authorization.k8s.io/intelfpgawebhook-manager-rolebinding created
+service/intelfpgawebhook-webhook-service created
+deployment.apps/intelfpgawebhook-webhook created
+certificate.cert-manager.io/intelfpgawebhook-serving-cert created
+issuer.cert-manager.io/intelfpgawebhook-selfsigned-issuer created
+
+
+
+
+

Mappings deployment

+

Mappings deployment is a mandatory part of the webhook deployment. You should +prepare and deploy mappings that describe FPGA bitstreams available in your cluster.

+

Example mappings collection ./deployments/fpga_admissionwebhook/mappings-collection.yaml +can be used as an example for cluster mappings. This collection is not intended to be deployed as is, +it should be used as a reference and example of your own cluster mappings.

+

To deploy the mappings, run

+
$ kubectl apply -f <path to mappings yaml>
+
+
+

Note that the mappings are scoped to the namespaces they were created in +and they are applicable to pods created in the corresponding namespaces.

+
+
+
+
+

Next steps

+

Continue with FPGA prestart CRI-O hook.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/fpga_crihook/README.html b/0.26/cmd/fpga_crihook/README.html new file mode 100644 index 000000000..2f7b4da42 --- /dev/null +++ b/0.26/cmd/fpga_crihook/README.html @@ -0,0 +1,181 @@ + + + + + + + Intel FPGA prestart CRI-O webhook for Kubernetes — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel FPGA prestart CRI-O webhook for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

The FPGA CRI-O webhook is one of the components used to add support for Intel FPGA +devices to Kubernetes.

+

The FPGA prestart CRI-O hook is triggered by container annotations, such as set by the +FPGA device plugin. It performs discovery of the requested FPGA +function bitstream and then programs FPGA devices based on the environment variables +in the workload description.

+

The CRI-O prestart hook is only required when the +FPGA admission webhook is configured for orchestration +programmed mode, and is benign (un-used) otherwise.

+
+

Note: The fpga CRI-O webhook is usually installed by the same DaemonSet as the +FPGA device plugin. If building and installing the CRI-O webhook by hand, it is +recommended you reference the +fpga plugin DaemonSet YAML for +more details.

+
+
+
+

Dependencies

+

This component is one of a set of components that work together. You may also want to +install the following:

+ +

All components have the same basic dependencies as the +generic plugin framework dependencies

+

See the development guide for details if you want to deploy a customized version of the CRI hook.

+
+
+

Configuring CRI-O

+

Recent versions of CRI-O are shipped with default configuration +file that prevents CRI-O to discover and configure hooks automatically. +For FPGA orchestration programmed mode, the OCI hooks are the key component. +Please ensure that your /etc/crio/crio.conf parameter hooks_dir is either unset +(to enable default search paths for OCI hooks configuration) or contains the directory +/etc/containers/oci/hooks.d.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/fpga_plugin/README.html b/0.26/cmd/fpga_plugin/README.html new file mode 100644 index 000000000..5488772a4 --- /dev/null +++ b/0.26/cmd/fpga_plugin/README.html @@ -0,0 +1,356 @@ + + + + + + + Intel FPGA device plugin for Kubernetes — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel FPGA device plugin for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

This FPGA device plugin is part of a collection of Kubernetes components found within this +repository that enable integration of Intel FPGA hardware into Kubernetes.

+

The following hardware platforms are supported:

+
    +

  • Intel Arria 10

    • +

  • Intel Stratix 10

    • +
+

The components support the Open Programmable Acceleration Engine (OPAE) +interface.

+

The components together implement the following features:

+
    +

  • discovery of pre-programmed accelerator functions

    • +

  • discovery of programmable regions

    • +

  • orchestration of FPGA programming

    • +

  • access control for FPGA hardware

    • +
+
+

Component Overview

+

The following components are part of this repository, and work together to support Intel FPGAs under +Kubernetes:

+
    +

  • FPGA device plugin (this component)

    +

    A Kubernetes device plugin +that discovers available FPGA resources on a node and advertises them to the Kubernetes control plane +via the node kubelet.

    +
    • +

  • FPGA admission controller webhook

    +

    A Kubernetes admission controller webhook +which can be used to dynamically convert logical resource names in pod specifications into actual FPGA +resource names, as advertised by the device plugin.

    +

    The webhook can also set environment variables to instruct the CRI-O prestart hook to program the FPGA +before launching the container.

    +
    +

    NOTE: Installation of the FPGA admission controller webhook can be skipped if the +FPGA device plugin is operated with the Intel Device Plugins Operator +since it integrates the controller’s functionality. +However, the mappings still must be deployed.”

    +
    +
    • +

  • FPGA CRI-O prestart hook

    +

    A CRI-O prestart hook that, upon instruction from the FPGA admission +controller, allocates and programs the FPGA before the container is launched.

    +
    • +
+

The repository also contains an FPGA helper tool that may be useful during +development, initial deployment and debugging.

+
+
+

Modes and Configuration Options

+

The FPGA plugin set can run in one of two modes:

+
    +

  • region mode, where the plugins locate and advertise +regions of the FPGA, and facilitate programing of those regions with the +requested bistreams.

    • +

  • af mode, where the FPGA bitstreams are already loaded +onto the FPGA, and the plugins discover and advertises the existing +Accelerator Functions (AF).

    • +
+

The example YAML deployments described in this document only currently support +af mode. To utilise region mode, either modify the existing YAML appropriately, +or deploy ‘by hand’.

+

Overview diagrams of af and region modes are below:

+

region mode:

+

Overview of region mode

+

af mode:

+

Overview of af mode

+
+
+
+

Installation

+

The below sections cover how to use this component.

+
+

Prerequisites

+

All components have the same basic dependencies as the +generic plugin framework dependencies

+

To obtain a fully operational FPGA enabled cluster, you must install all three +major components:

+ +

The CRI-O hook is only required if region mode is being used, but is installed by default by the +FPGA plugin DaemonSet YAML, and is benign +in af mode.

+

If using the af mode, and therefore not using the +CRI-O prestart hook, runtimes other than CRI-O can be used (that is, the CRI-O hook presently +only works with the CRI-O runtime).

+

The FPGA device plugin requires a Linux Kernel FPGA driver to be installed and enabled to +operate. The plugin supports the use of either of following two drivers, and auto detects +which is present and thus to use:

+
    +

  • The Linux Kernel in-tree DFL driver

    • +

  • The out of tree OPAE driver

    • +
+

Install this component (FPGA device plugin) first, and then follow the links +and instructions to install the other components.

+

The FPGA webhook deployment depends on having cert-manager +installed. See its installation instructions here.

+
$ kubectl get pods -n cert-manager
+NAME                                      READY   STATUS    RESTARTS   AGE
+cert-manager-7747db9d88-bd2nl             1/1     Running   0          1m
+cert-manager-cainjector-87c85c6ff-59sb5   1/1     Running   0          1m
+cert-manager-webhook-64dc9fff44-29cfc     1/1     Running   0          1m
+
+
+
+
+

Pre-built Images

+

Pre-built images of the components are available on the Docker hub. +These images are automatically built and uploaded to the hub from the latest main branch of +this repository.

+

Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers (of the form x.y.z, matching the branch/tag release number in this repo).

+

The following images are available on the Docker hub:

+ +

Depending on the FPGA mode, run either

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/fpga_plugin/overlays/af?ref=<RELEASE_VERSION>'
+namespace/intelfpgaplugin-system created
+customresourcedefinition.apiextensions.k8s.io/acceleratorfunctions.fpga.intel.com created
+customresourcedefinition.apiextensions.k8s.io/fpgaregions.fpga.intel.com created
+mutatingwebhookconfiguration.admissionregistration.k8s.io/intelfpgaplugin-mutating-webhook-configuration created
+clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-manager-role created
+clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-node-getter created
+clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-get-nodes created
+clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-manager-rolebinding created
+service/intelfpgaplugin-webhook-service created
+deployment.apps/intelfpgaplugin-webhook created
+daemonset.apps/intelfpgaplugin-fpgadeviceplugin created
+certificate.cert-manager.io/intelfpgaplugin-serving-cert created
+issuer.cert-manager.io/intelfpgaplugin-selfsigned-issuer created
+
+
+

or

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/fpga_plugin/overlays/region?ref=<RELEASE_VERSION>'
+namespace/intelfpgaplugin-system created
+customresourcedefinition.apiextensions.k8s.io/acceleratorfunctions.fpga.intel.com created
+customresourcedefinition.apiextensions.k8s.io/fpgaregions.fpga.intel.com created
+mutatingwebhookconfiguration.admissionregistration.k8s.io/intelfpgaplugin-mutating-webhook-configuration created
+clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-manager-role created
+clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-node-getter created
+clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-get-nodes created
+clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-manager-rolebinding created
+service/intelfpgaplugin-webhook-service created
+deployment.apps/intelfpgaplugin-webhook created
+daemonset.apps/intelfpgaplugin-fpgadeviceplugin created
+certificate.cert-manager.io/intelfpgaplugin-serving-cert created
+issuer.cert-manager.io/intelfpgaplugin-selfsigned-issuer created
+
+
+

Where <RELEASE_VERSION> needs to be substituted with the desired release tag or main to get devel images.

+

The command should result in two pods running:

+
$ kubectl get pods -n intelfpgaplugin-system
+NAME                                       READY   STATUS    RESTARTS   AGE
+intelfpgaplugin-fpgadeviceplugin-skcw5     1/1     Running   0          57s
+intelfpgaplugin-webhook-7d6bcb8b57-k52b9   1/1     Running   0          57s
+
+
+

If you need the FPGA plugin on some nodes to operate in a different mode then add this +annotation to the nodes:

+
$ kubectl annotate node <node_name> 'fpga.intel.com/device-plugin-mode=region'
+
+
+

or

+
$ kubectl annotate node <node_name> 'fpga.intel.com/device-plugin-mode=af'
+
+
+

And restart the pods on the nodes.

+
+

Note: The FPGA plugin DaemonSet YAML +also deploys the FPGA CRI-O hook initcontainer image, but it will be +benign (un-used) when running the FPGA plugin in af mode.

+
+
+

Verify Plugin Registration

+

Verify the FPGA plugin has been deployed on the nodes. The below shows the output +you can expect in region mode, but similar output should be expected for af +mode:

+
$ kubectl describe nodes | grep fpga.intel.com
+fpga.intel.com/region-ce48969398f05f33946d560708be108a:  1
+fpga.intel.com/region-ce48969398f05f33946d560708be108a:  1
+
+
+
+

Note: The FPGA plugin DaemonSet YAML +also deploys the FPGA CRI-O hook initcontainer image as well. You may +also wish to build that image locally before deploying the FPGA plugin to avoid deploying +the Docker hub default image.

+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/fpga_tool/README.html b/0.26/cmd/fpga_tool/README.html new file mode 100644 index 000000000..ad569bf63 --- /dev/null +++ b/0.26/cmd/fpga_tool/README.html @@ -0,0 +1,161 @@ + + + + + + + Intel FPGA test tool — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel FPGA test tool

+
+

Introduction

+

This directory contains an FPGA test tool that can be used to locate, examine and program Intel +FPGAs.

+
+

Command line and usage

+

The tool has the following command line arguments:

+
info, fpgainfo, install, list, fmeinfo, portinfo, list-fme, list-port, pr, release, assign
+
+
+

and the following command line options:

+
Usage of ./fpga_tool:
+  -b string
+        Path to bitstream file (GBS or AOCX)
+  -d string
+        Path to device node (FME or Port)
+  -dry-run
+        Don't write/program, just validate and log
+  -force
+        Force overwrite operation for installing bitstreams
+  -q    Quiet mode. Only errors will be reported
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/gpu_fakedev/README.html b/0.26/cmd/gpu_fakedev/README.html new file mode 100644 index 000000000..bb6381d23 --- /dev/null +++ b/0.26/cmd/gpu_fakedev/README.html @@ -0,0 +1,153 @@ + + + + + + + Fake (GPU) device file generator — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Fake (GPU) device file generator

+

Table of Contents

+ +
+

Introduction

+

This is a tool for generating (large number of) fake device files for +k8s device scheduling scalability testing. But it can also be used +just to test (GPU) device plugin functionality without having +corresponding device HW.

+

Its “intel-gpu-fakedev” container is intended to be run as first init +container in a device plugin pod, so that device plugin (and its NFD +labeler) see the fake (sysfs + devfs) files generated by the tool, +instead of real host sysfs and devfs content.

+
+
+

Configuration

+

Configs subdirectory contains example JSON configuration +file(s) for the generator. Currently there’s only one example JSON +file, but each new device variant adding feature(s) that have specific +support in device plugin, could have their own fake device config.

+
+
+

Potential improvements

+

If support for mixed device environment is needed, tool can be updated +to use node / configuration file mapping. Such mappings could be e.g. +in configuration files themselves as node name include / exlude lists, +and tool would use first configuration file matching the node it’s +running on. For now, one would need to use different pod / config +specs for different nodes to achieve that…

+

Currently JSON config file options and the generated files are tied to +what GPU plugin uses, but if needed, they could be changed to fake +also sysfs + devfs device files used by other plugins.

+
+ +
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/gpu_nfdhook/README.html b/0.26/cmd/gpu_nfdhook/README.html new file mode 100644 index 000000000..fcec75d90 --- /dev/null +++ b/0.26/cmd/gpu_nfdhook/README.html @@ -0,0 +1,295 @@ + + + + + + + Intel GPU NFD hook — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel GPU NFD hook

+

Table of Contents

+ +
+

Introduction

+

This is the Node Feature Discovery +binary hook implementation for the Intel GPUs. The intel-gpu-initcontainer (which +is built with the other images) can be used as part of the gpu-plugin deployment +to copy hook to the host systems on which gpu-plugin itself is deployed.

+

When NFD worker runs this hook, it will add a number of labels to the nodes, +which can be used for example to deploy services to nodes with specific GPU +types. Selected numeric labels can be turned into kubernetes extended resources +by the NFD, allowing for finer grained resource management for GPU-using PODs.

+

In the NFD deployment, the hook requires /host-sys -folder to have the host /sys-folder content mounted. Write access is not necessary.

+
+
+

GPU memory

+

GPU memory amount is read from sysfs gt/gt* files and turned into a label. +There are two supported environment variables named GPU_MEMORY_OVERRIDE and +GPU_MEMORY_RESERVED. Both are supposed to hold numeric byte amounts. For systems with +older kernel drivers or GPUs which do not support reading the GPU memory +amount, the GPU_MEMORY_OVERRIDE environment variable value is turned into a GPU +memory amount label instead of a read value. GPU_MEMORY_RESERVED value will be +scoped out from the GPU memory amount found from sysfs.

+
+
+

Default labels

+

Following labels are created by default. You may turn numeric labels into extended resources with NFD.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
nametypedescription
gpu.intel.com/millicoresnumbernode GPU count * 1000. Can be used as a finer grained shared execution fraction.
gpu.intel.com/memory.maxnumbersum of detected GPU memory amounts in bytes OR environment variable value * GPU count
gpu.intel.com/cardsstringlist of card names separated by '.'. The names match host card*-folders under /sys/class/drm/. Deprecated, use gpu-numbers.
gpu.intel.com/gpu-numbersstringlist of numbers separated by '.'. The numbers correspond to device file numbers for the primary nodes of given GPUs in kernel DRI subsystem, listed as /dev/dri/card<num> in devfs, and /sys/class/drm/card<num> in sysfs.
gpu.intel.com/tilesnumbersum of all detected GPU tiles in the system.
gpu.intel.com/numa-gpu-mapstringlist of numa node to gpu mappings.

If the value of the gpu-numbers label would not fit into the 63 character length limit, you will also get labels gpu-numbers2, +gpu-numbers3… until all the gpu numbers have been labeled.

+

The tile count gpu.intel.com/tiles describes the total amount of tiles on the system. System is expected to be homogeneous, and thus the number of tiles per GPU can be calculated by dividing the tile count with GPU count.

+

The numa-gpu-map label is a list of numa to gpu mapping items separated by _. Each list item has a numa node id combined with a list of gpu indices. e.g. 0-1.2.3 would mean: numa node 0 has gpus 1, 2 and 3. More complex example would be: 0-0.1_1-3.4 where numa node 0 would have gpus 0 and 1, and numa node 1 would have gpus 3 and 4. As with gpu-numbers, this label will be extended to multiple labels if the length of the value exceeds the max label length.

+
+
+

PCI-groups (optional)

+

GPUs which share the same pci paths under /sys/devices/pci* can be grouped into a label. GPU nums are separated by ‘.’ and +groups are separated by ‘_’. The label is created only if environment variable named GPU_PCI_GROUPING_LEVEL has a value greater +than zero. GPUs are considered to belong to the same group, if as many identical folder names are found for the GPUs, as is the value +of the environment variable. Counting starts from the folder name which starts with pci.

+

For example, the SG1 card has 4 GPUs, which end up sharing pci-folder names under /sys/devices. With a GPU_PCI_GROUPING_LEVEL +of 3, a node with two such SG1 cards could produce a pci-groups label with a value of 0.1.2.3_4.5.6.7.

+ + + + + + + + + + + + + + + +
nametypedescription
gpu.intel.com/pci-groupsstringlist of pci-groups separated by '_'. GPU numbers in the groups are separated by '.'. The numbers correspond to device file numbers for the primary nodes of given GPUs in kernel DRI subsystem, listed as /dev/dri/card<num> in devfs, and /sys/class/drm/card<num> in sysfs.

If the value of the pci-groups label would not fit into the 63 character length limit, you will also get labels pci-groups2, +pci-groups3… until all the pci groups have been labeled.

+
+
+

Capability labels (optional)

+

Capability labels are created from information found inside debugfs, and therefore +unfortunately require running the NFD worker as root. Due to coming from debugfs, +which is not guaranteed to be stable, these are not guaranteed to be stable either. +If you do not need these, simply do not run NFD worker as root, that is also more secure. +Depending on your kernel driver, running the NFD hook as root may introduce following labels:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
nametypedescription
gpu.intel.com/platform_genstringGPU platform generation name, typically an integer. Deprecated.
gpu.intel.com/media_versionstringGPU platform Media pipeline generation name, typically a number. Deprecated.
gpu.intel.com/graphics_versionstringGPU platform graphics/compute pipeline generation name, typically a number. Deprecated.
gpu.intel.com/platform_<PLATFORM_NAME>.countnumberGPU count for the named platform.
gpu.intel.com/platform_<PLATFORM_NAME>.tilesnumberGPU tile count in the GPUs of the named platform.
gpu.intel.com/platform_<PLATFORM_NAME>.presentstring"true" for indicating the presense of the GPU platform.
+
+

Limitations

+

For the above to work as intended, GPUs on the same node must be identical in their capabilities.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/gpu_plugin/README.html b/0.26/cmd/gpu_plugin/README.html new file mode 100644 index 000000000..3af408701 --- /dev/null +++ b/0.26/cmd/gpu_plugin/README.html @@ -0,0 +1,530 @@ + + + + + + + Intel GPU device plugin for Kubernetes — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel GPU device plugin for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

Intel GPU plugin facilitates Kubernetes workload offloading by providing access to +discrete (including Intel® Data Center GPU Flex Series) and integrated Intel GPU devices +supported by the host kernel.

+

Use cases include, but are not limited to:

+
    +

  • Media transcode

    • +

  • Media analytics

    • +

  • Cloud gaming

    • +

  • High performance computing

    • +

  • AI training and inference

    • +
+

For example containers with Intel media driver (and components using that), can offload +video transcoding operations, and containers with the Intel OpenCL / oneAPI Level Zero +backend libraries can offload compute operations to GPU.

+
+
+

Modes and Configuration Options

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FlagArgumentDefaultMeaning
-enable-monitoring-disabledEnable 'i915_monitoring' resource that provides access to all Intel GPU devices on the node
-resource-manager-disabledEnable fractional resource management, see also dependencies
-shared-dev-numint1Number of containers that can share the same GPU device
-allocation-policystringnone3 possible values: balanced, packed, none. It is meaningful when shared-dev-num > 1, balanced mode is suitable for workload balance among GPU devices, packed mode is suitable for making full use of each GPU device, none mode is the default. Allocation policy does not have effect when resource manager is enabled.

The plugin also accepts a number of other arguments (common to all plugins) related to logging. +Please use the -h option to see the complete list of logging related options.

+
+
+

Installation

+

The following sections detail how to obtain, build, deploy and test the GPU device plugin.

+

Examples are provided showing how to deploy the plugin either using a DaemonSet or by hand on a per-node basis.

+
+

Prerequisites

+

Access to a GPU device requires firmware, kernel and user-space +drivers supporting it. Firmware and kernel driver need to be on the +host, user-space drivers in the GPU workload containers.

+

Intel GPU devices supported by the current kernel can be listed with:

+
$ grep i915 /sys/class/drm/card?/device/uevent
+/sys/class/drm/card0/device/uevent:DRIVER=i915
+/sys/class/drm/card1/device/uevent:DRIVER=i915
+
+
+
+

Drivers for discrete GPUs

+
+
Kernel driver
+
+
Intel DKMS packages
+

i915 GPU driver DKMS^dkms package is recommended until Intel +discrete GPU support in upstream is complete. It can be installed +from Intel package repositories for a subset of older kernel versions +used in enterprise / LTS distributions: +https://dgpu-docs.intel.com/installation-guides/index.html

+
+
+
Upstream kernel
+

With upstream 6.x kernels, discrete GPU support needs to be enabled using +kernel i915.force_probe=<PCI_ID> command line option until relevant kernel +driver features have been completed also in upstream: +https://www.kernel.org/doc/html/latest/gpu/rfc/index.html

+

PCI IDs for the Intel GPUs on given host can be listed with:

+
$ lspci | grep -e VGA -e Display | grep Intel
+88:00.0 Display controller: Intel Corporation Device 56c1 (rev 05)
+8d:00.0 Display controller: Intel Corporation Device 56c1 (rev 05)
+
+
+

(lspci lists GPUs with display support as “VGA compatible controller”, +and server GPUs without display support, as “Display controller”.)

+

Mesa “Iris” 3D driver header provides a mapping between GPU PCI IDs and their Intel brand names: +https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/include/pci_ids/iris_pci_ids.h

+

If your kernel build does not find the correct firmware version for +a given GPU from the host (see dmesg | grep i915 output), latest +firmware versions are available in upstream: +https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/i915

+
+
+
+
User-space drivers
+

Until new enough user-space drivers (supporting also discrete GPUs) +are available directly from distribution package repositories, they +can be installed to containers from Intel package repositories. See: +https://dgpu-docs.intel.com/installation-guides/index.html

+

Example container is listed in Testing and demos.

+

Validation status against upstream kernel is listed in the user-space drivers release notes:

+
    +

  • Media driver: https://github.com/intel/media-driver/releases

    • +

  • Compute driver: https://github.com/intel/compute-runtime/releases

    • +
+
+
+
+

Drivers for older (integrated) GPUs

+

For the older (integrated) GPUs, new enough firmware and kernel driver +are typically included already with the host OS, and new enough +user-space drivers (for the GPU containers) are in the host OS +repositories.

+
+
+
+

Pre-built Images

+

Pre-built images +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository.

+

Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format x.y.z, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command

+

Note: Replace <RELEASE_VERSION> with the desired release tag or main to get devel images.

+

See the development guide for details if you want to deploy a customized version of the plugin.

+
+

Install to all nodes

+

Simplest option to enable use of Intel GPUs in Kubernetes Pods.

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin?ref=<RELEASE_VERSION>'
+
+
+
+
+

Install to nodes with Intel GPUs with NFD

+

Deploying GPU plugin to only nodes that have Intel GPU attached. Node Feature Discovery is required to detect the presence of Intel GPUs.

+
# Start NFD - if your cluster doesn't have NFD installed yet
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=<RELEASE_VERSION>'
+
+# Create NodeFeatureRules for detecting GPUs on nodes
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=<RELEASE_VERSION>'
+
+# Create GPU plugin daemonset
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/nfd_labeled_nodes?ref=<RELEASE_VERSION>'
+
+
+
+
+

Install to nodes with NFD, Monitoring and Shared-dev

+

Same as above, but configures GPU plugin with logging, monitoring and shared-dev features enabled. This option is useful when there is a desire to retrieve GPU metrics from nodes. For example with XPU-Manager or collectd.

+
# Start NFD - if your cluster doesn't have NFD installed yet
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=<RELEASE_VERSION>'
+
+# Create NodeFeatureRules for detecting GPUs on nodes
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=<RELEASE_VERSION>'
+
+# Create GPU plugin daemonset
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/monitoring_shared-dev_nfd/?ref=<RELEASE_VERSION>'
+
+
+
+
+

Install to nodes with Intel GPUs with Fractional resources

+

With the experimental fractional resource feature you can use additional kubernetes extended +resources, such as GPU memory, which can then be consumed by deployments. PODs will then only +deploy to nodes where there are sufficient amounts of the extended resources for the containers.

+

(For this to work properly, all GPUs in a given node should provide equal amount of resources +i.e. heteregenous GPU nodes are not supported.)

+

Enabling the fractional resource feature isn’t quite as simple as just enabling the related +command line flag. The DaemonSet needs additional RBAC-permissions +and access to the kubelet podresources gRPC service, plus there are other dependencies to +take care of, which are explained below. For the RBAC-permissions, gRPC service access and +the flag enabling, it is recommended to use kustomization by running:

+
# Start NFD with GPU related configuration changes
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/gpu?ref=<RELEASE_VERSION>'
+
+# Create NodeFeatureRules for detecting GPUs on nodes
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=<RELEASE_VERSION>'
+
+# Create GPU plugin daemonset
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/fractional_resources?ref=<RELEASE_VERSION>'
+
+
+
+
Fractional resources details
+

Usage of these fractional GPU resources requires that the cluster has node +extended resources with the name prefix gpu.intel.com/. Those can be created with NFD +by running the hook installed by the plugin initcontainer. When fractional resources are +enabled, the plugin lets a scheduler extender +do card selection decisions based on resource availability and the amount of extended +resources requested in the pod spec.

+

The scheduler extender then needs to annotate the pod objects with unique +increasing numeric timestamps in the annotation gas-ts and container card selections in +gas-container-cards annotation. The latter has container separator ‘|’ and card separator +‘,’. Example for a pod with two containers and both containers getting two cards: +gas-container-cards:card0,card1|card2,card3. Enabling the fractional-resource support +in the plugin without running such an annotation adding scheduler extender in the cluster +will only slow down GPU-deployments, so do not enable this feature unnecessarily.

+

In multi-tile systems, containers can request individual tiles to improve GPU resource usage. +Tiles targeted for containers are specified to pod via gas-container-tiles annotation where the the annotation +value describes a set of card and tile combinations. For example in a two container pod, the annotation +could be gas-container-tiles:card0:gt0+gt1|card1:gt1,card2:gt0. Similarly to gas-container-cards, the container +details are split via |. In the example above, the first container gets tiles 0 and 1 from card 0, +and the second container gets tile 1 from card 1 and tile 0 from card 2.

+
+

Note: It is also possible to run the GPU device plugin using a non-root user. To do this, +the nodes’ DAC rules must be configured to device plugin socket creation and kubelet registration. +Furthermore, the deployments securityContext must be configured with appropriate runAsUser/runAsGroup.

+
+
+
+
+
+

Verify Plugin Registration

+

You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes:

+
$ kubectl get nodes -o=jsonpath="{range .items[*]}{.metadata.name}{'\n'}{' i915: '}{.status.allocatable.gpu\.intel\.com/i915}{'\n'}"
+master
+ i915: 1
+
+
+
+
+
+

Testing and Demos

+

The GPU plugin functionality can be verified by deploying an OpenCL image which runs clinfo outputting the GPU capabilities (detected by driver installed to the image).

+
    +

  1. Make the image available to the cluster:

    +

    Build image:

    +
    $ make intel-opencl-icd
+
    +
    +

    Tag and push the intel-opencl-icd image to a repository available in the cluster. Then modify the intelgpu-job.yaml’s image location accordingly:

    +
    $ docker tag intel/intel-opencl-icd:devel <repository>/intel/intel-opencl-icd:latest
+$ docker push <repository>/intel/intel-opencl-icd:latest
+$ $EDITOR ${INTEL_DEVICE_PLUGINS_SRC}/demo/intelgpu-job.yaml
+
    +
    +

    If you are running the demo on a single node cluster, and do not have your own registry, you can add image to node image cache instead. For example, to import docker image to containerd cache:

    +
    $ IMAGE_NAME=opencl-icd.tar
+$ docker save -o $IMAGE_NAME intel/intel-opencl-icd:devel
+$ ctr -n=k8s.io images import $IMAGE_NAME
+$ rm $IMAGE_NAME
+
    +
    +
    2. +

  2. Create a job:

    +
    $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/intelgpu-job.yaml
+job.batch/intelgpu-demo-job created
+
    +
    +
    3. +

  3. Review the job’s logs:

    +
    $ kubectl get pods | fgrep intelgpu
+# substitute the 'xxxxx' below for the pod name listed in the above
+$ kubectl logs intelgpu-demo-job-xxxxx
+<log output>
+
    +
    +

    If the pod did not successfully launch, possibly because it could not obtain +the requested GPU resource, it will be stuck in the Pending status:

    +
    $ kubectl get pods
+NAME                      READY   STATUS    RESTARTS   AGE
+intelgpu-demo-job-xxxxx   0/1     Pending   0          8s
+
    +
    +

    This can be verified by checking the Events of the pod:

    +
    $ kubectl describe pod intelgpu-demo-job-xxxxx
+...
+Events:
+  Type     Reason            Age        From               Message
+  ----     ------            ----       ----               -------
+  Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 Insufficient gpu.intel.com/i915.
+
    +
    +
    4. +
+
+
+

Issues with media workloads on multi-GPU setups

+

Unlike with 3D & compute, and OneVPL media API, QSV (MediaSDK) & VA-API +media APIs do not offer device discovery functionality for applications. +There is nothing (e.g. environment variable) with which the default +device could be overridden either.

+

As result, most (all?) media applications using VA-API or QSV, fail to +locate the correct GPU device file unless it is the first (”renderD128”) +one, or device file name is explictly specified with an application option.

+

Kubernetes device plugins expose only requested number of device +files, and their naming matches host device file names (for several +reasons unrelated to media). Therefore, on multi-GPU hosts, the only +GPU device file mapped to the media container can be some other one +than “renderD128”, and media applications using VA-API or QSV need to +be explicitly told which one to use.

+

These options differ from application to application. Relevant FFmpeg +options are documented here:

+
    +

  • VA-API: https://trac.ffmpeg.org/wiki/Hardware/VAAPI

    • +

  • QSV: https://github.com/Intel-Media-SDK/MediaSDK/wiki/FFmpeg-QSV-Multi-GPU-Selection-on-Linux

    • +
+
+

Workaround for QSV and VA-API

+

Render device shell script locates and outputs the +correct device file name. It can be added to the container and used +to give device file name for the application.

+

Use it either from another script invoking the application, or +directly from the Pod YAML command line. In latter case, it can be +used either to add the device file name to the end of given command +line, like this:

+
command: ["render-device.sh", "vainfo", "--display", "drm", "--device"]
+
+=> /usr/bin/vainfo --display drm --device /dev/dri/renderDXXX
+
+
+

Or inline, like this:

+
command: ["/bin/sh", "-c",
+          "vainfo --device $(render-device.sh 1) --display drm"
+         ]
+
+
+

If device file name is needed for multiple commands, one can use shell variable:

+
command: ["/bin/sh", "-c",
+          "dev=$(render-device.sh 1) && vainfo --device $dev && <more commands>"
+         ]
+
+
+

With argument N, script outputs name of the Nth suitable GPU device +file, which can be used when more than one GPU resource was requested.

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/iaa_plugin/README.html b/0.26/cmd/iaa_plugin/README.html new file mode 100644 index 000000000..79c81b64c --- /dev/null +++ b/0.26/cmd/iaa_plugin/README.html @@ -0,0 +1,244 @@ + + + + + + + Intel IAA device plugin for Kubernetes — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel IAA device plugin for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

The IAA device plugin for Kubernetes supports acceleration using the Intel Analytics accelerator(IAA).

+

The IAA plugin discovers IAA work queues and presents them as a node resources.

+

The IAA plugin and operator optionally support provisioning of IAA devices and workqueues with the help of accel-config utility through initcontainer.

+
+
+

Installation

+

The following sections detail how to use the IAA device plugin.

+
+

Pre-built Images

+

Pre-built images +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository.

+

Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format x.y.z, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/iaa_plugin?ref=<RELEASE_VERSION>'
+daemonset.apps/intel-iaa-plugin created
+
+
+

Where <RELEASE_VERSION> needs to be substituted with the desired release tag or main to get devel images.

+

Nothing else is needed. See the development guide for details if you want to deploy a customized version of the plugin.

+
+

Automatic Provisioning

+

There’s a sample idxd initcontainer included that provisions IAA devices and workqueues (1 engine / 1 group / 1 wq (user/dedicated)), to deploy:

+
$ kubectl apply -k deployments/iaa_plugin/overlays/iaa_initcontainer/
+
+
+

The provisioning script and template are available for customization.

+

The provisioning config can be optionally stored in the ProvisioningConfig configMap which is then passed to initcontainer through the volume mount.

+

There’s also a possibility for a node specific congfiguration through passing a nodename via NODE_NAME into initcontainer’s environment and passing a node specific profile via configMap volume mount.

+

To create a custom provisioning config:

+
$ kubectl create configmap --namespace=inteldeviceplugins-system intel-iaa-config --from-file=demo/iaa.conf
+
+
+
+
+
+

Verify Plugin Registration

+

You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes:

+
$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{"  "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\(  iaa\)'
+master
+  iaa.intel.com/wq-user-dedicated: 2
+  iaa.intel.com/wq-user-shared: 10
+node1
+ iaa.intel.com/wq-user-dedicated: 4
+ iaa.intel.com/wq-user-shared: 30
+
+
+
+
+
+

Testing and Demos

+

We can test the plugin is working by deploying the provided example iaa-qpl-demo test image.

+
    +

  1. Build a Docker image with an accel-config tests:

    +
    $ make iaa-qpl-demo
+...
+Successfully tagged intel/iaa-qpl-demo:devel
+
    +
    +
    2. +

  2. Create a pod running unit tests off the local Docker image:

    +
    $ kubectl apply -f ./demo/iaa-qpl-demo-pod.yaml
+pod/iaa-qpl-demo created
+
    +
    +
    3. +

  3. Wait until pod is completed:

    +
    $ kubectl get pods  |grep iaa-qpl-demo
+iaa-qpl-demo    0/1     Completed   0          31m
+
+If the pod did not successfully launch, possibly because it could not obtain the IAA
+resource, it will be stuck in the `Pending` status:
+
+```bash
+$ kubectl get pods
+NAME                      READY   STATUS    RESTARTS   AGE
+iaa-qpl-demo              0/1     Pending   0          7s
+
    +
    +

    This can be verified by checking the Events of the pod:

    +
    $ kubectl describe pod iaa-qpl-demo | grep -A3 Events:
+Events:
+  Type     Reason            Age    From               Message
+  ----     ------            ----   ----               -------
+  Warning  FailedScheduling  2m26s  default-scheduler  0/1 nodes are available: 1 Insufficient iaa.intel.com/wq-user-dedicated, 1 Insufficient iaa.intel.com/wq-user-shared.
+
    +
    +
    4. +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/operator/README.html b/0.26/cmd/operator/README.html new file mode 100644 index 000000000..b9683b04a --- /dev/null +++ b/0.26/cmd/operator/README.html @@ -0,0 +1,284 @@ + + + + + + + Intel Device Plugins Operator — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel Device Plugins Operator

+

Table of Contents

+ +
+

Introduction

+

Intel Device Plugins Operator is a Kubernetes custom controller whose goal is to serve the +installation and lifecycle management of Intel device plugins for Kubernetes. +It provides a single point of control for GPU, QAT, SGX, FPGA, DSA and DLB devices to a cluster +administrators.

+
+
+

Installation

+

Install NFD (if it’s not already installed) and node labelling rules (requires NFD v0.10+):

+
# either with default NFD installation
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=<RELEASE_VERSION>'
+# or when setting up with SGX
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/sgx?ref=<RELEASE_VERSION>'
+# and finally, NodeFeatureRules
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=<RELEASE_VERSION>'
+
+
+

Make sure both NFD master and worker pods are running:

+
$ kubectl get pods -n node-feature-discovery
+NAME                          READY   STATUS    RESTARTS   AGE
+nfd-master-599c58dffc-9wql4   1/1     Running   0          25h
+nfd-worker-qqq4h              1/1     Running   0          25h
+
+
+

Note that labelling is not performed immediately. Give NFD 1 minute to pick up the rules and label nodes.

+

As a result all found devices should have correspondent labels, e.g. for Intel DLB devices the label is +intel.feature.node.kubernetes.io/dlb:

+
$ kubectl get no -o json | jq .items[].metadata.labels |grep intel.feature.node.kubernetes.io/dlb
+  "intel.feature.node.kubernetes.io/dlb": "true",
+
+
+

Full list of labels can be found in the deployments/operator/samples directory:

+
$ grep -r feature.node.kubernetes.io/ deployments/operator/samples/
+deployments/operator/samples/deviceplugin_v1_dlbdeviceplugin.yaml:    intel.feature.node.kubernetes.io/dlb: 'true'
+deployments/operator/samples/deviceplugin_v1_qatdeviceplugin.yaml:    intel.feature.node.kubernetes.io/qat: 'true'
+deployments/operator/samples/deviceplugin_v1_sgxdeviceplugin.yaml:    intel.feature.node.kubernetes.io/sgx: 'true'
+deployments/operator/samples/deviceplugin_v1_gpudeviceplugin.yaml:    intel.feature.node.kubernetes.io/gpu: "true"
+deployments/operator/samples/deviceplugin_v1_fpgadeviceplugin.yaml:    intel.feature.node.kubernetes.io/fpga-arria10: 'true'
+deployments/operator/samples/deviceplugin_v1_dsadeviceplugin.yaml:    intel.feature.node.kubernetes.io/dsa: 'true'
+
+
+

The default operator deployment depends on cert-manager running in the cluster. +See installation instructions here.

+

Make sure all the pods in the cert-manager namespace are up and running:

+
$ kubectl get pods -n cert-manager
+NAME                                      READY   STATUS    RESTARTS   AGE
+cert-manager-7747db9d88-bd2nl             1/1     Running   0          21d
+cert-manager-cainjector-87c85c6ff-59sb5   1/1     Running   0          21d
+cert-manager-webhook-64dc9fff44-29cfc     1/1     Running   0          21d
+
+
+

Also if your cluster operates behind a corporate proxy make sure that the API +server is configured not to send requests to cluster services through the +proxy. You can check that with the following command:

+
$ kubectl describe pod kube-apiserver --namespace kube-system | grep -i no_proxy | grep "\.svc"
+
+
+

In case there’s no output and your cluster was deployed with kubeadm open +/etc/kubernetes/manifests/kube-apiserver.yaml at the control plane nodes and +append .svc and .svc.cluster.local to the no_proxy environment variable:

+
apiVersion: v1
+kind: Pod
+metadata:
+  ...
+spec:
+  containers:
+  - command:
+    - kube-apiserver
+    - --advertise-address=10.237.71.99
+    ...
+    env:
+    - name: http_proxy
+      value: http://proxy.host:8080
+    - name: https_proxy
+      value: http://proxy.host:8433
+    - name: no_proxy
+      value: 127.0.0.1,localhost,.example.com,10.0.0.0/8,.svc,.svc.cluster.local
+    ...
+
+
+

Note: To build clusters using kubeadm with the right no_proxy settings from the very beginning, +set the cluster service names to $no_proxy before kubeadm init:

+
$ export no_proxy=$no_proxy,.svc,.svc.cluster.local
+
+
+

Finally deploy the operator itself:

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/operator/default?ref=<RELEASE_VERSION>'
+
+
+

Now you can deploy the device plugins by creating corresponding custom resources. +The samples for them are available here.

+
+
+

Usage

+

Deploy your device plugin by applying its custom resource, e.g. +GpuDevicePlugin with

+
$ kubectl apply -f https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/main/deployments/operator/samples/deviceplugin_v1_gpudeviceplugin.yaml
+
+
+

Observe it is up and running:

+
$ kubectl get GpuDevicePlugin
+NAME                     DESIRED   READY   NODE SELECTOR   AGE
+gpudeviceplugin-sample   1         1                       5s
+
+
+

In order to limit the deployment to a specific device type, +use one of kustomizations under deployments/operator/device.

+

For example, to limit the deployment to FPGA, use:

+
$ kubectl apply -k deployments/operator/device/fpga
+
+
+

Operator also supports deployments with multiple selected device types. +In this case, create a new kustomization with the necessary resources +that passes the desired device types to the operator using --device +command line argument multiple times.

+
+
+

Upgrade

+

The upgrade of the deployed plugins can be done by simply installing a new release of the operator.

+

The operator auto-upgrades operator-managed plugins (CR images and thus corresponding deployed daemonsets) to the current release of the operator.

+

The [registry-url]/[namespace]/[image] are kept intact on the upgrade.

+

No upgrade is done for:

+
    +

  • Non-operator managed deployments

    • +

  • Operator deployments without numeric tags

    • +
+
+
+

Known issues

+

When the operator is run with leader election enabled, that is with the option +--leader-elect, make sure the cluster is not overloaded with excessive +number of pods. Otherwise a heart beat used by the leader election code may trigger +a timeout and crash. We are going to use different clients for the controller and +leader election code to alleviate the issue. See more details in +https://github.com/intel/intel-device-plugins-for-kubernetes/issues/476.

+

In case the deployment is limited to specific device type(s), +the CRDs for other device types are still created, but no controllers +for them are registered.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/operator/ocp_quickstart_guide/README.html b/0.26/cmd/operator/ocp_quickstart_guide/README.html new file mode 100644 index 000000000..9a5952f73 --- /dev/null +++ b/0.26/cmd/operator/ocp_quickstart_guide/README.html @@ -0,0 +1,207 @@ + + + + + + + Intel® Device Plugins Operator for Red Hat OpenShift Container Platform — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+
    +
    • +
  • Intel® Device Plugins Operator for Red Hat OpenShift Container Platform
    • +
  • + View page source +
    • +
+
+
+
+
+ +
+

Intel® Device Plugins Operator for Red Hat OpenShift Container Platform

+
+

Table of Contents

+ +
+
+

Introduction

+

The Intel Device Plugins Operator for OpenShift Container Platform is a collection of device plugins advertising Intel specific hardware resources to the kubelet. It provides a single point of control for Intel® Software Guard Extensions (Intel® SGX), Intel GPUs, Intel® QuickAccess Technology (Intel® QAT), Intel® Data Streaming Accelerator (Intel® DSA), and Intel® In-Memory Analytics Accelerator (Intel® IAA) devices to cluster administrators. The v0.24.0 release of the operator only supports Intel SGX and Intel QAT device plugins. GPU, Intel DSA, Intel IAA, and other device plugins will be supported in future releases.

+
+
+

Minimum Hardware Requirements

+
+

Intel SGX Enabled Server

+
    +

  • Third Generation Intel® Xeon® Scalable Platform, code-named “Ice Lake” or later

    • +

  • Configure BIOS using below details +SGX Server BIOS +[Note: The BIOS configuration shown above is just for the reference. Please contact your BIOS vendor for details]

    • +
+
+
+
+

Installation

+
+

Prerequisites

+
    +

  • Make sure Red Hat OpenShift Cluster is ready to use and the developing machine is RHEL and oc command is installed and configured properly. Please note that the following operation is verified on Red Hat OpenShift Cluster 4.11 and working machine RHEL-8.6

    • +

  • Install the oc command to your development machine

    • +

  • Follow the link to install NFD operator (if it’s not already installed).
    Note: Please only install the NFD operator and use steps below to create the NodeFeatureDiscovery instance.

    +
      +

    • Create the NodeFeatureDiscovery instance

      • +
    +
    $ oc apply -f https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/v0.24.0/deployments/nfd/overlays/node-feature-discovery/node-feature-discovery-openshift.yaml
+
    +
    +
      +

    • Create the NodeFeatureRule instance

      • +
    +
    $ oc apply -f https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/v0.24.0/deployments/nfd/overlays/node-feature-rules/node-feature-rules-openshift.yaml
+
    +
    +
    • +

  • Deploy SELinux Policy for OCP 4.10 -
    The SGX device plugin and Init container run as a label container_device_plugin_t and container_device_plugin_init_t respectively. This requires a custom SELinux policy to be deployed before the SGX plugin can be run. To deploy this policy, run

    +
    $ oc apply -f https://raw.githubusercontent.com/intel/user-container-selinux/main/policy-deployment.yaml
+
    +
    +
    • +
+
+
+

Install Operator using OpenShift Web Console

+
    +

  1. In OpenShift web console navigate to Operator -> OperatorHub

    2. +

  2. Search for Intel Device Plugins Operator -> Click Install

    3. +
+
+
+

Verify Operator installation

+
    +

  1. Go to Operator -> Installed Operators

    2. +

  2. Verify the status of operator as Succeeded

    3. +

  3. Click Intel Device Plugins Operator to view the details
    Verify Operator

    4. +
+
+
+
+

Deploying Intel Device Plugins

+
+

Intel SGX Device Plugin

+

Follow the steps below to deploy Intel SGX Device Plugin Custom Resource

+
    +

  1. Go to Operator -> Installed Operators

    2. +

  2. Open Intel Device Plugins Operator

    3. +

  3. Navigate to tab Intel Software Guard Extensions Device Plugin

    4. +

  4. Click Create SgxDevicePlugin -> set correct parameters -> Click Create +OR for any customizations, please select YAML view and edit details. Once done, click Create

    5. +

  5. Verify CR by checking the status of DaemonSet intel-sgx-plugin

    6. +

  6. Now SgxDevicePlugin is ready to deploy any workloads

    7. +
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/qat_plugin/README.html b/0.26/cmd/qat_plugin/README.html new file mode 100644 index 000000000..26f117611 --- /dev/null +++ b/0.26/cmd/qat_plugin/README.html @@ -0,0 +1,418 @@ + + + + + + + Intel QuickAssist Technology (QAT) device plugin for Kubernetes — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel QuickAssist Technology (QAT) device plugin for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

This Intel QAT device plugin provides support for Intel QAT devices under Kubernetes. +The supported devices are determined by the VF device drivers available in your Linux +Kernel. See the Prerequisites section for more details.

+

Supported Devices include, but may not be limited to, the following:

+ +

The QAT device plugin provides access to QAT hardware accelerated cryptographic and compression features. +Demonstrations are provided utilising DPDK and OpenSSL.

+

Kata Containers QAT integration is documented in the +Kata Containers documentation repository.

+
+
+

Modes and Configuration Options

+

The QAT plugin can take a number of command line arguments, summarised in the following table:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FlagArgumentMeaning
-dpdk-driverstringDPDK Device driver for configuring the QAT device (default: vfio-pci)
-kernel-vf-driversstringComma separated VF Device Driver of the QuickAssist Devices in the system. Devices supported: DH895xCC, C62x, C3xxx, 4xxx/401xx, C4xxx and D15xx (default: c6xxvf,4xxxvf)
-max-num-devicesintmaximum number of QAT devices to be provided to the QuickAssist device plugin (default: 32)
-modestringplugin mode which can be either dpdk or kernel (default: dpdk)
-allocation-policystring2 possible values: balanced and packed. Balanced mode spreads allocated QAT VF resources balanced among QAT PF devices, and packed mode packs one QAT PF device full of QAT VF resources before allocating resources from the next QAT PF. (There is no default.)

The plugin also accepts a number of other arguments related to logging. Please use the -h option to see +the complete list of logging related options.

+

For more details on the -dpdk-driver choice, see +DPDK Linux Driver Guide.

+
+

Note:: With Linux 5.9+ kernels the vfio-pci module must be loaded with +disable_denylist=1 parameter for the QAT device plugin to work correctly with +devices prior to Gen4 (4xxx).

+
+

For more details on the available options to the -kernel-vf-drivers option, see the list of +vf drivers available in the Linux Kernel.

+

If the -mode parameter is set to kernel, no other parameter documented above are valid, +except the klog logging related parameters. +kernel mode implements resource allocation based on system configured logical instances.

+
+

Note: kernel mode is excluded by default from all builds (including those hosted on the Docker hub), +by default. See the Build the plugin image section for more details.

+
+

The kernel mode does not guarantee full device isolation between containers +and therefore it’s not recommended. This mode will be deprecated and removed once libqat +implements non-UIO based device access.

+
+
+

Installation

+

The below sections cover how to obtain, build and install this component.

+

The component can be installed either using a DaemonSet or running ‘by hand’ on each node.

+
+

Prerequisites

+

The component has the same basic dependancies as the +generic plugin framework dependencies.

+

You will also need appropriate hardware installed.

+

The QAT plugin requires Linux Kernel VF QAT drivers to be available. These drivers +are available via two methods. One of them must be installed and enabled:

+ +

The demonstrations have their own requirements, listed in their own specific sections.

+
+
+

Pre-built Images

+

Pre-built images +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository.

+

Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format x.y.z, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_plugin?ref=<RELEASE_VERSION>'
+
+
+

Where <RELEASE_VERSION> needs to be substituted with the desired release tag or main to get devel images.

+

An alternative kustomization for deploying the plugin is with the debug mode switched on:

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_plugin/overlays/debug?ref=<RELEASE_VERSION>'
+
+
+
+

Note: It is also possible to run the QAT device plugin using a non-root user. To do this, +the nodes’ DAC rules must be configured to allow PCI driver unbinding/binding, device plugin +socket creation and kubelet registration. Furthermore, the deployments securityContext must +be configured with appropriate runAsUser/runAsGroup.

+
+
+

Automatic Provisioning

+

There’s a sample qat initcontainer. Regardless of device types, the script running inside the initcontainer enables QAT SR-IOV VFs.

+

To deploy, run as follows:

+
$ kubectl apply -k deployments/qat_plugin/overlays/qat_initcontainer/
+
+
+

In addition to the default configuration, you can add device-specific configurations via ConfigMap.

+ + + + + + + + + + + + + + + + + + + +
DevicePossible ConfigurationHow To CustomizeOptionsNotes
4xxx, 401xxcfg_services reports the configured services (crypto services or compression services) of the QAT device.ServicesEnabled=<value>compress:dc, crypto:sym;asymLinux 6.0+ kernel is required.

To create a provisioning config after customizing, run as follows:

+
$ kubectl create configmap --namespace=inteldeviceplugins-system qat-config --from-file=deployments/qat_plugin/overlays/qat_initcontainer/qat.conf
+
+
+
+

Note: When deploying the overlay qat_initcontainer, such a manual creation is not necessary since ConfigMap is generated automatically. Just set the values in the config file and deploy the overlay.

+
+

When using the operator for deploying the plugin with provisioning config, use provisioningConfig field for the name of the ConfigMap, then the config is passed to initcontainer through the volume mount.

+

There’s also a possibility for a node specific congfiguration through passing a nodename via NODE_NAME into initcontainer’s environment and passing a node specific profile (qat-$NODE_NAME.conf) via ConfigMap volume mount.

+
+
+

Verify Plugin Registration

+

Verification of the plugin deployment and detection of QAT hardware can be confirmed by +examining the resource allocations on the nodes:

+
$ kubectl describe node <node name> | grep qat.intel.com/generic
+ qat.intel.com/generic: 10
+ qat.intel.com/generic: 10
+
+
+
+
+
+
+

Demos and Testing

+

The below sections cover DPDK and OpenSSL demos, both of which utilise the +QAT device plugin under Kubernetes.

+
+

DPDK QAT demos

+

The Data Plane Development Kit (DPDK) QAT demos use DPDK +crypto-perf and +compress-perf utilities to exercise +DPDK QAT Poll-Mode Drivers (PMD). For more information on the tools’ parameters, refer to the +website links.

+
+

DPDK Prerequisites

+

For the DPDK QAT demos to work, the DPDK drivers must be loaded and configured. +For more information, refer to: +DPDK Getting Started Guide for Linux and +DPDK Getting Started Guide, Linux Drivers section

+
+
+

Deploy the pod

+

In the pod specification file, add container resource request and limit. +For example, qat.intel.com/generic: <number of devices> for a container requesting QAT devices.

+

For a DPDK-based workload, you may need to add hugepage request and limit.

+
$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_dpdk_app/base/
+$ kubectl get pods
+  NAME                     READY     STATUS    RESTARTS   AGE
+  qat-dpdk                 1/1       Running   0          27m
+  intel-qat-plugin-5zgvb   1/1       Running   0          3h
+
+
+
+

Note: If the igb_uio VF driver is used with the QAT device plugin, +the workload be deployed with SYS_ADMIN capabilities added.

+
+
+
+

Manual Test Run

+

Manually execute the dpdk-test-crypto-perf application to review the logs:

+
$ kubectl exec -it qat-dpdk bash
+
+$ dpdk-test-crypto-perf -l 6-7 -w $QAT1 \
+-d /usr/lib64/librte_mempool_ring.so.1.1 \
+-d /usr/lib64/librte_pmd_qat.so.1.1 -- \
+--ptest throughput --devtype crypto_qat \
+--optype cipher-only --cipher-algo aes-cbc --cipher-op encrypt \
+--cipher-key-sz 16 --total-ops 10000000 --burst-sz 32 --buffer-sz 64
+
+
+
+

Note: Adapt the .so versions to what the DPDK version in the container provides.

+
+
+
+

Automated Test Run

+

It is also possible to deploy and run crypto-perf using the following +kustomize overlays:

+
$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_dpdk_app/test-crypto1
+$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_dpdk_app/test-compress1
+$ kubectl logs qat-dpdk-test-crypto-perf-tc1
+$ kubectl logs qat-dpdk-test-compress-perf-tc1
+
+
+
+

Note: for test-crypto1 and test-compress1 to work, the cluster must enable +Kubernetes CPU manager’s static policy.

+
+
+
+
+

OpenSSL QAT Demo

+

Please refer to the Kata Containers documentation for details on the OpenSSL +QAT acceleration demo.

+
+
+
+

Checking for Hardware

+

In order to utilise the QAT device plugin, QuickAssist SR-IOV virtual functions must be configured. +You can verify this on your nodes by checking for the relevant PCI identifiers:

+
for i in 0442 0443 18a1 37c9 6f55 19e3 4941 4943; do lspci -d 8086:$i; done
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/sgx_admissionwebhook/README.html b/0.26/cmd/sgx_admissionwebhook/README.html new file mode 100644 index 000000000..8168bfe55 --- /dev/null +++ b/0.26/cmd/sgx_admissionwebhook/README.html @@ -0,0 +1,214 @@ + + + + + + + Intel SGX admission controller for Kubernetes — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel SGX admission controller for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

The SGX admission controller is one of the components used to add support for Intel SGX +functionality to Kubernetes.

+
+

NOTE: Installation of the SGX admission controller can be skipped if the +SGX device plugin is operated with the Intel Device Plugins Operator +since it integrates the controller’s functionality.

+
+

The SGX admission webhook is responsible for performing Pod mutations based on the sgx.intel.com/quote-provider +pod annotation set by the user. The purpose of the webhook is to hide the details of setting the necessary +device resources and volume mounts for using SGX remote attestation in the cluster. Furthermore, +the SGX admission webhook is responsible for writing a pod/sandbox sgx.intel.com/epc annotation that is used by +Kata Containers to dynamically adjust its virtualized SGX encrypted page cache (EPC) bank(s) size.

+
+
+

Installation

+

The following sections detail how to obtain, build and deploy the admission +controller webhook plugin.

+
+

Pre-requisites

+

The default webhook deployment depends on having cert-manager +installed. See its installation instructions here.

+

Also if your cluster operates behind a corporate proxy make sure that the API +server is configured not to send requests to cluster services through the +proxy. You can check that with the following command:

+
$ kubectl describe pod kube-apiserver --namespace kube-system | grep -i no_proxy | grep "\.svc"
+
+
+

In case there’s no output and your cluster was deployed with kubeadm open +/etc/kubernetes/manifests/kube-apiserver.yaml at the control plane nodes and +append .svc and .svc.cluster.local to the no_proxy environment variable:

+
apiVersion: v1
+kind: Pod
+metadata:
+  ...
+spec:
+  containers:
+  - command:
+    - kube-apiserver
+    - --advertise-address=10.237.71.99
+    ...
+    env:
+    - name: http_proxy
+      value: http://proxy.host:8080
+    - name: https_proxy
+      value: http://proxy.host:8433
+    - name: no_proxy
+      value: 127.0.0.1,localhost,.example.com,10.0.0.0/8,.svc,.svc.cluster.local
+    ...
+
+
+

Note: To build clusters using kubeadm with the right no_proxy settings from the very beginning, +set the cluster service names to $no_proxy before kubeadm init:

+
$ export no_proxy=$no_proxy,.svc,.svc.cluster.local
+
+
+
+
+

Deployment

+

To deploy the webhook with cert-manager, run

+
$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_admissionwebhook/overlays/default-with-certmanager?ref=main
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/sgx_plugin/README.html b/0.26/cmd/sgx_plugin/README.html new file mode 100644 index 000000000..0da3cdff6 --- /dev/null +++ b/0.26/cmd/sgx_plugin/README.html @@ -0,0 +1,382 @@ + + + + + + + Intel Software Guard Extensions (SGX) device plugin for Kubernetes — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel Software Guard Extensions (SGX) device plugin for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

The Intel SGX device plugin and related components allow workloads to use Intel SGX on +platforms with SGX Flexible Launch Control enabled, e.g.,:

+
    +

  • 3rd Generation Intel® Xeon® Scalable Platform, code-named “Ice Lake”

    • +

  • Intel® Xeon® E3

    • +

  • Intel® NUC Kit NUC7CJYH

    • +
+

The SGX solution comes in three parts:

+ +

This README covers setting up all three components.

+
+

Modes and Configuration options

+

The SGX plugin can take a number of command line arguments, summarised in the following table:

+ + + + + + + + + + + + + + + + + + + + +
FlagArgumentMeaning
-enclave-limitintthe number of containers per worker node allowed to use /dev/sgx_enclave device node (default: 20)
-provision-limitintthe number of containers per worker node allowed to use /dev/sgx_provision device node (default: 20)

The plugin also accepts a number of other arguments related to logging. Please use the -h option to see +the complete list of logging related options.

+
+
+
+

Installation

+

The following sections cover how to use the necessary Kubernetes SGX specific +components.

+
+

Prerequisites

+

The component has the same basic dependencies as the +generic plugin framework dependencies.

+

The SGX device plugin requires Linux Kernel SGX drivers to be available. These drivers +are available in Linux since 5.11. The SGX DCAP out-of-tree driver +(v1.41 and later) is also known to work.

+

The hardware platform must support SGX Flexible Launch Control.

+

The SGX deployment depends on having cert-manager +installed. See its installation instructions here.

+
+
+

Pre-built Images

+

Pre-built images +are available on Docker Hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository.

+

Release tagged images of the components are also available on Docker Hub, tagged with their +release version numbers in the format x.y.z, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy Intel SGX components in your cluster is to follow the steps +below.

+

The deployment YAML files supplied with the components in this repository use the images with the devel +tag by default. If you do not build your own local images, your Kubernetes cluster may pull down +the devel images from Docker Hub by default.

+

Where <RELEASE_VERSION> needs to be substituted with the desired release tag or main to get devel images.

+
+
+

Installation Using the Operator

+

First, deploy node-feature-discovery:

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/sgx?ref=<RELEASE_VERSION>'
+$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=<RELEASE_VERSION>'
+
+
+

Note: The default configuration assumes that the in-tree driver is used and enabled (CONFIG_X86_SGX=y). If +the SGX DCAP out-of-tree driver is used, the kernel.config match expression must be removed.

+

Next, deploy the Intel Device plugin operator:

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/operator/default?ref=<RELEASE_VERSION>'
+
+
+

Note: See the operator deployment details for its dependencies and for setting it up on systems behind proxies.

+

Finally, deploy the SGX device plugin with the operator

+
$ kubectl apply -f 'https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/<RELEASE_VERSION>/deployments/operator/samples/deviceplugin_v1_sgxdeviceplugin.yaml'
+
+
+
+
+

Installation Using kubectl

+

There are two alternative ways to deploy SGX device plugin using kubectl.

+

The first approach involves deployment of the SGX DaemonSet YAML +and node-feature-discovery +with the necessary configuration.

+

There is a kustomization for deploying everything:

+
$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_plugin/overlays/epc-nfd/
+
+
+

The second approach has a lesser deployment footprint. It does not require NFD, but a helper daemonset that creates sgx.intel.com/capable='true' node label and advertises EPC capacity to the API server.

+

The following kustomization is used for this approach:

+
$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_plugin/overlays/epc-register/
+
+
+

Additionally, SGX admission webhook must be deployed

+
$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_admissionwebhook/
+
+
+
+
+

Verify Plugin Registration

+

Verification of the plugin deployment and detection of SGX hardware can be confirmed by +examining the resource allocations on the nodes:

+
$ kubectl describe node <node name> | grep sgx.intel.com
+                   nfd.node.kubernetes.io/extended-resources: sgx.intel.com/epc
+ sgx.intel.com/enclave:    20
+ sgx.intel.com/epc:        98566144
+ sgx.intel.com/provision:  20
+ sgx.intel.com/enclave:    20
+ sgx.intel.com/epc:        98566144
+ sgx.intel.com/provision:  20
+ sgx.intel.com/enclave    1           1
+ sgx.intel.com/epc        400         400
+ sgx.intel.com/provision  1           1
+
+
+
+
+
+

Testing and Demos

+
+

SGX ECDSA Remote Attestation

+

The SGX remote attestation allows a relying party to verify that the software is running inside an Intel® SGX enclave on a platform +that has the trusted computing base up to date.

+

The demo guides to run an SGX DCAP/ECDSA quote generation in on a single-node kubernetes cluster using Intel® reference +SGX PCK Certificate Cache Service (PCCS) that is configured to service localhost connections.

+

Read more about SGX Remote Attestation.

+
+

Remote Attestation Prerequisites

+

For the SGX ECDSA Remote Attestation demo to work, the platform must be correctly registered and a PCCS running.

+

For documentation to set up Intel® reference PCCS, refer to: +Intel® Software Guard Extensions (Intel® SGX) Services and +Intel® Software Guard Extensions SDK for Linux

+

Furthermore, the Kubernetes cluster must be set up according the instructions above.

+
+
+

Build the image

+

The demo uses container images build from Intel® SGX SDK and DCAP releases.

+

To build the demo images:

+
$ cd ${INTEL_DEVICE_PLUGINS_SRC}
+$ make sgx-aesmd-demo
+...
+Successfully tagged intel/sgx-aesmd-demo:devel
+$ make sgx-sdk-demo
+...
+Successfully tagged intel/sgx-sdk-demo:devel
+
+
+
+
+

Deploy the pods

+

The demo runs Intel aesmd (architectural enclaves service daemon) that is responsible +for generating SGX quotes for workloads. It is deployed with hostNetwork: true +to allow connections to localhost PCCS.

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_aesmd?ref=<RELEASE_VERSION>'
+$ kubectl get pods
+  NAME                     READY     STATUS    RESTARTS   AGE
+  intel-sgx-aesmd-mrnm8                1/1     Running   0          3h47m
+  sgxdeviceplugin-sample-z5dcq-llwlw   1/1     Running   0          28m
+
+
+
+

Note: For quick experiments, kind can be used to deploy the cluster. With kind, host path /var/run/aesmd/ must be mounted to the nodes manually using Extra Mounts.
Example:

+
$ cat kind_config.yaml
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+name: <your_node_name>
+nodes:
+- role: control-plane
+  extraMounts:
+  - hostPath: /var/run/aesmd
+    containerPath: /var/run/aesmd
+    propagation: Bidirectional
+
+
+

And bootstrap kind with it
$ kind create cluster --config kind_config.yaml

+
+

The sample application runs SGX DCAP Quote Generation sample:

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/sgx_enclave_apps/overlays/sgx_ecdsa_aesmd_quote?ref=<RELEASE_VERSION>'
+$ kubectl get pods
+  NAME                                 READY   STATUS      RESTARTS   AGE
+  intel-sgx-aesmd-mrnm8                1/1     Running     0          3h55m
+  ecdsa-quote-intelsgx-demo-job-vtq84  0/1     Completed   0          4s
+  sgxdeviceplugin-sample-z5dcq-llwlw   1/1     Running     0          35m
+$ kubectl logs ecdsa-quote-intelsgx-demo-job-vtq84
+
+  Step1: Call sgx_qe_get_target_info:succeed!
+  Step2: Call create_app_report:succeed!
+  Step3: Call sgx_qe_get_quote_size:succeed!
+  Step4: Call sgx_qe_get_quote:succeed!cert_key_type = 0x5
+
+
+
+

Note: The deployment example above uses kustomize +that is available in kubectl since Kubernetes v1.14 release.

+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/vpu_plugin/README.html b/0.26/cmd/vpu_plugin/README.html new file mode 100644 index 000000000..8fba7858d --- /dev/null +++ b/0.26/cmd/vpu_plugin/README.html @@ -0,0 +1,341 @@ + + + + + + + Intel VPU device plugin for Kubernetes — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Intel VPU device plugin for Kubernetes

+

Table of Contents

+ +
+

Introduction

+

The VPU device plugin supports below cards:

+

Intel VCAC-A. +This card has:

+
    +

  • 1 Intel Core i3-7100U processor

    • +

  • 12 MyriadX VPUs

    • +

  • 8GB DDR4 memory

    • +

  • PCIe interface to Xeon E3/E5 server

    • +
+

Intel Mustang V100. +This card has:

+
    +

  • 8 MyriadX VPUs

    • +

  • PCIe interface to 6th+ Generation Core PC or Xeon E3/E5 server

    • +
+

Gen 3 Intel® Movidius™ VPU HDDL VE3 +This card has:

+
    +

  • 3 Intel® Movidius Gen 3 Intel® Movidius™ VPU SoCs

    • +
+

Intel® Movidius™ S VPU +This card has:

+
    +

  • 6 Intel® Movidius Gen 3 Intel® Movidius™ VPU SoCs

    • +
+
+

Note: This device plugin need HDDL daemon service to be running either natively or from a container. +To get VCAC-A or Mustang card running hddl, please refer to: +https://github.com/OpenVisualCloud/Dockerfiles/blob/master/VCAC-A/script/setup_hddl.sh

+
+
+
+

Installation

+

The following sections detail how to use the VPU device plugin.

+
+

Pre-built Images

+

Pre-built images +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository.

+

Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format x.y.z, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command

+
$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/vpu_plugin?ref=<RELEASE_VERSION>'
+daemonset.apps/intel-vpu-plugin created
+
+
+

Where <RELEASE_VERSION> needs to be substituted with the desired release tag or main to get devel images.

+

For xlink device, deploy DaemonSet as

+
$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/vpu_plugin/overlays/xlink
+daemonset.apps/intel-vpu-plugin created
+
+
+

Nothing else is needed. See the development guide for details if you want to deploy a customized version of the plugin.

+
+

Note: It is also possible to run the VPU device plugin using a non-root user. To do this, +the nodes’ DAC rules must be configured to device plugin socket creation and kubelet registration. +Furthermore, the deployments securityContext must be configured with appropriate runAsUser/runAsGroup.

+
+
+
+

Verify Plugin Registration

+

You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes:

+
$ kubectl get nodes -o=jsonpath="{range .items[*]}{.metadata.name}{'\n'}{' hddl: '}{.status.allocatable.vpu\.intel\.com/hddl}{'\n'}"
+vcaanode00
+ hddl: 12
+
+
+
+
+
+

Testing and Demos

+

We can test the plugin is working by deploying the provided example OpenVINO image with HDDL plugin enabled.

+
+

Build a Docker image with an classification example

+
$ cd $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes
+$ make ubuntu-demo-openvino
+...
+Successfully tagged intel/ubuntu-demo-openvino:devel
+
+
+
+
+

Create a job running unit tests off the local Docker image

+
$ cd $(go env GOPATH)/src/github.com/intel/intel-device-plugins-for-kubernetes
+$ kubectl apply -f demo/intelvpu-job.yaml
+job.batch/intelvpu-demo-job created
+
+
+
+
+

Review the job logs

+
$ kubectl get pods | fgrep intelvpu
+# substitute the 'xxxxx' below for the pod name listed in the above
+$ kubectl logs intelvpu-demo-job-xxxxx
++ export HDDL_INSTALL_DIR=/root/hddl
++ HDDL_INSTALL_DIR=/root/hddl
++ export LD_LIBRARY_PATH=/root/inference_engine_samples_build/intel64/Release/lib/
++ LD_LIBRARY_PATH=/root/inference_engine_samples_build/intel64/Release/lib/
++ /root/inference_engine_samples_build/intel64/Release/classification_sample_async -m /root/openvino_models/ir/FP16/classification/squeezenet/1.1/caffe/squeezenet1.1.xml -i /root/car.png -d HDDL
+[ INFO ] InferenceEngine:
+    API version ............ 2.0
+    Build .................. custom_releases/2019/R2_f5827d4773ebbe727c9acac5f007f7d94dd4be4e
+    Description ....... API
+[ INFO ] Parsing input parameters
+[ INFO ] Parsing input parameters
+[ INFO ] Files were added: 1
+[ INFO ]     /root/car.png
+[ INFO ] Creating Inference Engine
+    HDDL
+    HDDLPlugin version ......... 2.0
+    Build ........... 27579
+
+[ INFO ] Loading network files
+[ INFO ] Preparing input blobs
+[ WARNING ] Image is resized from (787, 259) to (227, 227)
+[ INFO ] Batch size is 1
+[ INFO ] Loading model to the device
+[07:49:01.0427][6]I[ServiceStarter.cpp:40] Info: Waiting for HDDL Service getting ready ...
+[07:49:01.0428][6]I[ServiceStarter.cpp:45] Info: Found HDDL Service is running.
+[HDDLPlugin] [07:49:01.0429][6]I[HddlClient.cpp:256] Hddl api version: 2.2
+[HDDLPlugin] [07:49:01.0429][6]I[HddlClient.cpp:259] Info: Create Dispatcher2.
+[HDDLPlugin] [07:49:01.0432][10]I[Dispatcher2.cpp:148] Info: SenderRoutine starts.
+[HDDLPlugin] [07:49:01.0432][6]I[HddlClient.cpp:270] Info: RegisterClient HDDLPlugin.
+[HDDLPlugin] [07:49:01.0435][6]I[HddlClient.cpp:275] Client Id: 3
+[ INFO ] Create infer request
+[HDDLPlugin] [07:49:01.7235][6]I[HddlBlob.cpp:166] Info: HddlBlob initialize ion ...
+[HDDLPlugin] [07:49:01.7237][6]I[HddlBlob.cpp:176] Info: HddlBlob initialize ion successfully.
+[ INFO ] Start inference (10 asynchronous executions)
+[ INFO ] Completed 1 async request execution
+[ INFO ] Completed 2 async request execution
+[ INFO ] Completed 3 async request execution
+[ INFO ] Completed 4 async request execution
+[ INFO ] Completed 5 async request execution
+[ INFO ] Completed 6 async request execution
+[ INFO ] Completed 7 async request execution
+[ INFO ] Completed 8 async request execution
+[ INFO ] Completed 9 async request execution
+[ INFO ] Completed 10 async request execution
+[ INFO ] Processing output blobs
+
+Top 10 results:
+
+Image /root/car.png
+
+classid probability label
+------- ----------- -----
+817     0.8295898   sports car, sport car
+511     0.0961304   convertible
+479     0.0439453   car wheel
+751     0.0101318   racer, race car, racing car
+436     0.0074234   beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon
+656     0.0042267   minivan
+586     0.0029869   half track
+717     0.0018148   pickup, pickup truck
+864     0.0013924   tow truck, tow car, wrecker
+581     0.0006595   grille, radiator grille
+
+[HDDLPlugin] [07:49:01.9231][11]I[Dispatcher2.cpp:212] Info: Listen Thread wake up and to exit.
+[HDDLPlugin] [07:49:01.9232][6]I[Dispatcher2.cpp:81] Info: Client dispatcher exit.
+[HDDLPlugin] [07:49:01.9235][6]I[HddlClient.cpp:203] Info: Hddl client unregistered.
+[ INFO ] Execution successful
+
+[ INFO ] This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool
+
+
+

If the pod did not successfully launch, possibly because it could not obtain the vpu HDDL +resource, it will be stuck in the Pending status:

+
$ kubectl get pods
+NAME                      READY   STATUS    RESTARTS   AGE
+intelvpu-demo-job-xxxxx   0/1     Pending   0          8s
+
+
+

This can be verified by checking the Events of the pod:

+
$ kubectl describe pod intelvpu-demo-job-xxxxx
+...
+Events:
+Type     Reason            Age        From               Message
+----     ------            ----       ----               -------
+Warning  FailedScheduling  <unknown>  default-scheduler  0/1 nodes are available: 1 Insufficient vpu.intel.com/hddl.
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/cmd/xpumanager_sidecar/README.html b/0.26/cmd/xpumanager_sidecar/README.html new file mode 100644 index 000000000..03f91fe4d --- /dev/null +++ b/0.26/cmd/xpumanager_sidecar/README.html @@ -0,0 +1,202 @@ + + + + + + + XeLink sidecar for Intel XPU Manager — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ + + + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/demo/readme.html b/0.26/demo/readme.html new file mode 100644 index 000000000..5698bcbf8 --- /dev/null +++ b/0.26/demo/readme.html @@ -0,0 +1,381 @@ + + + + + + + Demo — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Demo

+

Table of Contents

+ +
+

Demo overview

+

Acceleration of compute and data processing of workloads like video +transcoding, compression, and encryption is enabled in Kubernetes with +the Device Plugin Framework. This repository +contains a set of Kubernetes plugins and instructions to enable Intel devices +for the acceleration of your workloads orchestrated by Kubernetes.

+

The current list of supported Intel Device Plugins includes:

+ +

We have included an example demo and configuration information for the Intel +Device Plugins for Kubernetes below. Please join us on the sig-node-rtk channel +on kubernetes.slack.com to ask questions, +contribute to the community, and learn about the work we are doing with +Kubernetes and the Device Plugin Framework.

+
+
+

Intel® GPU Device Plugin demo video

+

The screencast demonstrates the deployment of the Intel® GPU Device Plugin for +Kubernetes including Kubeless Function as a Service (FaaS) media transcoder +JavaScript function. The media transcoding workload is scheduled on two different worker nodes. +Only one worker node has a GPU. The time difference in transcoding speed is captured.

+
+

Demo platform configuration

+
    +

  • Hardware 2-nodes

    +
      +

    • 1x Virtual Machine on Intel® Xeon® E5-2687 CPU @ 3.0 GHz

      • +

    • 1x Intel® NUC KIT NUC6i7KYK (Skull Canyon) with Intel integrated GPU

      • +
    +
    • +

  • Software

    +
      +

    • Ubuntu* 18.04 (Kernel: 4.15.0-36-generic)

      • +

    • Kubernetes* 1.11

      • +

    • Docker* 18.3.1

      • +

    • Intel® GPU Device Plugin built from main branch

      • +
    +
    • +
+
+
+

Screencast

+

Intel® GPU Device Plugin demo

+
+
+
+

Intel® FPGA Device Plugin demo videos

+

The screencasts demonstrate the deployment of the Intel® FPGA Device Plugin for +Kubernetes.

+
    +

  • Demo 1 executes a native loopback 3 (NLB3) workload in preprogrammed mode

    • +

  • Demo 2 executes NLB3 workload in orchestrated mode

    • +

  • Demo 3 runs an OpenCL workload to do English letters recognition, and compares time used with and without FPGA to show the acceleration.

    • +
+

The demos begin with a fully configured Kubernetes cluster.

+
+

Demo 1 and 2 (NLB3) platform configuration

+
    +

  • Hardware

    +
      +

    • 1-node, Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz

      • +

    • Total memory 62 GB

      • +

    • 2 x Intel® Arria® 10 GX FPGA Programmable Acceleration Card(PAC)

      • +
    +
    • +

  • Software

    +
      +

    • openSUSE Leap 15.1 (Kernel: 5.4.5-1.g47eef04-default)

      • +

    • Kubernetes* 1.17

      • +

    • CRI-O 1.13.1 (for orchestrated mode)

      • +

    • Docker 19.03.1 (for preprogrammed mode)

      • +

    • runc 1.0.0-rc8

      • +

    • Intel® FPGA Device Plugin built from main branch

      • +
    +
    • +
+
+
+

Demo 3 (OpenCL) platform configuration

+
    +

  • Hardware

    +
      +

    • Multi-node, FPGA host has 24 cores Intel(R) Xeon(R) Gold 6252N CPU @ 2.30GHz

      • +

    • Total memory 195 GB

      • +

    • Intel® FPGA Programmable Acceleration Card D5005

      • +
    +
    • +

  • Software

    +
      +

    • Centos 7 (Kernel: 3.10.0-1062.12.1.el7.x86_64) on worker node host

      • +

    • Kubernetes* 1.17

      • +

    • CRI-O 1.17.3

      • +

    • Intel® FPGA Device Plugin built from main branch

      • +
    +
    • +
+
+
+

Demo steps

+
    +

  1. Validate the status of the Kubernetes cluster.

    2. +

  2. Provision the Intel® FPGA Device Plugin.

    3. +

  3. Create bitstream storage (for orchestrated mode only)

    4. +

  4. Run the NLB3 or OpenCL workload.

    5. +
+
+
+

Screencasts

+
    +

  • Intel® FPGA Device Plugin deployment in preprogrammed mode and NLB workload:

    • +
+

+
    +

  • Intel® FPGA Device Plugin deployment with orchestrated/region mode and NLB workload:

    • +
+

+
    +

  • Intel® FPGA Device Plugin deployment with orchestrated/region mode and OpenCL workload:

    • +
+

+
+
+
+

Intel® SGX Device Plugin demo videos

+

This video demonstrates the Intel® Software Guard Extensions ECDSA Quote Generation in Kubernetes*

+
    +

  • Hardware

    +
      +

    • 1-node, 3rd Generation Intel® Xeon® Scalable Platform, code-named “Ice Lake”

      • +
    +
    • +

  • Software

    +
      +

    • 18.04.5 LTS

      • +

    • Kubernetes* 1.19

      • +

    • containerd 1.3.3

      • +

    • Intel® SGX Device Plugin built from main branch

      • +

    • Intel® SGX SDK 2.12 and DCAP 1.9

      • +

    • node-feature-discovery 0.6.0

      • +
    +
    • +
+
+

Demo steps

+
    +

  1. Validate the status of the Kubernetes cluster.

    2. +

  2. Provision node-feature-discovery.

    3. +

  3. Provision the Intel® SGX Device Plugin using Intel® Device Plugin Operator.

    4. +

  4. Check the SGX resources and labels are correctly registered.

    5. +

  5. Run Intel® SGX DCAP ECDSA Quote Generation in both “out-of-proc” and “in-proc” modes.

    6. +
+
+
+

Screencasts

+

Intel® SGX Device Plugin and SGX DCAP ECDSA Quote Generation demo +

+
+
+
+

Intel® QuickAssist Technology Device Plugin OpenSSL demo video

+

The screencast demonstrates the deployment of the Intel® QAT Device Plugin for +Kubernetes and executes a sample QAT accelerated OpenSSL workload with the OCI +compatible Kata Containers runtime, a lightweight Virtual Machine (VM) that feels +and performs like traditional containers, but provides the workload isolation +and security advantages of VMs. The demo begins with a fully configured Kubernetes cluster +and Kata Containers runtime for workloads.

+
+

Demo platform configuration

+
    +

  • Hardware

    +
      +

    • 1-node, 2x Intel® Xeon® CPU E5-2687W v4 @ 3.00GHz

      • +

    • Total memory 251 GB DDR4

      • +

    • Intel® QAT C62x chipset

      • +
    +
    • +

  • Software

    +
      +

    • OpenSUSE* 15 (Kernel:4.12.14-lp150.12.22-default)

      • +

    • Kubernetes* 1.12

      • +

    • Containerd 1.2

      • +

    • Kata Containers* 1.3.0

      • +

    • Intel® QAT Device Plugin built from main

      • +

    • QAT 1.7 L.4.3.0-00033

      • +
    +
    • +
+
+
+

Demo steps

+
    +

  1. Load the host drivers and prepare the virtual function (VF) devices.

    2. +

  2. Check the Kubernetes cluster is in good shape.

    3. +

  3. Deploy the Intel® QAT device plugin for Kubernetes.

    4. +

  4. Deploy an Intel® QAT Accelerated OpenSSL workload.

    5. +

  5. Testing!

    6. +
+
+
+

Screencast

+

Intel® QAT Device Plugin deployment

+

+
+
+
+

Intel® QuickAssist Technology Device Plugin with DPDK demo video

+
+

Demo steps

+
    +

  1. Check health of Kubernetes nodes.

    2. +

  2. Check for allocatable resources.

    3. +

  3. List QAT Virtual Functions.

    4. +

  4. Deploy QAT Device Plugin as a Daemonset.

    5. +

  5. Check again for allocatable resources.

    6. +

  6. List QAT Virtual Functions again, ensuring they are now bound to DPDK driver.

    7. +

  7. View pod specification file for pod requesting QAT VFs.

    8. +

  8. Create pod requesting QAT VFs.

    9. +

  9. Get a shell to the running container and run a DPDK application.

    10. +
+
+
+

Screencast

+

Intel® QAT Device Plugin with DPDK:

+

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/docs/extensions.html b/0.26/docs/extensions.html new file mode 100644 index 000000000..011c4f13a --- /dev/null +++ b/0.26/docs/extensions.html @@ -0,0 +1,349 @@ + + + + + + + Extensions — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Extensions

+
+ +
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/genindex.html b/0.26/genindex.html new file mode 100644 index 000000000..143e5b2bc --- /dev/null +++ b/0.26/genindex.html @@ -0,0 +1,109 @@ + + + + + + Index — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+
    +
    • +
  • Index
    • +
  • +
    • +
+
+
+
+
+ + +

Index

+ +
+ +
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.26/index.html b/0.26/index.html new file mode 100644 index 000000000..5f62e3d9b --- /dev/null +++ b/0.26/index.html @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/0.26/objects.inv b/0.26/objects.inv new file mode 100644 index 000000000..d8d8ac525 Binary files /dev/null and b/0.26/objects.inv differ diff --git a/0.26/search.html b/0.26/search.html new file mode 100644 index 000000000..533020b24 --- /dev/null +++ b/0.26/search.html @@ -0,0 +1,124 @@ + + + + + + Search — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+
    +
    • +
  • Search
    • +
  • +
    • +
+
+
+
+
+ + + + +
+ +
+ +
+
+ +
+
+
+
+ + + + + + + + + \ No newline at end of file diff --git a/0.26/searchindex.js b/0.26/searchindex.js new file mode 100644 index 000000000..c67a0c7e0 --- /dev/null +++ b/0.26/searchindex.js @@ -0,0 +1 @@ +Search.setIndex({"docnames": ["DEVEL", "README", "SECURITY", "cmd/dlb_plugin/README", "cmd/dsa_plugin/README", "cmd/fpga_admissionwebhook/README", "cmd/fpga_crihook/README", "cmd/fpga_plugin/README", "cmd/fpga_tool/README", "cmd/gpu_fakedev/README", "cmd/gpu_nfdhook/README", "cmd/gpu_plugin/README", "cmd/iaa_plugin/README", "cmd/operator/README", "cmd/operator/ocp_quickstart_guide/README", "cmd/qat_plugin/README", "cmd/sgx_admissionwebhook/README", "cmd/sgx_plugin/README", "cmd/vpu_plugin/README", "cmd/xpumanager_sidecar/README", "demo/readme", "docs/extensions", "index"], "filenames": ["DEVEL.md", "README.md", "SECURITY.md", "cmd/dlb_plugin/README.md", "cmd/dsa_plugin/README.md", "cmd/fpga_admissionwebhook/README.md", "cmd/fpga_crihook/README.md", "cmd/fpga_plugin/README.md", "cmd/fpga_tool/README.md", "cmd/gpu_fakedev/README.md", "cmd/gpu_nfdhook/README.md", "cmd/gpu_plugin/README.md", "cmd/iaa_plugin/README.md", "cmd/operator/README.md", "cmd/operator/ocp_quickstart_guide/README.md", "cmd/qat_plugin/README.md", "cmd/sgx_admissionwebhook/README.md", "cmd/sgx_plugin/README.md", "cmd/vpu_plugin/README.md", "cmd/xpumanager_sidecar/README.md", "demo/readme.md", "docs/extensions.rst", "index.rst"], "titles": ["Instructions for Device Plugin Development and Maintenance", "Overview", "<no title>", "Intel DLB device plugin for Kubernetes", "Intel DSA device plugin for Kubernetes", "Intel FPGA admission controller for Kubernetes", "Intel FPGA prestart CRI-O webhook for Kubernetes", "Intel FPGA device plugin for Kubernetes", "Intel FPGA test tool", "Fake (GPU) device file generator", "Intel GPU NFD hook", "Intel GPU device plugin for Kubernetes", "Intel IAA device plugin for Kubernetes", "Intel Device Plugins Operator", "Intel\u00ae Device Plugins Operator for Red Hat OpenShift Container Platform", "Intel QuickAssist Technology (QAT) device plugin for Kubernetes", "Intel SGX admission controller for Kubernetes", "Intel Software Guard Extensions (SGX) device plugin for Kubernetes", "Intel VPU device plugin for Kubernetes", "XeLink sidecar for Intel XPU Manager", "Demo", "Extensions", "Intel\u00ae Device Plugins for Kubernetes"], "terms": {"tabl": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19, 20], "content": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19, 20], "With": [0, 10, 11, 15, 17], "git": [0, 11], "instal": [0, 1, 6, 8, 21, 22], "system": [0, 4, 5, 7, 10, 11, 12, 13, 15, 16, 17], "just": [0, 8, 9, 11, 14, 15], "clone": 0, "repositori": [0, 1, 3, 4, 7, 11, 12, 15, 17, 18, 19, 20, 22], "export": [0, 5, 9, 13, 16, 18], "intel_device_plugins_src": [0, 3, 4, 11, 17], "path": [0, 5, 6, 8, 10, 17], "http": [0, 1, 3, 4, 5, 7, 11, 12, 13, 14, 15, 16, 17, 18, 19], "github": [0, 1, 3, 4, 5, 7, 11, 12, 13, 15, 16, 17, 18, 19, 22], "com": [0, 1, 2, 3, 4, 5, 7, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], "go": [0, 1, 3, 4, 12, 13, 14, 18], "environ": [0, 1, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16], "cd": [0, 3, 17, 18], "make": [0, 3, 4, 5, 11, 12, 13, 14, 16, 17, 18, 19], "target": [0, 11], "note": [0, 1, 3, 5, 6, 7, 11, 13, 14, 15, 16, 17, 18, 19], "all": [0, 1, 5, 6, 7, 10, 13, 15, 17, 21, 22], "avail": [0, 1, 3, 4, 5, 7, 11, 12, 13, 15, 17, 18, 19], "i": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], "roughli": 0, "output": [0, 3, 5, 7, 11, 13, 16, 18], "l": [0, 3, 15, 20], "cmd": 0, "To": [0, 1, 4, 5, 7, 11, 12, 13, 14, 15, 16, 17, 18], "administr": [0, 5, 13, 14], "sudo": [0, 3], "e": [0, 1, 2, 5, 9, 10, 11, 13, 17, 19], "The": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], "dockerfil": [0, 3, 18], "ar": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 17, 18, 19, 20], "gener": [0, 1, 5, 6, 7, 10, 14, 15, 17, 18, 20], "fly": 0, "from": [0, 1, 3, 4, 5, 7, 10, 11, 12, 13, 15, 16, 17, 18, 19, 20], "suffix": 0, "file": [0, 1, 3, 4, 5, 6, 8, 10, 11, 12, 15, 17, 18, 19, 20], "docker": [0, 1, 3, 4, 7, 11, 12, 15, 17, 19, 20, 21, 22], "includ": [0, 1, 2, 4, 9, 11, 12, 15, 20], "snippet": 0, "which": [0, 1, 4, 5, 7, 10, 11, 12, 15], "stitch": 0, "togeth": [0, 5, 6, 7, 19], "cpp": [0, 18], "preprocessor": 0, "you": [0, 2, 3, 4, 5, 6, 7, 10, 11, 12, 13, 15, 16, 17, 18, 19], "need": [0, 1, 3, 4, 7, 9, 10, 11, 12, 15, 17, 18], "g": [0, 1, 3, 5, 9, 10, 11, 13, 17], "ubuntu": [0, 3, 18, 20], "found": [0, 1, 5, 7, 10, 13, 18], "essenti": [0, 3, 5], "apt": [0, 3], "don": [0, 8], "t": [0, 8, 11], "edit": [0, 14], "input": [0, 18], "simplest": [0, 11], "wai": [0, 1, 3, 4, 5, 11, 12, 15, 17, 18], "But": [0, 9], "veri": [0, 5, 13, 16], "slow": [0, 11], "can": [0, 1, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], "drastic": 0, "speed": [0, 20], "up": [0, 5, 10, 13, 17, 18], "first": [0, 3, 5, 7, 9, 11, 17, 19], "onc": [0, 14, 15], "vendor": [0, 14], "bring": 0, "librari": [0, 11], "builder": 0, "without": [0, 9, 11, 13, 20], "download": 0, "them": [0, 4, 5, 7, 9, 12, 13, 15], "again": [0, 20], "each": [0, 1, 9, 10, 11, 15], "still": [0, 7, 13], "further": 0, "licens": [0, 22], "pre": [0, 21, 22], "creat": [0, 3, 4, 5, 7, 10, 11, 12, 13, 14, 15, 17, 20, 21, 22], "instead": [0, 9, 10, 11], "re": 0, "built": [0, 10, 20, 21, 22], "everi": 0, "time": [0, 13, 20], "rather": 0, "unnecessari": 0, "iter": 0, "one": [0, 5, 6, 7, 9, 11, 13, 15, 16, 20], "exampl": [0, 3, 4, 5, 7, 9, 10, 11, 12, 13, 15, 16, 17, 19, 20, 21, 22], "If": [0, 2, 3, 4, 6, 7, 9, 10, 11, 12, 15, 17, 18, 19], "onli": [0, 3, 6, 7, 8, 9, 10, 11, 14, 15, 20], "know": 0, "what": [0, 9, 15], "its": [0, 1, 5, 7, 9, 13, 16, 17, 19], "see": [0, 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 13, 15, 16, 17, 18, 19], "folder": [0, 10], "opt": 0, "name": [0, 1, 3, 4, 5, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], "makefil": 0, "deriv": 0, "filenam": 0, "under": [0, 1, 3, 7, 10, 13, 15, 19], "templat": [0, 3, 4, 12], "recap": 0, "repeat": [0, 3], "last": [0, 5], "step": [0, 14, 17, 21, 22], "unless": [0, 11], "chang": [0, 1, 9, 11], "depend": [0, 3, 7, 10, 11, 13, 15, 16, 17, 21, 22], "pull": [0, 1, 17], "start": [0, 5, 10, 11, 15, 18], "tool": [0, 7, 15, 18, 21, 22], "default": [0, 1, 3, 4, 5, 6, 7, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21, 22], "set": [0, 1, 5, 6, 7, 11, 13, 14, 15, 16, 17, 20], "argument": [0, 8, 11, 13, 15, 17, 19], "support": [0, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 22], "valu": [0, 5, 10, 11, 13, 15, 16], "buildah": 0, "podman": 0, "updat": [0, 3, 9], "modul": [0, 1, 3, 15], "easiest": [0, 3, 4, 11, 12, 15, 17, 18], "us": [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 18, 19, 20, 21, 22], "script": [0, 1, 4, 11, 12, 15, 18], "upgrade_k8": 0, "sh": [0, 1, 11, 18], "copi": [0, 10], "k": [0, 3, 4, 5, 7, 11, 12, 13, 15, 16, 17, 18, 19], "issu": [0, 2, 21, 22], "insid": [0, 10, 15, 17], "repo": [0, 1, 7], "root": [0, 10, 11, 15, 18], "k8": [0, 5, 7, 9, 11, 17], "final": [0, 13, 17], "fix": 0, "compil": 0, "There": [0, 4, 10, 11, 12, 15, 17], "few": 0, "when": [0, 6, 7, 10, 11, 13, 15], "crd": [0, 1, 5, 13], "gen": [0, 18], "go111modul": 0, "u": [0, 1, 20], "sig": [0, 20], "releas": [0, 1, 3, 4, 7, 8, 11, 12, 13, 14, 15, 17, 18, 19], "ver": 0, "v0": [0, 1, 13, 14], "4": [0, 3, 4, 10, 12, 14, 17, 18, 19, 20], "1": [0, 1, 3, 4, 5, 7, 10, 11, 12, 13, 15, 16, 17, 18, 19, 22], "webhook": [0, 7, 13, 16, 17, 21, 22], "artifact": 0, "envtest": 0, "custom": [0, 1, 3, 4, 5, 6, 11, 12, 13, 14, 15, 18, 19], "deviceplugin": 0, "un": [0, 6, 7], "deploi": [0, 3, 4, 5, 6, 7, 10, 11, 12, 13, 16, 18, 19, 20, 21, 22], "kubectl": [0, 3, 4, 5, 7, 11, 12, 13, 15, 16, 18, 19, 21, 22], "appli": [0, 3, 4, 5, 7, 11, 12, 13, 14, 15, 16, 17, 18, 19], "delet": 0, "deploy": [0, 1, 3, 4, 7, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22], "check": [0, 3, 4, 5, 11, 12, 13, 14, 16, 18, 19, 20, 21, 22], "field": [0, 5, 15], "mention": 0, "below": [0, 1, 3, 7, 11, 14, 15, 17, 18, 20], "base": [0, 1, 3, 5, 6, 11, 15, 16, 17], "csv": 0, "manifest": [0, 5, 13, 16], "have": [0, 2, 5, 6, 7, 9, 10, 11, 13, 15, 16, 17, 20], "correct": [0, 11, 14], "manual": [0, 17, 21, 22], "sdk": [0, 1, 11, 17, 20], "doe": [0, 11, 15, 17], "ani": [0, 5, 14, 18], "other": [0, 3, 7, 9, 10, 11, 13, 14, 15, 17, 19], "spec": [0, 1, 5, 9, 11, 13, 16], "replac": [0, 3, 11, 19], "metadata": [0, 3, 4, 5, 11, 12, 13, 16, 18, 19], "annot": [0, 1, 6, 7, 11, 16], "containerimag": 0, "createdat": 0, "fork": 0, "commun": [0, 1, 15, 20], "usernam": 0, "bundl": 0, "tag": [0, 1, 3, 4, 7, 11, 12, 13, 15, 17, 18, 19], "0": [0, 1, 3, 4, 5, 7, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], "x": [0, 1, 3, 4, 7, 11, 12, 15, 17, 18, 19, 20], "y": [0, 1, 3, 4, 7, 11, 12, 15, 17, 18, 19], "channel": [0, 20], "alpha": 0, "default_channel": 0, "push": [0, 11], "registri": [0, 1, 11, 13], "hub": [0, 1, 3, 4, 7, 11, 12, 15, 17, 18, 19], "specifi": [0, 1, 11], "front": 0, "put": 0, "option": [0, 4, 8, 9, 12, 13, 21, 22], "verifi": [0, 21, 22], "ok": 0, "via": [0, 1, 4, 7, 11, 12, 15, 19], "olm": 0, "your": [0, 2, 3, 4, 5, 6, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], "cluster": [0, 1, 3, 4, 5, 7, 11, 12, 13, 14, 15, 16, 17, 18, 20], "namespac": [0, 1, 4, 5, 7, 12, 13, 15, 16, 19], "testoper": 0, "n": [0, 3, 4, 7, 11, 12, 13, 18, 19], "do": [0, 3, 10, 11, 15, 17, 18, 20], "verif": [0, 15, 17, 21, 22], "clean": 0, "cleanup": 0, "uninstal": 0, "commit": 0, "add": [0, 1, 5, 6, 7, 10, 11, 15, 16, 19], "am": 0, "submit": 0, "pr": [0, 8], "page": [0, 1, 16], "after": [0, 3, 15], "merg": 0, "current": [0, 1, 7, 9, 11, 13, 20], "requir": [0, 1, 3, 5, 6, 7, 10, 11, 13, 15, 17], "alreadi": [0, 7, 11, 13, 14], "configur": [0, 1, 5, 13, 14, 16, 18, 21, 22], "node": [0, 1, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22], "hardwar": [0, 1, 7, 11, 17, 20, 21, 22], "also": [0, 1, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19], "execut": [0, 10, 15, 18, 20], "must": [0, 1, 7, 10, 11, 15, 17, 18], "two": [0, 7, 10, 11, 15, 17, 20], "condit": 0, "satisfi": 0, "v": [0, 3, 4, 12], "In": [0, 5, 10, 11, 13, 14, 15, 16, 19], "case": [0, 5, 11, 13, 16], "want": [0, 3, 4, 5, 6, 11, 12, 18, 19], "certain": 0, "qat": [0, 13, 14, 20, 21, 22], "ones": 0, "arg": 0, "ginkgo": 0, "focu": 0, "kubeconfig": 0, "embed": 0, "authent": 0, "info": [0, 8, 18, 19], "full": [0, 5, 11, 13, 15], "list": [0, 1, 3, 8, 9, 10, 11, 13, 15, 17, 18, 19, 20], "obtain": [0, 3, 4, 5, 7, 11, 12, 15, 16, 18, 19], "help": [0, 4, 12], "It": [0, 1, 6, 11, 13, 14, 15, 17, 18], "possibl": [0, 2, 4, 11, 12, 15, 18], "sure": [0, 5, 13, 14, 16], "kind": [0, 5, 13, 16, 17], "host": [0, 5, 9, 10, 11, 13, 15, 16, 17, 20], "runtim": [0, 7, 11, 20], "provid": [0, 1, 2, 3, 4, 5, 9, 11, 12, 13, 14, 15, 16, 18, 19, 20], "packag": [0, 21, 22], "integr": [0, 1, 5, 7, 15, 16, 20, 21, 22], "call": [0, 17], "thi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20], "For": [0, 1, 2, 3, 5, 6, 9, 10, 11, 13, 15, 17, 18], "setup": [0, 3, 5, 18, 21, 22], "latest": [0, 1, 3, 4, 7, 11, 12, 15, 17, 18, 19], "k8s_version": 0, "kubebuilder_asset": 0, "p": 0, "hassl": 0, "own": [0, 1, 5, 9, 11, 15, 17], "grpc": [0, 11], "server": [0, 5, 11, 13, 16, 17, 18], "pkg": 0, "instanti": 0, "manag": [0, 5, 7, 10, 11, 13, 15, 16, 17, 22], "method": [0, 15], "func": 0, "main": [0, 1, 3, 4, 5, 7, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], "dpapi": 0, "newmanag": 0, "constructor": 0, "accept": [0, 11, 15, 17, 19], "paramet": [0, 6, 14, 15, 18], "string": [0, 8, 10, 11, 15, 19], "like": [0, 5, 9, 11, 20], "color": 0, "expos": [0, 11], "space": [0, 21, 22], "yellow": 0, "pleas": [0, 2, 6, 11, 14, 15, 17, 18, 19, 20], "regist": [0, 1, 3, 4, 11, 12, 13, 17, 18, 20], "mani": [0, 10], "multipl": [0, 10, 11, 13], "refer": [0, 1, 5, 6, 14, 15, 17, 18], "an": [0, 1, 2, 4, 5, 7, 8, 10, 11, 12, 15, 17, 20, 21, 22], "object": [0, 5, 11], "implement": [0, 1, 7, 10, 15], "mandatori": [0, 5], "interfac": [0, 1, 5, 7, 18], "scanner": 0, "defin": 0, "scan": 0, "goroutin": 0, "infinit": 0, "loop": 0, "A": [0, 1, 2, 7, 18, 19], "send": [0, 2, 5, 13, 16], "notifi": 0, "instanc": [0, 14, 15], "itself": [0, 10, 13], "organ": 0, "devicetre": 0, "fill": 0, "adddevic": 0, "dp": 0, "devtre": 0, "newdevicetre": 0, "devid": 0, "deviceinfo": 0, "state": 0, "health": [0, 20], "pluginapi": 0, "devicespec": 0, "hostpath": [0, 17], "devpath": 0, "containerpath": [0, 17], "permiss": [0, 11], "rw": 0, "mai": [0, 3, 5, 6, 7, 10, 13, 15, 17], "postalloc": 0, "modifi": [0, 7, 11], "allocaterespons": 0, "respons": [0, 1, 5, 16, 17], "befor": [0, 3, 5, 7, 13, 14, 15, 16, 19], "thei": [0, 5, 9, 11, 20], "sent": 0, "kubelet": [0, 1, 7, 11, 14, 15, 18], "fpga": [0, 13, 21, 22], "whole": 0, "alloc": [0, 3, 4, 7, 11, 12, 15, 17, 18], "function": [0, 1, 3, 5, 6, 7, 9, 11, 15, 16, 20], "decid": 0, "resort": 0, "return": 0, "type": [0, 1, 3, 4, 10, 11, 12, 13, 15, 18], "usedefaultmethoderror": 0, "framework": [0, 1, 5, 6, 7, 15, 17, 20], "klog": [0, 15], "encourag": 0, "maintain": 0, "uniform": 0, "command": [0, 3, 4, 5, 7, 11, 12, 13, 14, 15, 16, 17, 18, 21, 22], "line": [0, 11, 13, 15, 17, 21, 22], "initialis": 0, "so": [0, 9, 11, 15], "initflag": 0, "should": [0, 1, 5, 7, 11, 13], "necessari": [0, 1, 10, 13, 15, 16, 17], "number": [0, 1, 3, 4, 7, 9, 10, 11, 12, 13, 15, 17, 18, 19], "view": [0, 14, 20], "h": [0, 11, 15, 17, 19], "tri": 0, "adher": 0, "advis": 0, "level": [0, 11], "filter": 0, "more": [0, 1, 2, 5, 6, 10, 11, 13, 15, 17], "difficult": 0, "addit": [0, 11, 15, 19], "prepend": 0, "disabl": [0, 11], "skip_head": 0, "ha": [0, 1, 3, 4, 7, 8, 10, 11, 12, 15, 17, 18, 20], "produc": [0, 10], "ideal": 0, "within": [0, 7], "errorf": 0, "messag": [0, 3, 4, 11, 12, 18], "outsid": 0, "augment": 0, "stack": 0, "dump": 0, "withstack": 0, "err": 0, "wrap": 0, "some": [0, 7, 11, 19], "These": [0, 1, 3, 4, 7, 11, 12, 15, 17, 18, 19], "struct": 0, "format": [0, 1, 3, 4, 5, 11, 12, 15, 17, 18, 19], "intern": [0, 5], "death": 0, "where": [0, 3, 4, 5, 7, 10, 11, 12, 15, 17, 18], "cannot": 0, "pass": [0, 4, 12, 13, 15], "out": [0, 7, 10, 17, 20], "farther": 0, "nor": 0, "handl": [0, 2], "gracefulli": 0, "otherwis": [0, 6, 13], "warningf": 0, "warn": [0, 3, 4, 11, 12, 18], "due": [0, 10], "extern": [0, 1], "contribut": [0, 20], "par": 0, "featur": [0, 1, 7, 9, 10, 11, 13, 14, 15, 17, 19, 20], "qualiti": 0, "wise": 0, "correspond": [0, 1, 3, 4, 5, 9, 10, 11, 12, 13, 15, 17, 18, 19], "kustom": [0, 11, 13, 15, 17, 19], "yaml": [0, 1, 3, 4, 5, 6, 7, 11, 12, 13, 14, 16, 17, 18, 19], "unit": [0, 3, 4, 12, 21, 22], "80": 0, "coverag": 0, "linter": 0, "lint": 0, "api": [0, 5, 13, 16, 17, 18, 19, 21, 22], "ad": [0, 9, 11, 15, 18], "v1": [0, 1, 5, 13, 16, 17], "valid": [0, 8, 11, 15, 20], "document": [0, 1, 7, 11, 15, 17], "written": 0, "readm": [0, 1, 17], "md": 0, "end": [0, 3, 4, 10, 11, 12, 19], "demo": [0, 21, 22], "contain": [1, 5, 6, 7, 8, 9, 11, 13, 15, 16, 17, 18, 19, 20, 22], "along": 1, "util": [1, 4, 12, 15], "26": 1, "here": [1, 5, 7, 11, 13, 16, 17], "build": [1, 3, 4, 5, 6, 7, 11, 12, 13, 15, 16, 21, 22], "run": [1, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 20, 21, 22], "appropri": [1, 7, 11, 15, 18], "driver": [1, 7, 10, 15, 17, 20, 21, 22], "fulli": [1, 7, 20], "work": [1, 2, 3, 4, 5, 6, 7, 10, 11, 12, 14, 15, 17, 18, 20, 22], "least": 1, "16": [1, 15], "section": [1, 3, 4, 5, 7, 11, 12, 15, 16, 17, 18, 19], "detail": [1, 2, 3, 4, 5, 6, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22], "exist": [1, 7], "access": [1, 5, 7, 10, 11, 15], "discret": [1, 21, 22], "subdirectori": [1, 9], "both": [1, 10, 11, 13, 15, 20], "video": [1, 11, 22], "opencl": [1, 11, 22], "sampl": [1, 4, 12, 13, 15, 17, 18, 20], "intelgpu": [1, 11], "job": [1, 3, 4, 11, 17, 21, 22], "passthrough": 1, "follow": [1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], "arria": [1, 7, 20], "10": [1, 3, 4, 5, 7, 12, 13, 14, 15, 16, 18, 19, 20], "stratix": [1, 7], "come": [1, 10, 17], "three": [1, 5, 7, 17], "part": [1, 5, 7, 10, 17], "crio": [1, 6], "individu": [1, 11], "sub": [1, 4], "compon": [1, 3, 4, 5, 6, 11, 12, 15, 16, 17, 18, 19, 21, 22], "brief": 1, "show": [1, 3, 7, 11, 20], "sourc": [1, 3, 22], "relat": [1, 11, 15, 17], "opa": [1, 7], "nlb": [1, 20], "discov": [1, 2, 4, 6, 7, 12], "report": [1, 2, 8, 9, 15], "perform": [1, 5, 6, 11, 13, 16, 18, 19, 20], "map": [1, 7, 9, 10, 11, 21, 22], "user": [1, 3, 4, 5, 12, 14, 15, 16, 18, 21, 22], "friendli": [1, 5], "id": [1, 5, 10, 11, 18], "bitstream": [1, 5, 6, 7, 8, 20], "program": [1, 5, 6, 7, 8], "inform": [1, 2, 10, 15, 19, 20], "discoveri": [1, 6, 7, 10, 11, 13, 14, 17, 19, 20], "request": [1, 5, 6, 7, 11, 13, 15, 16, 18, 20], "variabl": [1, 5, 6, 7, 10, 11, 13, 16], "descript": [1, 2, 6, 10, 18], "adapt": [1, 15], "code": [1, 13, 14, 17, 20, 22], "dpdk": [1, 21, 22], "openssl": [1, 21, 22], "relev": [1, 3, 4, 11, 12, 15, 18], "kata": [1, 15, 16, 20], "vcac": [1, 18], "card": [1, 10, 11, 18, 20], "www": [1, 11], "dam": 1, "public": [1, 2], "en": 1, "datasheet": 1, "media": [1, 10, 20, 21, 22], "analyt": [1, 11, 12, 14], "acceler": [1, 4, 5, 7, 12, 14, 15, 20], "celestica": 1, "pdf": [1, 3], "core": [1, 18, 20], "i3": [1, 18], "7100u": [1, 18], "processor": [1, 15, 18], "12": [1, 4, 18, 20], "myriadx": [1, 18], "8gb": [1, 18], "ddr4": [1, 18, 20], "openvino": [1, 18], "allow": [1, 10, 15, 17], "softwar": [1, 3, 14, 15, 20, 21, 22], "guard": [1, 14, 20, 21, 22], "extens": [1, 14, 20, 22], "platform": [1, 7, 10, 17, 22], "flexibl": [1, 5, 17], "launch": [1, 3, 4, 7, 11, 12, 17, 18], "enabl": [1, 3, 6, 7, 11, 13, 15, 17, 18, 20], "3rd": [1, 17, 20], "xeon": [1, 14, 15, 17, 18, 20], "scalabl": [1, 9, 14, 17, 20], "famili": 1, "ic": [1, 14, 17, 20], "lake": [1, 14, 17, 20], "e3": [1, 17, 18], "nuc": [1, 17, 20], "kit": [1, 15, 17, 20], "nuc7cjyh": [1, 17], "aesmd": [1, 17], "given": [1, 10, 11], "resourc": [1, 3, 4, 5, 7, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22], "directli": [1, 11], "pod": [1, 3, 4, 5, 7, 9, 10, 11, 12, 13, 16, 18, 19, 20, 21, 22], "mutat": [1, 5, 7, 16], "quot": [1, 16, 17, 20], "purpos": [1, 16], "hide": [1, 16], "volum": [1, 4, 12, 15, 16], "mount": [1, 4, 10, 12, 15, 16, 17], "remot": [1, 16, 21, 22], "attest": [1, 16, 21, 22], "furthermor": [1, 11, 15, 16, 17, 18], "write": [1, 8, 10, 16, 19], "sandbox": [1, 16], "dynam": [1, 7, 16], "adjust": [1, 16], "virtual": [1, 3, 15, 16, 20], "encrypt": [1, 2, 15, 16, 20], "cach": [1, 11, 16, 17], "bank": [1, 16], "": [1, 3, 4, 5, 7, 9, 11, 12, 13, 14, 15, 16, 18, 19, 22], "size": [1, 16, 18], "standalon": 1, "extend": [1, 10, 11, 17], "nfd": [1, 9, 13, 14, 17, 19, 21, 22], "data": [1, 4, 11, 14, 15, 20], "stream": [1, 4, 14], "load": [1, 3, 5, 7, 15, 18, 20], "balanc": [1, 11, 15], "simplifi": 1, "unifi": 1, "definit": [1, 5], "watch": 1, "crud": 1, "those": [1, 7, 11, 15, 19], "give": [1, 5, 11, 13], "usag": [1, 11, 21, 22], "operatorhub": [1, 14, 22], "io": [1, 5, 7, 11, 13, 17, 22], "ocp": [1, 14], "red": 1, "hat": 1, "openshift": 1, "interconnect": [1, 19], "how": [1, 2, 3, 4, 5, 7, 11, 12, 15, 16, 17, 18, 19, 22], "demonstr": [1, 15, 20], "varieti": 1, "get": [1, 3, 4, 5, 7, 10, 11, 12, 13, 15, 17, 18, 19, 20, 22], "accesss": 1, "limit": [1, 11, 13, 15, 17, 21, 22], "summari": 1, "pf": [1, 4, 15, 21, 22], "vf": [1, 15, 20, 21, 22], "libdlb": [1, 3], "wq": [1, 4, 12], "share": [1, 4, 10, 12, 21, 22], "dedic": [1, 4, 12, 18], "accel": [1, 4, 12], "config": [1, 4, 9, 12, 15, 17], "intelfpga": 1, "i915": [1, 11], "qpl": [1, 12], "cy": 1, "dc": [1, 15], "crypto": [1, 15], "perf": [1, 15], "intelsgx": [1, 17], "hddl": [1, 18], "intelvpu": [1, 18], "new": [1, 3, 9, 11, 13, 22], "task": [1, 4], "guid": [1, 3, 4, 6, 11, 12, 15, 17, 18, 19], "made": 1, "area": 1, "match": [1, 7, 9, 10, 11, 17], "branch": [1, 3, 4, 7, 11, 12, 15, 17, 18, 19, 20], "statu": [1, 3, 4, 7, 11, 12, 13, 14, 15, 17, 18, 20], "25": 1, "24": [1, 14, 20], "23": 1, "unsupport": 1, "22": [1, 20], "21": [1, 3], "20": [1, 4, 17], "19": [1, 20], "18": [1, 20], "17": [1, 20], "15": [1, 4, 20], "11": [1, 3, 4, 14, 17, 18, 20], "automat": [1, 3, 6, 7, 11, 17, 18, 19, 21, 22], "upload": [1, 3, 4, 7, 11, 12, 15, 17, 18, 19], "z": [1, 3, 4, 7, 11, 12, 15, 17, 18, 19], "imagepullpolici": 1, "ifnotpres": 1, "polici": [1, 11, 14, 15], "open": [1, 5, 7, 13, 14, 16], "identifi": [1, 15], "binari": [1, 10, 22], "distribut": [1, 3, 11], "dockerhub": 1, "text": 1, "locat": [1, 7, 8, 11], "anoth": [1, 3, 11], "potenti": 2, "secur": [2, 10, 20], "vulner": 2, "project": [2, 9, 22], "mail": 2, "intel": [2, 9, 21], "sensit": 2, "our": 2, "pgp": 2, "kei": [2, 6, 15], "much": 2, "version": [2, 3, 4, 6, 7, 11, 12, 15, 17, 18, 19, 22], "affect": 2, "known": [2, 17, 21, 22], "exploit": 2, "member": 2, "product": 2, "team": 2, "review": [2, 3, 4, 11, 15, 21, 22], "contact": [2, 14], "collabor": 2, "resolv": 2, "guidelin": 2, "linux": [3, 7, 11, 15, 17], "kernel": [3, 7, 10, 15, 17, 20, 21, 22], "oper": [3, 4, 5, 7, 8, 11, 12, 15, 16, 20, 21, 22], "instruct": [3, 5, 7, 13, 16, 17, 20, 22], "_": [3, 4, 10, 11, 12, 15, 18, 20], "directori": [3, 6, 8, 13], "doc": [3, 11], "successfulli": [3, 4, 11, 12, 17, 18], "visibl": 3, "devf": [3, 9, 10], "dev": [3, 10, 17, 21, 22], "dlb0": 3, "dlb1": 3, "sr": [3, 15], "iov": [3, 15], "continu": [3, 5], "eventdev": 3, "unbind": [3, 15], "bind": [3, 15], "patch": [3, 19], "wget": 3, "q": [3, 8], "fast": 3, "org": [3, 11], "rel": 3, "tar": [3, 11], "xz": 3, "o": [3, 4, 5, 7, 11, 12, 13, 18, 19, 20, 21, 22], "jx": 3, "downloadmirror": 3, "734482": 3, "dlb_linux_src_release_7": 3, "7": [3, 4, 10, 12, 15, 18, 20], "0_2022_06_17": 3, "txz": 3, "p1": 3, "dpdk_dlb_": 3, "_diff": 3, "sed": 3, "270b": 3, "2710": 3, "2714": 3, "2711": 3, "usertool": 3, "devbind": 3, "py": 3, "grep": [3, 4, 5, 7, 11, 12, 13, 15, 16, 17], "a10": 3, "0000": 3, "6d": 3, "00": [3, 11], "drv": 3, "unus": 3, "72": 3, "echo": 3, "tee": 3, "sy": [3, 10, 11], "bu": 3, "pci": [3, 11, 15, 21, 22], "sriov_numvf": 3, "fail": [3, 11], "sai": 3, "No": [3, 13], "bound": [3, 20], "vfio": [3, 15], "appear": 3, "dlb10": 3, "dlb11": 3, "dlb8": 3, "dlb9": 3, "a14": 3, "2": [3, 4, 5, 10, 11, 12, 15, 18, 22], "3": [3, 10, 11, 18, 22], "assign": [3, 8], "process": [3, 18, 20], "among": [3, 11, 15], "vfn": 3, "m": [3, 18], "2048": 3, "vf0_resourc": 3, "num_atomic_inflight": 3, "num_dir_credit": 3, "64": [3, 15, 20], "num_dir_port": 3, "num_hist_list_entri": 3, "8192": 3, "num_ldb_credit": 3, "num_ldb_port": 3, "32": [3, 15], "num_ldb_queu": 3, "num_sched_domain": 3, "num_sn0_slot": 3, "num_sn1_slot": 3, "back": 3, "app": [3, 4, 5, 7, 12, 18, 19], "altern": [3, 15, 17], "ld_library_path": [3, 18], "pwd": 3, "dir_traff": 3, "128": 3, "d": [3, 6, 8, 15, 18, 19], "dlbn": 3, "meson": 3, "python3": 3, "pyelftool": 3, "libnuma": 3, "pip": 3, "ninja": 3, "debian": 3, "prefix": [3, 5, 11, 19], "installdir": 3, "builddir": 3, "c": [3, 11], "huge": 3, "vdev": 3, "dlb2_event": 3, "dev_id": 3, "order_queu": 3, "nb_flow": 3, "nb_pkt": 3, "512": 3, "plcore": 3, "wlcore": 3, "either": [3, 6, 7, 10, 11, 13, 15, 18], "daemonset": [3, 4, 6, 7, 11, 12, 13, 14, 15, 17, 18, 19, 20], "hand": [3, 6, 7, 11, 15], "per": [3, 10, 11, 17], "basi": [3, 11], "thu": [3, 4, 7, 10, 11, 12, 13, 15, 17, 18], "dlb_plugin": 3, "ref": [3, 4, 5, 7, 11, 12, 13, 15, 16, 17, 18, 19], "release_vers": [3, 4, 7, 11, 12, 13, 15, 17, 18, 19], "substitut": [3, 4, 7, 11, 12, 15, 17, 18], "desir": [3, 4, 7, 11, 12, 13, 15, 17, 18, 19], "devel": [3, 4, 7, 11, 12, 15, 17, 18, 19], "noth": [3, 4, 11, 12, 18], "els": [3, 4, 12, 18], "develop": [3, 4, 6, 7, 11, 12, 14, 15, 18, 19, 22], "been": [3, 4, 7, 10, 11, 12, 18], "expect": [3, 4, 7, 10, 11, 12, 18], "search": [3, 4, 6, 11, 12, 14, 18], "rang": [3, 4, 11, 12, 18, 19], "item": [3, 4, 10, 11, 12, 13, 18, 19], "allocat": [3, 4, 11, 12, 18, 20], "master": [3, 4, 11, 12, 13, 18, 19], "we": [3, 4, 12, 13, 18, 20], "off": [3, 4, 12, 21, 22], "local": [3, 4, 5, 7, 12, 13, 16, 17, 21, 22], "f": [3, 4, 5, 11, 12, 13, 14, 17, 18], "wait": [3, 4, 12, 18], "until": [3, 4, 10, 11, 12], "complet": [3, 4, 11, 12, 15, 17, 18, 19], "readi": [3, 4, 7, 11, 12, 13, 14, 15, 17, 18], "restart": [3, 4, 7, 11, 12, 13, 15, 17, 18], "ag": [3, 4, 7, 11, 12, 13, 15, 17, 18], "79m": 3, "18h": 3, "log": [3, 4, 8, 11, 15, 17, 21, 22], "did": [3, 4, 11, 12, 18], "possibli": [3, 4, 11, 12, 18], "becaus": [3, 4, 11, 12, 18], "could": [3, 4, 9, 10, 11, 12, 18], "stuck": [3, 4, 11, 12, 18], "pend": [3, 4, 11, 12, 18], "event": [3, 4, 11, 12, 18], "describ": [3, 4, 5, 7, 10, 11, 12, 13, 15, 16, 17, 18], "a3": [3, 4, 12], "reason": [3, 4, 11, 12, 18], "failedschedul": [3, 4, 11, 12, 18], "85": 3, "schedul": [3, 4, 9, 11, 12, 18, 19, 20], "insuffici": [3, 4, 11, 12, 18], "queue": [4, 12], "present": [4, 7, 10, 12], "workqueu": [4, 12], "through": [4, 5, 12, 13, 15, 16], "initcontain": [4, 7, 10, 11, 12, 15], "dsa_plugin": 4, "idxd": [4, 12], "engin": [4, 7, 12, 18], "group": [4, 12, 21, 22], "overlai": [4, 7, 11, 12, 13, 14, 15, 16, 17, 18], "dsa_initcontain": 4, "store": [4, 5, 12, 19], "provisioningconfig": [4, 12, 15], "configmap": [4, 12, 15], "specif": [4, 7, 9, 10, 12, 13, 14, 15, 17, 20], "congfigur": [4, 12, 15], "nodenam": [4, 12, 15], "profil": [4, 12, 15], "inteldeviceplugin": [4, 12, 15], "conf": [4, 6, 12, 15], "8": [4, 5, 11, 13, 14, 16, 18], "node1": [4, 12], "31m": [4, 12], "tail": 4, "debug": [4, 7, 15], "6": [4, 10, 11, 14, 15, 18, 20], "consid": [4, 10], "9": [4, 15, 18, 20], "13": [4, 20], "14": [4, 17, 20], "2m26": [4, 12], "devic": [5, 6, 8, 10, 16, 19, 21], "skip": [5, 7, 16], "plugin": [5, 6, 9, 10, 16, 19, 21], "sinc": [5, 7, 15, 16, 17], "cri": [5, 7, 20, 21, 22], "hook": [5, 6, 7, 11, 21, 22], "therefor": [5, 7, 10, 11, 15], "determin": [5, 15], "keep": 5, "bypass": 5, "restrict": 5, "deni": 5, "try": 5, "knowledg": 5, "interfaceid": 5, "prestart": [5, 7, 21, 22], "same": [5, 6, 7, 10, 11, 15, 17], "basic": [5, 6, 7, 15, 17], "cert": [5, 7, 13, 16, 17], "behind": [5, 13, 16, 17], "corpor": [5, 11, 13, 16], "proxi": [5, 13, 16, 17], "servic": [5, 7, 10, 11, 13, 15, 16, 17, 18, 20], "kube": [5, 13, 16], "apiserv": [5, 13, 16], "no_proxi": [5, 13, 16], "svc": [5, 13, 16], "wa": [5, 11, 13, 16], "kubeadm": [5, 13, 16], "etc": [5, 6, 13, 16, 19], "plane": [5, 7, 13, 15, 16, 17, 22], "append": [5, 13, 16], "apivers": [5, 13, 16, 17], "advertis": [5, 7, 13, 14, 16, 17], "address": [5, 13, 16], "237": [5, 13, 16], "71": [5, 13, 16], "99": [5, 13, 16], "env": [5, 13, 16, 18], "http_proxi": [5, 13, 16], "8080": [5, 13, 16], "https_proxi": [5, 13, 16], "8433": [5, 13, 16], "127": [5, 13, 16], "localhost": [5, 13, 16, 17], "right": [5, 13, 16], "begin": [5, 13, 16, 20], "init": [5, 9, 13, 14, 16], "instrument": 5, "Being": 5, "translat": 5, "actual": [5, 7], "v2": 5, "acceleratorfunct": [5, 7], "arria10": [5, 13], "dcp1": 5, "nlb0": 5, "preprogram": [5, 20], "afuid": 5, "d8424dc4a4a3c413f89e433683f9040b": 5, "69528db6eb31577a8c3668f9faa081f6": 5, "mode": [5, 6, 8, 20, 21, 22], "af": [5, 7], "695": 5, "d84": 5, "avkntusxv3qmnmj5": 5, "qcb9thctcsko8qt": 5, "j5dnop5ba": 5, "indic": [5, 10], "charact": [5, 10], "region": [5, 7, 20], "base64": 5, "encod": 5, "concaten": 5, "would": [5, 9, 10], "though": 5, "probabl": [5, 18], "renam": 5, "someth": 5, "orchestr": [5, 6, 7, 20], "reflect": 5, "fpgaregion": [5, 7], "respect": [5, 14], "fpga_admissionwebhook": 5, "com_af": 5, "com_region": 5, "between": [5, 11, 15], "collect": [5, 7, 14], "intelfpgawebhook": 5, "customresourcedefinit": [5, 7], "apiextens": [5, 7], "mutatingwebhookconfigur": [5, 7], "admissionregistr": [5, 7], "clusterrol": [5, 7], "rbac": [5, 7, 11], "author": [5, 7, 22], "role": [5, 7, 17], "clusterrolebind": [5, 7], "rolebind": [5, 7], "certif": [5, 7, 17], "serv": [5, 7, 13], "issuer": [5, 7], "selfsign": [5, 7], "prepar": [5, 18, 20], "intend": [5, 9, 10], "scope": [5, 10], "were": [5, 9, 18], "applic": [5, 11, 15, 17, 20], "trigger": [6, 13], "workload": [6, 9, 14, 15, 17, 19, 20, 21, 22], "admiss": [6, 7, 17, 21, 22], "benign": [6, 7], "usual": 6, "recommend": [6, 11, 15], "control": [6, 7, 11, 13, 14, 17, 21, 22], "recent": 6, "ship": 6, "prevent": 6, "oci": [6, 20], "ensur": [6, 20], "hooks_dir": 6, "unset": 6, "programm": [7, 20], "convert": [7, 18, 19], "logic": [7, 15], "howev": 7, "upon": 7, "helper": [7, 17], "dure": 7, "initi": [7, 18], "facilit": [7, 11], "bistream": 7, "onto": 7, "utilis": [7, 15], "diagram": 7, "cover": [7, 15, 17], "major": 7, "being": 7, "than": [7, 10, 11], "auto": [7, 13], "detect": [7, 10, 11, 15, 17], "tree": [7, 11, 17], "dfl": 7, "link": [7, 14, 15, 19], "7747db9d88": [7, 13], "bd2nl": [7, 13], "1m": 7, "cainjector": [7, 13], "87c85c6ff": [7, 13], "59sb5": [7, 13], "64dc9fff44": [7, 13], "29cfc": [7, 13], "form": 7, "admisson": 7, "fpga_plugin": 7, "intelfpgaplugin": 7, "getter": 7, "fpgadeviceplugin": 7, "result": [7, 11, 13, 18], "skcw5": 7, "57": 7, "7d6bcb8b57": 7, "k52b9": 7, "differ": [7, 9, 11, 13, 20], "node_nam": [7, 15], "And": [7, 17], "similar": 7, "ce48969398f05f33946d560708be108a": 7, "well": [7, 21, 22], "wish": 7, "avoid": 7, "examin": [8, 15, 17], "fpgainfo": 8, "fmeinfo": 8, "portinfo": 8, "fme": 8, "port": 8, "fpga_tool": 8, "b": 8, "gb": [8, 20], "aocx": 8, "dry": 8, "forc": 8, "overwrit": 8, "quiet": 8, "error": [8, 22], "larg": 9, "test": [9, 19, 20, 21, 22], "hw": 9, "Its": 9, "fakedev": 9, "label": [9, 13, 14, 17, 18, 19, 20, 21, 22], "sysf": [9, 10], "real": 9, "json": [9, 13], "variant": 9, "mix": 9, "Such": 9, "themselv": 9, "exlud": 9, "now": [9, 13, 14, 20], "achiev": 9, "ti": 9, "suitabl": [9, 11], "activ": 9, "metric": [9, 11, 19], "prometheu": 9, "look": 9, "imag": [10, 13, 21, 22], "worker": [10, 13, 17, 20], "select": [10, 11, 13, 14], "numer": [10, 11, 13], "turn": 10, "kubernet": [10, 13, 14, 19, 20, 21], "finer": 10, "grain": 10, "amount": [10, 11], "read": [10, 17, 19], "gt": [10, 11, 15, 19], "gpu_memory_overrid": 10, "gpu_memory_reserv": 10, "suppos": 10, "hold": 10, "byte": 10, "older": [10, 21, 22], "millicor": 10, "count": [10, 19], "1000": 10, "fraction": [10, 21, 22], "max": [10, 15], "sum": 10, "OR": [10, 14], "separ": [10, 11, 15], "class": [10, 11], "drm": [10, 11], "deprec": [10, 15], "primari": 10, "dri": [10, 11], "subsystem": 10, "lt": [10, 11, 15, 20], "num": [10, 11, 15], "tile": [10, 11], "numa": 10, "fit": 10, "63": 10, "length": 10, "numbers2": 10, "numbers3": 10, "total": [10, 15, 20], "homogen": 10, "calcul": 10, "divid": 10, "combin": [10, 11], "mean": [10, 11, 15, 17, 19], "complex": 10, "As": [10, 11, 13], "exce": 10, "gpu_pci_grouping_level": 10, "greater": 10, "zero": [10, 11], "belong": 10, "ident": 10, "sg1": 10, "3_4": 10, "5": [10, 13, 15, 17, 18, 20], "groups2": 10, "groups3": 10, "debugf": 10, "unfortun": 10, "guarante": [10, 15], "stabl": 10, "simpli": [10, 13], "introduc": 10, "platform_gen": 10, "typic": [10, 11], "integ": 10, "media_vers": 10, "pipelin": 10, "graphics_vers": 10, "graphic": [10, 20], "comput": [10, 11, 17, 20], "platform_": 10, "platform_nam": 10, "true": [10, 13, 17], "presens": 10, "abov": [10, 11, 14, 15, 17, 18], "offload": 11, "center": 11, "flex": 11, "seri": [11, 15], "transcod": [11, 20], "cloud": 11, "game": 11, "high": 11, "ai": 11, "train": 11, "infer": [11, 18], "oneapi": 11, "backend": 11, "flag": [11, 15, 17, 19], "i915_monitor": 11, "int": [11, 15, 17, 19], "none": 11, "pack": [11, 15], "meaning": 11, "effect": 11, "common": 11, "firmwar": 11, "uevent": 11, "card0": 11, "card1": 11, "subset": 11, "enterpris": 11, "dgpu": 11, "index": 11, "html": 11, "force_prob": 11, "pci_id": 11, "rfc": 11, "lspci": [11, 15], "vga": 11, "displai": 11, "88": 11, "56c1": 11, "rev": 11, "05": 11, "8d": 11, "compat": [11, 20], "mesa": 11, "iri": 11, "3d": 11, "header": 11, "brand": 11, "gitlab": 11, "freedesktop": 11, "blob": [11, 18], "find": 11, "dmesg": 11, "pub": 11, "scm": 11, "enough": 11, "against": [11, 22], "gpu_plugin": 11, "attach": 11, "presenc": 11, "doesn": 11, "yet": 11, "nodefeaturerul": [11, 13, 14], "rule": [11, 13, 14, 15, 17, 18], "nfd_labeled_nod": 11, "retriev": [11, 19], "xpu": [11, 22], "collectd": 11, "monitoring_shar": 11, "dev_nfd": 11, "experiment": 11, "memori": [11, 14, 17, 18, 20, 21, 22], "consum": 11, "suffici": 11, "properli": [11, 14], "equal": 11, "heteregen": 11, "isn": 11, "quit": 11, "simpl": [11, 22], "podresourc": 11, "plu": 11, "take": [11, 15, 17], "care": 11, "explain": 11, "fractional_resourc": 11, "let": 11, "decis": [11, 19], "uniqu": 11, "increas": 11, "timestamp": 11, "ga": [11, 19], "latter": 11, "card2": 11, "card3": 11, "down": [11, 17], "unnecessarili": 11, "improv": 11, "gt0": 11, "gt1": 11, "similarli": 11, "split": 11, "second": [11, 17, 19], "non": [11, 13, 15, 18], "dac": [11, 15, 18], "socket": [11, 15, 18], "creation": [11, 15, 18], "securitycontext": [11, 15, 18], "runasus": [11, 15, 18], "runasgroup": [11, 15, 18], "jsonpath": [11, 18, 19], "clinfo": 11, "capabl": [11, 15, 17, 21, 22], "icd": 11, "Then": 11, "accordingli": 11, "editor": 11, "singl": [11, 13, 14, 17], "import": 11, "containerd": [11, 20], "image_nam": 11, "save": 11, "ctr": 11, "rm": 11, "batch": [11, 18], "fgrep": [11, 18], "xxxxx": [11, 18], "unknown": [11, 18], "unlik": 11, "onevpl": 11, "mediasdk": 11, "offer": 11, "overridden": 11, "most": 11, "renderd128": 11, "explictli": 11, "sever": 11, "unrel": 11, "explicitli": 11, "told": 11, "ffmpeg": 11, "trac": 11, "wiki": 11, "vaapi": 11, "render": 11, "shell": [11, 20], "invok": 11, "vainfo": 11, "usr": [11, 15], "bin": 11, "renderdxxx": 11, "Or": 11, "inlin": 11, "nth": 11, "iaa_plugin": 12, "iaa_initcontain": 12, "30": 12, "bash": [12, 15], "whose": 13, "goal": 13, "lifecycl": 13, "point": [13, 14], "gpu": [13, 14, 19, 21, 22], "sgx": [13, 21, 22], "dsa": [13, 14, 21, 22], "dlb": [13, 21, 22], "599c58dffc": 13, "9wql4": 13, "25h": 13, "qqq4h": 13, "immedi": 13, "minut": 13, "pick": 13, "jq": 13, "r": [13, 20], "deviceplugin_v1_dlbdeviceplugin": 13, "deviceplugin_v1_qatdeviceplugin": 13, "deviceplugin_v1_sgxdeviceplugin": [13, 17], "deviceplugin_v1_gpudeviceplugin": 13, "deviceplugin_v1_fpgadeviceplugin": 13, "deviceplugin_v1_dsadeviceplugin": 13, "21d": 13, "gpudeviceplugin": 13, "raw": [13, 14, 17], "githubusercont": [13, 14, 17], "observ": 13, "selector": 13, "order": [13, 15], "done": [13, 14, 15], "cr": [13, 14], "url": 13, "kept": 13, "intact": 13, "leader": 13, "elect": 13, "overload": 13, "excess": 13, "heart": 13, "beat": 13, "timeout": 13, "crash": 13, "client": [13, 18], "allevi": 13, "476": 13, "quickaccess": 14, "technologi": [14, 21, 22], "iaa": [14, 21, 22], "futur": 14, "third": 14, "later": [14, 17], "bio": 14, "shown": 14, "machin": [14, 20], "rhel": 14, "oc": 14, "nodefeaturediscoveri": 14, "selinux": 14, "container_device_plugin_t": 14, "container_device_plugin_init_t": 14, "navig": 14, "click": 14, "succeed": 14, "tab": 14, "sgxdeviceplugin": [14, 17], "c62x": [15, 20], "chipset": [15, 20], "gen4": 15, "atom": 15, "c3000": 15, "8925": 15, "8955": 15, "cryptograph": 15, "compress": [15, 20], "summaris": [15, 17], "comma": 15, "dh895xcc": 15, "c3xxx": 15, "4xxx": 15, "401xx": 15, "c4xxx": 15, "d15xx": 15, "c6xxvf": 15, "4xxxvf": 15, "maximum": 15, "spread": 15, "next": [15, 17, 21, 22], "choic": 15, "disable_denylist": 15, "correctli": [15, 17, 20], "prior": 15, "except": 15, "exclud": 15, "isol": [15, 20], "remov": [15, 17, 19], "libqat": 15, "uio": 15, "One": 15, "upstream": [15, 21, 22], "qat_plugin": 15, "switch": 15, "regardless": 15, "qat_initcontain": 15, "cfg_servic": 15, "servicesen": 15, "sym": 15, "asym": 15, "confirm": [15, 17], "exercis": 15, "poll": 15, "pmd": 15, "websit": 15, "hugepag": 15, "qat_dpdk_app": 15, "27m": 15, "5zgvb": 15, "3h": 15, "igb_uio": 15, "sys_admin": 15, "exec": 15, "w": 15, "qat1": 15, "lib64": 15, "librte_mempool_r": 15, "librte_pmd_qat": 15, "ptest": 15, "throughput": 15, "devtyp": 15, "crypto_qat": 15, "optyp": 15, "cipher": 15, "algo": 15, "ae": 15, "cbc": 15, "op": 15, "sz": 15, "10000000": 15, "burst": 15, "buffer": 15, "crypto1": 15, "compress1": 15, "tc1": 15, "cpu": [15, 20], "static": 15, "0442": 15, "0443": 15, "18a1": 15, "37c9": 15, "6f55": 15, "19e3": 15, "4941": 15, "4943": 15, "8086": 15, "epc": [16, 17, 22], "sgx_admissionwebhook": [16, 17], "certmanag": 16, "solut": 17, "enclav": 17, "sgx_enclav": 17, "provis": [17, 20, 21, 22], "sgx_provis": 17, "dcap": [17, 20], "41": 17, "suppli": 17, "assum": 17, "config_x86_sgx": 17, "express": 17, "approach": 17, "involv": 17, "everyth": 17, "sgx_plugin": 17, "lesser": 17, "footprint": 17, "capac": 17, "addition": 17, "98566144": 17, "400": 17, "reli": 17, "parti": 17, "trust": 17, "date": 17, "pck": 17, "pcc": 17, "connect": 17, "about": [17, 20], "accord": 17, "architectur": 17, "daemon": [17, 18], "hostnetwork": 17, "sgx_aesmd": 17, "mrnm8": 17, "3h47m": 17, "z5dcq": 17, "llwlw": 17, "28m": 17, "quick": 17, "experi": 17, "var": 17, "extra": 17, "cat": 17, "kind_config": 17, "v1alpha4": 17, "your_node_nam": 17, "extramount": 17, "propag": 17, "bidirect": 17, "bootstrap": 17, "sgx_enclave_app": 17, "sgx_ecdsa_aesmd_quot": 17, "3h55m": 17, "vtq84": 17, "35m": 17, "step1": 17, "sgx_qe_get_target_info": 17, "succe": 17, "step2": 17, "create_app_report": 17, "step3": 17, "sgx_qe_get_quote_s": 17, "step4": 17, "sgx_qe_get_quot": 17, "cert_key_typ": 17, "0x5": 17, "pcie": 18, "e5": [18, 20], "mustang": 18, "v100": 18, "6th": 18, "pc": 18, "movidiu": 18, "ve3": 18, "soc": 18, "nativ": [18, 20], "openvisualcloud": 18, "vpu_plugin": 18, "xlink": 18, "vcaanode00": 18, "gopath": 18, "src": 18, "hddl_install_dir": 18, "inference_engine_samples_build": 18, "intel64": 18, "lib": 18, "classification_sample_async": 18, "openvino_model": 18, "ir": 18, "fp16": 18, "squeezenet": 18, "caff": 18, "squeezenet1": 18, "xml": 18, "car": 18, "png": 18, "inferenceengin": 18, "custom_releas": 18, "2019": 18, "r2_f5827d4773ebbe727c9acac5f007f7d94dd4be4": 18, "pars": 18, "hddlplugin": 18, "27579": 18, "network": 18, "resiz": 18, "787": 18, "259": 18, "227": 18, "model": 18, "07": 18, "49": 18, "01": 18, "0427": 18, "servicestart": 18, "40": 18, "0428": 18, "45": 18, "0429": 18, "hddlclient": 18, "256": 18, "dispatcher2": 18, "0432": 18, "148": 18, "senderroutin": 18, "270": 18, "registercli": 18, "0435": 18, "275": 18, "7235": 18, "hddlblob": 18, "166": 18, "ion": 18, "7237": 18, "176": 18, "asynchron": 18, "async": 18, "top": 18, "classid": 18, "817": 18, "8295898": 18, "sport": 18, "511": 18, "0961304": 18, "479": 18, "0439453": 18, "wheel": 18, "751": 18, "0101318": 18, "racer": 18, "race": 18, "436": 18, "0074234": 18, "beach": 18, "wagon": 18, "station": 18, "estat": 18, "waggon": 18, "656": 18, "0042267": 18, "minivan": 18, "586": 18, "0029869": 18, "half": 18, "track": 18, "717": 18, "0018148": 18, "pickup": 18, "truck": 18, "864": 18, "0013924": 18, "tow": 18, "wrecker": 18, "581": 18, "0006595": 18, "grill": 18, "radiat": 18, "9231": 18, "212": 18, "listen": 18, "thread": 18, "wake": 18, "exit": 18, "9232": 18, "81": 18, "dispatch": 18, "9235": 18, "203": 18, "unregist": 18, "success": 18, "measur": 18, "benchmark_app": 18, "benefici": 19, "optim": 19, "lane": 19, "minimum": 19, "interv": 19, "topologi": 19, "fetch": 19, "startup": 19, "delai": 19, "xe": 19, "xpumanager_sidecar": 19, "xpumanag": 19, "kustom_xpumanag": 19, "smarter": 19, "0_0": 19, "join": 20, "rtk": 20, "slack": 20, "ask": 20, "question": 20, "learn": 20, "kubeless": 20, "faa": 20, "javascript": 20, "captur": 20, "1x": 20, "2687": 20, "ghz": 20, "nuc6i7kyk": 20, "skull": 20, "canyon": 20, "04": 20, "36": 20, "loopback": 20, "english": 20, "letter": 20, "recognit": 20, "compar": 20, "2650": 20, "00ghz": 20, "62": 20, "gx": 20, "pac": 20, "opensus": 20, "leap": 20, "g47eef04": 20, "03": 20, "runc": 20, "rc8": 20, "multi": [20, 21, 22], "gold": 20, "6252n": 20, "30ghz": 20, "195": 20, "d5005": 20, "cento": 20, "1062": 20, "el7": 20, "x86": 20, "storag": 20, "ecdsa": [20, 21, 22], "proc": 20, "lightweight": 20, "vm": 20, "feel": 20, "tradit": 20, "advantag": 20, "2x": 20, "2687w": 20, "v4": 20, "251": 20, "lp150": 20, "00033": 20, "good": 20, "shape": 20, "introduct": [21, 22], "dlb2": [21, 22], "registr": [21, 22], "requisit": [21, 22], "overview": [21, 22], "prerequisit": [21, 22], "dkm": [21, 22], "monitor": [21, 22], "workaround": [21, 22], "qsv": [21, 22], "va": [21, 22], "upgrad": [21, 22], "quickassist": [21, 22], "autom": [21, 22], "vpu": [21, 22], "classif": [21, 22], "xelink": 22, "sidecar": 22, "helm": 22, "chart": 22, "mainten": 22, "dai": 22, "newer": 22, "modif": 22, "publish": 22, "e2": 22, "convent": 22, "checklist": 22, "screencast": 22, "nlb3": 22}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"instruct": 0, "devic": [0, 1, 3, 4, 7, 9, 11, 12, 13, 14, 15, 17, 18, 20, 22], "plugin": [0, 1, 3, 4, 7, 11, 12, 13, 14, 15, 17, 18, 20, 22], "develop": [0, 1], "mainten": 0, "dai": 0, "how": 0, "": 0, "get": 0, "sourc": 0, "code": 0, "build": [0, 17, 18], "run": [0, 15, 18], "binari": 0, "contain": [0, 14], "imag": [0, 1, 3, 4, 7, 11, 12, 15, 17, 18, 19], "against": 0, "newer": 0, "version": [0, 1], "kubernet": [0, 1, 3, 4, 5, 6, 7, 11, 12, 15, 16, 17, 18, 22], "work": 0, "intel": [0, 1, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22], "oper": [0, 1, 13, 14, 17], "modif": 0, "publish": 0, "new": 0, "operatorhub": 0, "io": 0, "e2": 0, "test": [0, 3, 4, 8, 11, 12, 15, 17, 18], "control": [0, 1, 5, 16], "local": [0, 18], "plane": 0, "simpl": 0, "log": [0, 18], "error": 0, "convent": 0, "checklist": 0, "overview": [1, 7, 20], "prerequisit": [1, 7, 11, 14, 15, 17], "gpu": [1, 9, 10, 11, 20], "fpga": [1, 5, 6, 7, 8, 20], "admiss": [1, 5, 16], "cri": [1, 6], "o": [1, 6], "prestart": [1, 6], "hook": [1, 10], "qat": [1, 15], "vpu": [1, 18], "sgx": [1, 14, 16, 17, 20], "webhook": [1, 5, 6], "epc": 1, "memori": [1, 10], "registr": [1, 3, 4, 7, 11, 12, 15, 17, 18], "dsa": [1, 4], "dlb": [1, 3], "iaa": [1, 12], "xelink": [1, 19], "xpu": [1, 19], "manag": [1, 19], "sidecar": [1, 19], "demo": [1, 3, 4, 11, 12, 15, 17, 18, 20], "workload": [1, 11], "author": 1, "support": 1, "pre": [1, 3, 4, 5, 7, 11, 12, 15, 16, 17, 18, 19], "built": [1, 3, 4, 7, 11, 12, 15, 17, 18, 19], "licens": 1, "helm": 1, "chart": 1, "introduct": [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19], "dlb2": 3, "driver": [3, 11], "configur": [3, 6, 7, 9, 11, 15, 17, 19, 20], "pf": 3, "vf": 3, "us": [3, 14, 17], "dpdk": [3, 15, 20], "tool": [3, 8, 9], "verif": 3, "well": 3, "instal": [3, 4, 5, 7, 11, 12, 13, 14, 15, 16, 17, 18, 19], "verifi": [3, 4, 7, 11, 12, 14, 15, 17, 18, 19], "automat": [4, 12, 15], "provis": [4, 12, 15], "depend": [5, 6], "requisit": [5, 16], "map": 5, "deploy": [5, 16], "next": 5, "step": [5, 20], "compon": 7, "mode": [7, 11, 15, 17, 19], "option": [7, 10, 11, 15, 17, 19], "command": 8, "line": 8, "usag": [8, 13], "fake": 9, "file": 9, "gener": 9, "potenti": 9, "improv": 9, "relat": 9, "nfd": [10, 11], "default": 10, "label": 10, "pci": 10, "group": 10, "capabl": 10, "limit": 10, "discret": 11, "kernel": 11, "dkm": 11, "packag": 11, "upstream": 11, "user": 11, "space": 11, "older": 11, "integr": 11, "all": 11, "node": 11, "monitor": 11, "share": 11, "dev": 11, "fraction": 11, "resourc": 11, "detail": 11, "issu": [11, 13], "media": 11, "multi": 11, "setup": 11, "workaround": 11, "qsv": 11, "va": 11, "api": 11, "upgrad": 13, "known": 13, "red": 14, "hat": 14, "openshift": 14, "platform": [14, 20], "tabl": 14, "content": 14, "minimum": 14, "hardwar": [14, 15], "requir": 14, "enabl": 14, "server": 14, "web": 14, "consol": 14, "deploi": [14, 15, 17], "quickassist": [15, 20], "technologi": [15, 20], "pod": [15, 17], "manual": 15, "autom": 15, "openssl": [15, 20], "check": 15, "softwar": 17, "guard": 17, "extens": [17, 21], "kubectl": 17, "ecdsa": 17, "remot": 17, "attest": 17, "docker": 18, "an": [18, 19], "classif": 18, "exampl": 18, "creat": 18, "job": 18, "unit": 18, "off": 18, "review": 18, "exist": 19, "function": 19, "video": 20, "screencast": 20, "1": 20, "2": 20, "nlb3": 20, "3": 20, "opencl": 20}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx": 57}, "alltitles": {"Instructions for Device Plugin Development and Maintenance": [[0, "instructions-for-device-plugin-development-and-maintenance"]], "Day-to-day Development How to\u2019s": [[0, "day-to-day-development-how-to-s"]], "Get the Source Code": [[0, "get-the-source-code"]], "Build and Run Plugin Binaries": [[0, "build-and-run-plugin-binaries"]], "Build Container Images": [[0, "build-container-images"]], "Build Against a Newer Version of Kubernetes": [[0, "build-against-a-newer-version-of-kubernetes"]], "Work with Intel Device Plugins Operator Modifications": [[0, "work-with-intel-device-plugins-operator-modifications"]], "Publish a New Version of the Intel Device Plugins Operator to operatorhub.io": [[0, "publish-a-new-version-of-the-intel-device-plugins-operator-to-operatorhub-io"]], "Run E2E Tests": [[0, "run-e2e-tests"]], "Run Controller Tests with a Local Control Plane": [[0, "run-controller-tests-with-a-local-control-plane"]], "How to Develop Simple Device Plugins": [[0, "how-to-develop-simple-device-plugins"]], "Logging": [[0, "logging"]], "Error Conventions": [[0, "error-conventions"]], "Checklist for New Device Plugins": [[0, "checklist-for-new-device-plugins"]], "Overview": [[1, "overview"]], "Prerequisites": [[1, "prerequisites"], [7, "prerequisites"], [11, "prerequisites"], [14, "prerequisites"], [15, "prerequisites"], [17, "prerequisites"]], "Plugins": [[1, "plugins"]], "GPU Device Plugin": [[1, "gpu-device-plugin"]], "FPGA Device Plugin": [[1, "fpga-device-plugin"]], "Device Plugin": [[1, "device-plugin"]], "Admission Controller": [[1, "admission-controller"]], "CRI-O Prestart Hook": [[1, "cri-o-prestart-hook"]], "QAT Device Plugin": [[1, "qat-device-plugin"]], "VPU Device Plugin": [[1, "vpu-device-plugin"]], "SGX Device Plugin": [[1, "sgx-device-plugin"]], "device plugin": [[1, "id1"]], "Intel SGX Admission Webhook": [[1, "intel-sgx-admission-webhook"]], "Intel SGX EPC memory registration": [[1, "intel-sgx-epc-memory-registration"]], "DSA Device Plugin": [[1, "dsa-device-plugin"]], "DLB Device Plugin": [[1, "dlb-device-plugin"]], "IAA Device Plugin": [[1, "iaa-device-plugin"]], "Device Plugins Operator": [[1, "device-plugins-operator"]], "XeLink XPU-Manager Sidecar": [[1, "xelink-xpu-manager-sidecar"]], "Demos": [[1, "demos"]], "Workload Authors": [[1, "workload-authors"]], "Developers": [[1, "developers"]], "Supported Kubernetes Versions": [[1, "supported-kubernetes-versions"]], "Pre-built plugin images": [[1, "pre-built-plugin-images"]], "License": [[1, "license"]], "Helm Charts": [[1, "helm-charts"]], "Intel DLB device plugin for Kubernetes": [[3, "intel-dlb-device-plugin-for-kubernetes"]], "Introduction": [[3, "introduction"], [4, "introduction"], [5, "introduction"], [6, "introduction"], [7, "introduction"], [8, "introduction"], [9, "introduction"], [10, "introduction"], [11, "introduction"], [12, "introduction"], [13, "introduction"], [14, "introduction"], [15, "introduction"], [16, "introduction"], [17, "introduction"], [18, "introduction"], [19, "introduction"]], "DLB2 driver configuration for PFs": [[3, "dlb2-driver-configuration-for-pfs"]], "VF configuration using a DPDK tool (but with dlb2 driver)": [[3, "vf-configuration-using-a-dpdk-tool-but-with-dlb2-driver"]], "Verification of well-configured devices:": [[3, "verification-of-well-configured-devices"]], "Installation": [[3, "installation"], [4, "installation"], [5, "installation"], [7, "installation"], [11, "installation"], [12, "installation"], [13, "installation"], [14, "installation"], [15, "installation"], [16, "installation"], [17, "installation"], [18, "installation"], [19, "installation"]], "Pre-built Images": [[3, "pre-built-images"], [4, "pre-built-images"], [7, "pre-built-images"], [11, "pre-built-images"], [12, "pre-built-images"], [15, "pre-built-images"], [17, "pre-built-images"], [18, "pre-built-images"], [19, "pre-built-images"]], "Verify Plugin Registration": [[3, "verify-plugin-registration"], [4, "verify-plugin-registration"], [7, "verify-plugin-registration"], [11, "verify-plugin-registration"], [12, "verify-plugin-registration"], [15, "verify-plugin-registration"], [17, "verify-plugin-registration"], [18, "verify-plugin-registration"]], "Testing and Demos": [[3, "testing-and-demos"], [4, "testing-and-demos"], [11, "testing-and-demos"], [12, "testing-and-demos"], [17, "testing-and-demos"], [18, "testing-and-demos"]], "Intel DSA device plugin for Kubernetes": [[4, "intel-dsa-device-plugin-for-kubernetes"]], "Automatic Provisioning": [[4, "automatic-provisioning"], [12, "automatic-provisioning"], [15, "automatic-provisioning"]], "Intel FPGA admission controller for Kubernetes": [[5, "intel-fpga-admission-controller-for-kubernetes"]], "Dependencies": [[5, "dependencies"], [6, "dependencies"]], "Pre-requisites": [[5, "pre-requisites"], [16, "pre-requisites"]], "Mappings": [[5, "mappings"]], "Deployment": [[5, "deployment"], [16, "deployment"]], "Webhook deployment": [[5, "webhook-deployment"]], "Mappings deployment": [[5, "mappings-deployment"]], "Next steps": [[5, "next-steps"]], "Intel FPGA prestart CRI-O webhook for Kubernetes": [[6, "intel-fpga-prestart-cri-o-webhook-for-kubernetes"]], "Configuring CRI-O": [[6, "configuring-cri-o"]], "Intel FPGA device plugin for Kubernetes": [[7, "intel-fpga-device-plugin-for-kubernetes"]], "Component Overview": [[7, "component-overview"]], "Modes and Configuration Options": [[7, "modes-and-configuration-options"], [11, "modes-and-configuration-options"], [15, "modes-and-configuration-options"], [19, "modes-and-configuration-options"]], "Intel FPGA test tool": [[8, "intel-fpga-test-tool"]], "Command line and usage": [[8, "command-line-and-usage"]], "Fake (GPU) device file generator": [[9, "fake-gpu-device-file-generator"]], "Configuration": [[9, "configuration"]], "Potential improvements": [[9, "potential-improvements"]], "Related tools": [[9, "related-tools"]], "Intel GPU NFD hook": [[10, "intel-gpu-nfd-hook"]], "GPU memory": [[10, "gpu-memory"]], "Default labels": [[10, "default-labels"]], "PCI-groups (optional)": [[10, "pci-groups-optional"]], "Capability labels (optional)": [[10, "capability-labels-optional"]], "Limitations": [[10, "limitations"]], "Intel GPU device plugin for Kubernetes": [[11, "intel-gpu-device-plugin-for-kubernetes"]], "Drivers for discrete GPUs": [[11, "drivers-for-discrete-gpus"]], "Kernel driver": [[11, "kernel-driver"]], "Intel DKMS packages": [[11, "intel-dkms-packages"]], "Upstream kernel": [[11, "upstream-kernel"]], "User-space drivers": [[11, "user-space-drivers"]], "Drivers for older (integrated) GPUs": [[11, "drivers-for-older-integrated-gpus"]], "Install to all nodes": [[11, "install-to-all-nodes"]], "Install to nodes with Intel GPUs with NFD": [[11, "install-to-nodes-with-intel-gpus-with-nfd"]], "Install to nodes with NFD, Monitoring and Shared-dev": [[11, "install-to-nodes-with-nfd-monitoring-and-shared-dev"]], "Install to nodes with Intel GPUs with Fractional resources": [[11, "install-to-nodes-with-intel-gpus-with-fractional-resources"]], "Fractional resources details": [[11, "fractional-resources-details"]], "Issues with media workloads on multi-GPU setups": [[11, "issues-with-media-workloads-on-multi-gpu-setups"]], "Workaround for QSV and VA-API": [[11, "workaround-for-qsv-and-va-api"]], "Intel IAA device plugin for Kubernetes": [[12, "intel-iaa-device-plugin-for-kubernetes"]], "Intel Device Plugins Operator": [[13, "intel-device-plugins-operator"]], "Usage": [[13, "usage"]], "Upgrade": [[13, "upgrade"]], "Known issues": [[13, "known-issues"]], "Intel\u00ae Device Plugins Operator for Red Hat OpenShift Container Platform": [[14, "intel-device-plugins-operator-for-red-hat-openshift-container-platform"]], "Table of Contents": [[14, "table-of-contents"]], "Minimum Hardware Requirements": [[14, "minimum-hardware-requirements"]], "Intel SGX Enabled Server": [[14, "intel-sgx-enabled-server"]], "Install Operator using OpenShift Web Console": [[14, "install-operator-using-openshift-web-console"]], "Verify Operator installation": [[14, "verify-operator-installation"]], "Deploying Intel Device Plugins": [[14, "deploying-intel-device-plugins"]], "Intel SGX Device Plugin": [[14, "intel-sgx-device-plugin"]], "Intel QuickAssist Technology (QAT) device plugin for Kubernetes": [[15, "intel-quickassist-technology-qat-device-plugin-for-kubernetes"]], "Demos and Testing": [[15, "demos-and-testing"]], "DPDK QAT demos": [[15, "dpdk-qat-demos"]], "DPDK Prerequisites": [[15, "dpdk-prerequisites"]], "Deploy the pod": [[15, "deploy-the-pod"]], "Manual Test Run": [[15, "manual-test-run"]], "Automated Test Run": [[15, "automated-test-run"]], "OpenSSL QAT Demo": [[15, "openssl-qat-demo"]], "Checking for Hardware": [[15, "checking-for-hardware"]], "Intel SGX admission controller for Kubernetes": [[16, "intel-sgx-admission-controller-for-kubernetes"]], "Intel Software Guard Extensions (SGX) device plugin for Kubernetes": [[17, "intel-software-guard-extensions-sgx-device-plugin-for-kubernetes"]], "Modes and Configuration options": [[17, "modes-and-configuration-options"]], "Installation Using the Operator": [[17, "installation-using-the-operator"]], "Installation Using kubectl": [[17, "installation-using-kubectl"]], "SGX ECDSA Remote Attestation": [[17, "sgx-ecdsa-remote-attestation"]], "Remote Attestation Prerequisites": [[17, "remote-attestation-prerequisites"]], "Build the image": [[17, "build-the-image"]], "Deploy the pods": [[17, "deploy-the-pods"]], "Intel VPU device plugin for Kubernetes": [[18, "intel-vpu-device-plugin-for-kubernetes"]], "Build a Docker image with an classification example": [[18, "build-a-docker-image-with-an-classification-example"]], "Create a job running unit tests off the local Docker image": [[18, "create-a-job-running-unit-tests-off-the-local-docker-image"]], "Review the job logs": [[18, "review-the-job-logs"]], "XeLink sidecar for Intel XPU Manager": [[19, "xelink-sidecar-for-intel-xpu-manager"]], "Install XPU-Manager with the Sidecar": [[19, "install-xpu-manager-with-the-sidecar"]], "Install Sidecar to an Existing XPU-Manager": [[19, "install-sidecar-to-an-existing-xpu-manager"]], "Verify Sidecar Functionality": [[19, "verify-sidecar-functionality"]], "Demo": [[20, "demo"]], "Demo overview": [[20, "demo-overview"]], "Intel\u00ae GPU Device Plugin demo video": [[20, "intel-gpu-device-plugin-demo-video"]], "Demo platform configuration": [[20, "demo-platform-configuration"], [20, "id3"]], "Screencast": [[20, "screencast"], [20, "id5"], [20, "id7"]], "Intel\u00ae FPGA Device Plugin demo videos": [[20, "intel-fpga-device-plugin-demo-videos"]], "Demo 1 and 2 (NLB3) platform configuration": [[20, "demo-1-and-2-nlb3-platform-configuration"]], "Demo 3 (OpenCL) platform configuration": [[20, "demo-3-opencl-platform-configuration"]], "Demo steps": [[20, "demo-steps"], [20, "id1"], [20, "id4"], [20, "id6"]], "Screencasts": [[20, "screencasts"], [20, "id2"]], "Intel\u00ae SGX Device Plugin demo videos": [[20, "intel-sgx-device-plugin-demo-videos"]], "Intel\u00ae QuickAssist Technology Device Plugin OpenSSL demo video": [[20, "intel-quickassist-technology-device-plugin-openssl-demo-video"]], "Intel\u00ae QuickAssist Technology Device Plugin with DPDK demo video": [[20, "intel-quickassist-technology-device-plugin-with-dpdk-demo-video"]], "Extensions": [[21, "extensions"]], "Intel\u00ae Device Plugins for Kubernetes": [[22, "intel-device-plugins-for-kubernetes"]]}, "indexentries": {}}) \ No newline at end of file diff --git a/0.27/.buildinfo b/0.27/.buildinfo new file mode 100644 index 000000000..e33ab616c --- /dev/null +++ b/0.27/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: 527d0961f8224af12e66dd0bbe3563fe +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/0.27/DEVEL.html b/0.27/DEVEL.html new file mode 100644 index 000000000..ed63377a8 --- /dev/null +++ b/0.27/DEVEL.html @@ -0,0 +1,456 @@ + + + + + + + Instructions for Device Plugin Development and Maintenance — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+
    +
    • +
  • Instructions for Device Plugin Development and Maintenance
    • +
  • + View page source +
    • +
+
+
+
+
+ +
+

Instructions for Device Plugin Development and Maintenance

+

Table of Contents

+ +
+

Day-to-day Development How to’s

+
+

Get the Source Code

+

With git installed on the system, just clone the repository:

+
$ export INTEL_DEVICE_PLUGINS_SRC=/path/to/intel-device-plugins-for-kubernetes
+$ git clone https://github.com/intel/intel-device-plugins-for-kubernetes ${INTEL_DEVICE_PLUGINS_SRC}
+
+
+
+
+

Build and Run Plugin Binaries

+

With go development environment installed on the system, build the plugin:

+
$ cd ${INTEL_DEVICE_PLUGINS_SRC}
+$ make <plugin-build-target>
+
+
+

Note: All the available plugin build targets is roughly the output of ls ${INTEL_DEVICE_PLUGINS_SRC}/cmd.

+

To test the plugin binary on the development system, run as administrator:

+
$ sudo -E ${INTEL_DEVICE_PLUGINS_SRC}/cmd/<plugin-build-target>/<plugin-build-target>
+
+
+
+
+

Build Container Images

+

The dockerfiles are generated on the fly from .in suffixed files and .docker include-snippets which are stitched together with +cpp preprocessor. You need to install cpp for that, e.g. in ubuntu it is found from build-essential (sudo apt install build-essential). +Don’t edit the generated dockerfiles. Edit the inputs.

+

The simplest way to build all the docker images, is:

+
$ make images
+
+
+

But it is very slow. You can drastically speed it up by first running once:

+
$ make vendor
+
+
+

Which brings the libraries into the builder container without downloading them again and again for each plugin.

+

But it is still slow. You can further speed it up by first running once:

+
$ make licenses
+
+
+

Which pre-creates the go-licenses for all plugins, instead of re-creating them for each built plugin, every time.

+

But it is still rather slow to build all the images, and unnecessary, if you iterate on just one. Instead, build just the one you are iterating on, example:

+
$ make <image-build-target>
+
+
+

Note: All the available image build targets is roughly the output of ls ${INTEL_DEVICE_PLUGINS_SRC}/build/docker/*.Dockerfile.

+

If you iterate on only one plugin and if you know what its target cmd is (see folder cmd/), you can opt to pre-create just its licenses, example:

+
$ make licenses/<plugin-build-target>
+
+
+

The container image target names in the Makefile are derived from the .Dockerfile.in suffixed filenames under folder build/docker/templates/.

+

Recap:

+
$ make vendor
+$ make licenses (or just make licenses/<plugin-build-target>)
+$ make <image-build-target>
+
+
+

Repeat the last step only, unless you change library dependencies. If you pull in new sources, start again from make vendor.

+

Note: The image build tool can be changed from the default docker by setting the BUILDER argument +to the Makefile: make <image-build-target> BUILDER=<builder>. Supported values are docker, buildah, and podman.

+
+
+

Build Against a Newer Version of Kubernetes

+

First, you need to update module dependencies. The easiest way is to use +scripts/upgrade_k8s.sh copied from a k/k issue:

+

Just run it inside the repo’s root, e.g.

+
$ ${INTEL_DEVICE_PLUGINS_SRC}/scripts/upgrade_k8s.sh <k8s version>
+
+
+

Finally, run:

+
$ make generate
+$ make test
+
+
+

and fix all new compilation issues.

+
+
+

Work with Intel Device Plugins Operator Modifications

+

There are few useful steps when working with changes to Device Plugins CRDs and controllers:

+
    +

  1. Install controller-gen: GO111MODULE=on go get -u sigs.k8s.io/controller-tools/cmd/controller-gen@<release ver>, e.g, v0.4.1

    2. +

  2. Generate CRD and Webhook artifacts: make generate

    3. +

  3. Test local changes using envtest: make envtest

    4. +

  4. Build a custom operator image: make intel-deviceplugin-operator

    5. +

  5. (Un)deploy operator: kubectl [apply|delete] -k deployments/operator/default

    6. +
+
+
+

Publish a New Version of the Intel Device Plugins Operator to operatorhub.io

+

Check if the fields mentioned below in the base CSV manifest file have the correct values. If not, fix them manually (operator-sdk does not support updating these fields in any other way).

+
    +

  • spec.version

    • +

  • spec.replaces

    • +

  • metadata.annotations.containerImage

    • +

  • metadata.annotations.createdAT

    • +
+

Fork the Community Operators repo and clone it:

+
$ git clone https://github.com/<GitHub Username>/community-operators
+
+
+

Generate bundle and build bundle image:

+
$ make bundle TAG=0.X.Y CHANNELS=alpha DEFAULT_CHANNEL=alpha
+$ make bundle-build
+
+
+

Push the image to a registry:

+
    +

  • If pushing to the Docker hub, specify docker.io/ in front of the image name for running bundle.

    • +

  • If pushing to the local registry, put the option --use-http for running bundle.

    • +
+

Verify the operator deployment works OK via OLM in your development cluster:

+
$ operator-sdk olm install
+$ kubectl create namespace testoperator
+$ operator-sdk run bundle <Registry>:<Tag> -n testoperator
+# do verification checks
+...
+# do clean up
+$ operator-sdk cleanup intel-device-plugins-operator --namespace testoperator
+$ kubectl delete namespace testoperator
+$ operator-sdk olm uninstall
+
+
+

Commit files:

+
$ cd community-operators
+$ git add operators/intel-device-plugins-operator/0.X.Y
+$ git commit -am 'operators intel-device-plugins-operator (0.X.Y)' -s
+
+
+

Submit a PR to Community Operators repo.

+

Check operator page +https://operatorhub.io/operator/intel-device-plugins-operator +after PR is merged.

+
+
+

Run E2E Tests

+

Currently the E2E tests require having a Kubernetes cluster already configured +on the nodes with the hardware required by the device plugins. Also all the +container images with the executables under test must be available in the +cluster. If these two conditions are satisfied, run the tests with:

+
$ go test -v ./test/e2e/...
+
+
+

In case you want to run only certain tests, e.g., QAT ones, run:

+
$ go test -v ./test/e2e/... -args -ginkgo.focus "QAT"
+
+
+

If you need to specify paths to your custom kubeconfig containing +embedded authentication info then add the -kubeconfig argument:

+
$ go test -v ./test/e2e/... -args -kubeconfig /path/to/kubeconfig
+
+
+

The full list of available options can be obtained with:

+
$ go test ./test/e2e/... -args -help
+
+
+

It is also possible to run the tests which don’t depend on hardware +without a pre-configured Kubernetes cluster. Just make sure you have +Kind installed on your host and run:

+
$ make test-with-kind
+
+
+
+
+

Run Controller Tests with a Local Control Plane

+

The controller-runtime library provides a package for integration testing by +starting a local control plane. The package is called +envtest. The +operator uses this package for its integration testing.

+

For setting up the environment for testing, setup-envtest can be used:

+
$ go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
+$ setup-envtest use <K8S_VERSION>
+$ KUBEBUILDER_ASSETS=$(setup-envtest use -i -p path <K8S_VERSION>) make envtest
+
+
+
+
+
+

How to Develop Simple Device Plugins

+

To create a simple device plugin without the hassle of developing your own gRPC +server, you can use a package included in this repository called +github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin.

+

All you have to do is instantiate a deviceplugin.Manager and call +its Run() method:

+
func main() {
+    ...
+
+    manager := dpapi.NewManager(namespace, plugin)
+    manager.Run()
+}
+
+
+

The manager’s constructor accepts two parameters:

+
    +

  1. namespace which is a string like “color.example.com”. All your devices +will be exposed under this name space, e.g. “color.example.com/yellow”. +Please note that one device plugin can register many such “colors”. +The manager will instantiate multiple gRPC servers for every registered “color”.

    2. +

  2. plugin which is a reference to an object implementing one mandatory +interface deviceplugin.Scanner.

    3. +
+

deviceplugin.Scanner defines one method Scan() which is called only once +for every device plugin by deviceplugin.Manager in a goroutine and operates +in an infinite loop. A Scan() implementation scans the host for devices and +sends all found devices to a deviceplugin.Notifier instance. The +deviceplugin.Notifier is implemented and provided by the deviceplugin +package itself. The found devices are organized in an instance of +deviceplugin.DeviceTree object. The object is filled in with its +AddDevice() method:

+
func (dp *devicePlugin) Scan(notifier deviceplugin.Notifier) error {
+    for {
+        devTree := deviceplugin.NewDeviceTree()
+        ...
+        devTree.AddDevice("yellow", devID, deviceplugin.DeviceInfo{
+            State: health,
+            Nodes: []pluginapi.DeviceSpec{
+                {
+                    HostPath:      devPath,
+                    ContainerPath: devPath,
+                    Permissions:   "rw",
+                },
+            },
+        })
+        ...
+        notifier.Notify(devTree)
+    }
+}
+
+
+

Optionally, your device plugin may also implement the +deviceplugin.PostAllocator interface. If implemented, its method +PostAllocate() modifies pluginapi.AllocateResponse responses just +before they are sent to kubelet. To see an example, refer to the FPGA +plugin which implements this interface to annotate its responses.

+

In case you want to implement the whole allocation functionality in your +device plugin, you can implement the optional deviceplugin.Allocator +interface. In this case PostAllocate() is not called. But if you decide in your +implementation of deviceplugin.Allocator that you need to resort to the default +implementation of the allocation functionality then return an error of the type +deviceplugin.UseDefaultMethodError.

+
+

Logging

+

The framework uses klog as its logging +framework. It is encouraged for plugins to also use klog to maintain uniformity +in the logs and command line options.

+

The framework initialises klog, so further calls to klog.InitFlags() by +plugins should not be necessary. This does add a number of log configuration +options to your plugin, which can be viewed with the -h command line option of your +plugin.

+

The framework tries to adhere to the Kubernetes +Logging Conventions. +The advise is to use the V() levels for Info() calls, as calling Info() +with no set level will make configuration and filtering of logging via the command +line more difficult.

+

The default is to not log Info() calls. This can be changed using the plugin command +line -v parameter. The additional annotations prepended to log lines by ‘klog’ can be disabled +with the -skip_headers option.

+
+
+

Error Conventions

+

The framework has a convention for producing and logging errors. Ideally plugins will also adhere +to the convention.

+

Errors generated within the framework and plugins are instantiated with the New() and +Errorf() functions of the errors package:

+
    return errors.New("error message")
+
+
+

Errors generated from outside the plugins and framework are augmented with their stack dump with code such as

+
    return errors.WithStack(err)
+
+
+

or

+
    return errors.Wrap(err, "some additional error message")
+
+
+

These errors are then logged using a default struct value format like:

+
    klog.Errorf("Example of an internal error death: %+v", err)
+
+
+

at the line where it’s certain that the error cannot be passed out farther nor handled gracefully. +Otherwise, they can be logged as simple values:

+
    klog.Warningf("Example of a warning due to an external error: %v", err)
+
+
+
+
+
+

Checklist for New Device Plugins

+

For new device plugins contributed to this repository, below is a +checklist to get the plugin on par feature and quality wise with +others:

+
    +

  1. Plugin binary available in cmd/, its corresponding Dockerfile in build/docker/ and deployment Kustomization/YAMLs in deployments/.

    2. +

  2. Plugin binary Go unit tests implemented and passing with >80% coverage: make test WHAT=./cmd/<plugin>.

    3. +

  3. Plugin binary linter checks passing: make lint.

    4. +

  4. Plugin e2e tests implemented in test/e2e/ and passing: go test -v ./test/e2e/... -args -ginkgo.focus "<plugin>".

    5. +

  5. Plugin CRD API added to pkg/apis/deviceplugin/v1 and CRDs generated: make generate.

    6. +

  6. Plugin CRD validation tests implemented in test/envtest/ and passing: make envtest.

    7. +

  7. Plugin CRD controller implemented in pkg/controllers/ and added to the manager in cmd/operator/main.go.

    8. +

  8. Plugin documentation written cmd/<plugin>/README.md and optionally end to end demos created in demo.

    9. +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.27/INSTALL.html b/0.27/INSTALL.html new file mode 100644 index 000000000..bbce74621 --- /dev/null +++ b/0.27/INSTALL.html @@ -0,0 +1,230 @@ + + + + + + + Installing device plugins to cluster — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Installing device plugins to cluster

+
+

Install device plugins via a DaemonSet

+

Each plugin can be installed via a DaemonSet. The install changes slightly based on the desired plugin. See install instructions per plugin.

+

Installing plugins via DaemonSets deployes them to the default (or currently active) namespace. Use kubectl’s --namespace argument to change the deployment namespace.

+
+
+

Install device plugins via device plugin operator

+

A more advanced install method is via device plugin operator. Operator configures plugin deployments based on the supplied device plugin CRDs (Custom Resource Definitions). See installation instructions in the operator README.

+

Operator installs device plugins to the same namespace where the operator itself is deployed. The default operator namespace is inteldeviceplugins-system.

+
+
+

Install with HELM charts

+

Device plugins can also be installed to a cluster using the device plugin operator Helm chart (depending on cert-manager and NFD). Individual plugin projects are under https://github.com/intel/helm-charts/tree/main/charts/.

+

These steps will install device plugin operator and plugins under inteldeviceplugins-system namespace. It’s possible to change the target namespace by changing the --namespace value in the helm install command.

+
+

Installing HELM repositories

+
helm repo add jetstack https://charts.jetstack.io # for cert-manager
+helm repo add nfd https://kubernetes-sigs.github.io/node-feature-discovery/charts # for NFD
+helm repo add intel https://intel.github.io/helm-charts/ # for device-plugin-operator and plugins
+helm repo update
+
+
+
+
+

Installing cert-manager

+
helm install --wait \
+  cert-manager jetstack/cert-manager \
+  --namespace cert-manager \
+  --create-namespace \
+  --version v1.11.0 \
+  --set installCRDs=true
+
+
+

NOTE: cert-manager install takes a while to complete.

+
+
+

Installing NFD

+
helm install nfd nfd/node-feature-discovery \
+  --namespace node-feature-discovery --create-namespace --version 0.12.1 \
+  --set 'master.extraLabelNs={gpu.intel.com,sgx.intel.com}' \
+  --set 'master.resourceLabels={gpu.intel.com/millicores,gpu.intel.com/memory.max,gpu.intel.com/tiles,sgx.intel.com/epc}'
+
+
+
+
+

Installing operator

+
helm install dp-operator intel/intel-device-plugins-operator --namespace inteldeviceplugins-system --create-namespace
+
+
+
+
+

Installing specific plugins

+

Replace PLUGIN with the desired plugin name. At least the following plugins are supported: gpu, sgx, qat, dlb, dsa & iaa.

+
helm install <PLUGIN> intel/intel-device-plugins-<PLUGIN> --namespace inteldeviceplugins-system --create-namespace \
+  --set nodeFeatureRule=true
+
+
+
+
+

Listing available versions

+

Use helm’s search functionality to list available versions.

+
helm search repo intel/intel-device-plugins-operator --versions
+helm search repo intel/intel-device-plugins-<plugin> --versions
+
+
+

For example, operator chart versions with development versions included.

+
$ helm search repo intel/intel-device-plugins-operator --versions --devel
+NAME                               	CHART VERSION	APP VERSION	DESCRIPTION
+intel/intel-device-plugins-operator	0.26.0       	0.26.0     	A Helm chart for Intel Device Plugins Operator ...
+intel/intel-device-plugins-operator	0.25.1       	0.25.1     	A Helm chart for Intel Device Plugins Operator ...
+intel/intel-device-plugins-operator	0.25.1-helm.0	0.25.0     	A Helm chart for Intel Device Plugins Operator ...
+intel/intel-device-plugins-operator	0.25.0       	0.25.0     	A Helm chart for Intel Device Plugins Operator ...
+intel/intel-device-plugins-operator	0.24.1       	0.24.1     	A Helm chart for Intel Device Plugins Operator ...
+intel/intel-device-plugins-operator	0.24.1-helm.0	0.24.0     	A Helm chart for Intel Device Plugins Operator ...
+intel/intel-device-plugins-operator	0.24.0       	0.24.0     	A Helm chart for Intel Device Plugins Operator ...
+
+
+
+
+

Customizing plugins

+

To customize plugin features, see the available chart values:

+
helm show values intel/intel-device-plugins-<PLUGIN>
+
+
+

For example, qat plugin has these values:

+
$ helm show values intel/intel-device-plugins-qat
+name: qatdeviceplugin-sample
+
+image:
+  hub: intel
+  tag: ""
+
+initImage:
+  hub: intel
+  tag: ""
+
+dpdkDriver: vfio-pci
+kernelVfDrivers:
+  - c6xxvf
+  - 4xxxvf
+maxNumDevices: 128
+logLevel: 4
+
+nodeSelector:
+  intel.feature.node.kubernetes.io/qat: 'true'
+
+nodeFeatureRule: true
+
+
+
+
+

Uninstall

+

Uninstall each installed component with helm uninstall:

+
# repeat first step as many times as there are plugins installed
+helm uninstall -n inteldeviceplugins-system <PLUGIN>
+helm uninstall -n inteldeviceplugins-system dp-operator
+helm uninstall -n node-feature-discovery nfd
+helm uninstall -n cert-manager cert-manager
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.27/README.html b/0.27/README.html new file mode 100644 index 000000000..50cc57616 --- /dev/null +++ b/0.27/README.html @@ -0,0 +1,524 @@ + + + + + + + Overview — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Overview

+

Build Status +Go Report Card +GoDoc

+

This repository contains a framework for developing plugins for the Kubernetes +device plugins framework, +along with a number of device plugin implementations utilizing that framework.

+

The v0.27.1 release +is the latest feature release with its documentation available here.

+

Table of Contents

+ +
+

Prerequisites

+

Prerequisites for building and running these device plugins include:

+ +
+
+

Plugins

+

The below sections detail existing plugins developed using the framework.

+
+

GPU Device Plugin

+

The GPU device plugin provides access to +discrete and integrated Intel GPU device files.

+

The demo subdirectory contains both a GPU plugin demo video +and an OpenCL sample deployment (intelgpu-job.yaml).

+
+
+

FPGA Device Plugin

+

The FPGA device plugin supports FPGA passthrough for +the following hardware:

+
    +

  • Intel® Arria® 10 devices

    • +

  • Intel® Stratix® 10 devices

    • +
+

The FPGA plugin comes as three parts.

+ +

Refer to each individual sub-components documentation for more details. +Brief overviews of the sub-components are below.

+

The demo subdirectory contains a +video showing deployment +and use of the FPGA plugin. Sources relating to the demo can be found in the +opae-nlb-demo subdirectory.

+
+

Device Plugin

+

The FPGA device plugin is responsible for +discovering and reporting FPGA devices to kubelet.

+
+
+

Admission Controller

+

The FPGA admission controller webhook +is responsible for performing mapping from user-friendly function IDs to the +Interface ID and Bitstream ID that are required for FPGA programming. It also +implements access control by namespacing FPGA configuration information.

+
+
+

CRI-O Prestart Hook

+

The FPGA prestart CRI-O hook performs discovery +of the requested FPGA function bitstream and programs FPGA devices based on the +environment variables in the workload description.

+
+
+
+

QAT Device Plugin

+

The QAT plugin supports device plugin for Intel QAT adapters, and includes +code showing deployment via DPDK.

+

The demo subdirectory includes details of both a +QAT DPDK demo +and a QAT OpenSSL demo. +Source for the OpenSSL demo can be found in the relevant subdirectory.

+

Details for integrating the QAT device plugin into Kata Containers +can be found in the +Kata Containers documentation repository.

+
+
+

VPU Device Plugin

+

The VPU device plugin supports Intel VCAC-A card +(https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/media-analytics-vcac-a-accelerator-card-by-celestica-datasheet.pdf) +the card has:

+
    +

  • 1 Intel Core i3-7100U processor

    • +

  • 12 MyriadX VPUs

    • +

  • 8GB DDR4 memory

    • +
+

The demo subdirectory includes details of a OpenVINO deployment and use of the +VPU plugin. Sources can be found in openvino-demo.

+
+
+

SGX Device Plugin

+

The SGX device plugin allows workloads to use +Intel® Software Guard Extensions (Intel® SGX) on +platforms with SGX Flexible Launch Control enabled, e.g.,:

+
    +

  • 3rd Generation Intel® Xeon® Scalable processor family, code-named “Ice Lake”

    • +

  • Intel® Xeon® E3 processor

    • +

  • Intel® NUC Kit NUC7CJYH

    • +
+

The Intel SGX plugin comes in three parts.

+ +

The demo subdirectory contains a video showing the deployment +and use of the Intel SGX device plugin. Sources relating to the demo can be found in the +sgx-sdk-demo and sgx-aesmd-demo subdirectories.

+

Brief overviews of the Intel SGX sub-components are given below.

+

+
+

device plugin

+

The SGX device plugin is responsible for discovering +and reporting Intel SGX device nodes to kubelet.

+

Containers requesting Intel SGX resources in the cluster should not use the +device plugins resources directly.

+
+
+

Intel SGX Admission Webhook

+

The Intel SGX admission webhook is responsible for performing Pod mutations based on +the sgx.intel.com/quote-provider pod annotation set by the user. The purpose +of the webhook is to hide the details of setting the necessary device resources +and volume mounts for using Intel SGX remote attestation in the cluster. Furthermore, +the Intel SGX admission webhook is responsible for writing a pod/sandbox +sgx.intel.com/epc annotation that is used by Kata Containers to dynamically +adjust its virtualized Intel SGX encrypted page cache (EPC) bank(s) size.

+

The Intel SGX admission webhook is available as part of +Intel Device Plugin Operator or +as a standalone SGX Admission webhook image.

+
+
+

Intel SGX EPC memory registration

+

The Intel SGX EPC memory available on each node is registered as a Kubernetes extended resource using +node-feature-discovery (NFD). An NFD Node Feature Rule is installed as part of +SGX device plugin +operator deployment and NFD is configured to register the Intel SGX EPC memory +extended resource.

+

Containers requesting Intel SGX EPC resources in the cluster use +sgx.intel.com/epc resource which is of +type memory.

+
+
+
+

DSA Device Plugin

+

The DSA device plugin supports acceleration using +the Intel Data Streaming accelerator(DSA).

+
+
+

DLB Device Plugin

+

The DLB device plugin supports Intel Dynamic Load +Balancer accelerator(DLB).

+
+
+

IAA Device Plugin

+

The IAA device plugin supports acceleration using +the Intel Analytics accelerator(IAA).

+
+
+
+

Device Plugins Operator

+

To simplify the deployment of the device plugins, a unified device plugins +operator is implemented.

+

Currently the operator has support for the DSA, DLB, FPGA, GPU, IAA, QAT, and +Intel SGX device plugins. Each device plugin has its own custom resource +definition (CRD) and the corresponding controller that watches CRUD operations +to those custom resources.

+

The Device plugins operator README gives the installation and usage details for the community operator available on operatorhub.io.

+

The Device plugins Operator for OCP gives the installation and usage details for the operator available on Red Hat OpenShift Container Platform.

+
+ +
+

Demos

+

The demo subdirectory contains a number of demonstrations for +a variety of the available plugins.

+
+
+

Workload Authors

+

For workloads to get accesss to devices managed by the plugins, the +Pod spec must specify the hardware resources needed:

+
spec:
+  containers:
+    - name: demo-container
+      image: <registry>/<image>:<version>
+      resources:
+        limits:
+          <device namespace>/<resource>: X
+
+
+

The summary of resources available via plugins in this repository is given in the list below.

+

Device Namespace : Registered Resource(s)

+ +
+
+

Developers

+

For information on how to develop a new plugin using the framework or work on development task in +this repository, see the Developers Guide.

+
+
+

Supported Kubernetes Versions

+

Releases are made under the github releases area. Supported releases and +matching Kubernetes versions are listed below:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
BranchKubernetes branch/versionStatus
release-0.27Kubernetes 1.27 branch v1.27.xsupported
release-0.26Kubernetes 1.26 branch v1.26.xsupported
release-0.25Kubernetes 1.25 branch v1.25.xsupported
release-0.24Kubernetes 1.24 branch v1.24.xunsupported
release-0.23Kubernetes 1.23 branch v1.23.xunsupported
release-0.22Kubernetes 1.22 branch v1.22.xunsupported
release-0.21Kubernetes 1.21 branch v1.21.xunsupported
release-0.20Kubernetes 1.20 branch v1.20.xunsupported
release-0.19Kubernetes 1.19 branch v1.19.xunsupported
release-0.18Kubernetes 1.18 branch v1.18.xunsupported
release-0.17Kubernetes 1.17 branch v1.17.xunsupported
release-0.15Kubernetes 1.15 branch v1.15.xunsupported
release-0.11Kubernetes 1.11 branch v1.11.xunsupported
+
+

Pre-built plugin images

+

Pre-built images of the plugins are available on the Docker hub. These images +are automatically built and uploaded to the hub from the latest main branch of +this repository.

+

Release tagged images of the components are also available on the Docker hub, +tagged with their release version numbers in the format x.y.z, corresponding to +the branches and releases in this repository.

+

Note: the default deployment files and operators are configured with +imagePullPolicy +IfNotPresent and can be changed with scripts/set-image-pull-policy.sh.

+
+
+

License

+

All of the source code required to build intel-device-plugins-for-kubernetes +is available under Open Source licenses. The source code files identify external Go +modules used. Binaries are distributed as container images on +DockerHub*. Those images contain license texts and source code under /licenses.

+
+

Helm Charts

+

Device Plugins Helm Charts are located in Intel Helm Charts repository Intel Helm Charts. This is another way of distributing Kubernetes resources of the device plugins framework.

+

To add repo:

+
helm repo add intel https://intel.github.io/helm-charts
+
+
+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.27/SECURITY.html b/0.27/SECURITY.html new file mode 100644 index 000000000..eaf00b536 --- /dev/null +++ b/0.27/SECURITY.html @@ -0,0 +1,118 @@ + + + + + + + <no title> — Intel® Device Plugins for Kubernetes documentation + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +

Reporting a Potential Security Vulnerability: If you have discovered +potential security vulnerability in this project, please send an e-mail to +secure@intel.com. Encrypt sensitive information using our +PGP public key.

+

Please provide as much information as possible, including:

+
    +

  • The projects and versions affected

    • +

  • Detailed description of the vulnerability

    • +

  • Information on known exploits

    • +
+

A member of the Intel Product Security Team will review your e-mail and +contact you to collaborate on resolving the issue. For more information on +how Intel works to resolve security issues, see Vulnerability Handling Guidelines.

+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/0.27/_images/FPGA-af.png b/0.27/_images/FPGA-af.png new file mode 100644 index 000000000..64934fdfa Binary files /dev/null and b/0.27/_images/FPGA-af.png differ diff --git a/0.27/_images/FPGA-region.png b/0.27/_images/FPGA-region.png new file mode 100644 index 000000000..b57d9a1ed Binary files /dev/null and b/0.27/_images/FPGA-region.png differ diff --git a/0.27/_images/SGX-BIOS.PNG b/0.27/_images/SGX-BIOS.PNG new file mode 100644 index 000000000..f8510ecec Binary files /dev/null and b/0.27/_images/SGX-BIOS.PNG differ diff --git a/0.27/_images/verify-operator.PNG b/0.27/_images/verify-operator.PNG new file mode 100644 index 000000000..55adad1b5 Binary files /dev/null and b/0.27/_images/verify-operator.PNG differ diff --git a/0.27/_sources/DEVEL.md.txt b/0.27/_sources/DEVEL.md.txt new file mode 100644 index 000000000..900dc6337 --- /dev/null +++ b/0.27/_sources/DEVEL.md.txt @@ -0,0 +1,369 @@ +# Instructions for Device Plugin Development and Maintenance + +Table of Contents + +* [Day-to-day Development How to's](#day-to-day-development) + * [Get the Source Code](#get-the-source-code) + * [Build and Run Plugin Binaries](#build-and-run-plugin-binaries) + * [Build Container Images](#build-container-images) + * [Build Against a Newer Version of Kubernetes](#build-against-a-newer-version-of-kubernetes) + * [Work with Intel Device Plugins Operator Modifications](#work-with-intel-device-plugins-operator-modifications) + * [Publish a New Version of the Intel Device Plugins Operator to operatorhub.io](#publish-a-new-version-of-the-intel-device-plugins-operator-to-operatorhubio) + * [Run E2E Tests](#run-e2e-tests) + * [Run Controller Tests with a Local Control Plane](#run-controller-tests-with-a-local-control-plane) +* [How to Develop Simple Device Plugins](#how-to-develop-simple-device-plugins) + * [Logging](#logging) + * [Error Conventions](#error-conventions) +* [Checklist for New Device Plugins](#checklist-for-new-device-plugins) + +## Day-to-day Development How to's +### Get the Source Code + +With `git` installed on the system, just clone the repository: + +```bash +$ export INTEL_DEVICE_PLUGINS_SRC=/path/to/intel-device-plugins-for-kubernetes +$ git clone https://github.com/intel/intel-device-plugins-for-kubernetes ${INTEL_DEVICE_PLUGINS_SRC} +``` + +### Build and Run Plugin Binaries + +With `go` development environment installed on the system, build the plugin: + +```bash +$ cd ${INTEL_DEVICE_PLUGINS_SRC} +$ make +``` + +**Note:** All the available plugin build targets is roughly the output of `ls ${INTEL_DEVICE_PLUGINS_SRC}/cmd`. + +To test the plugin binary on the development system, run as administrator: + +```bash +$ sudo -E ${INTEL_DEVICE_PLUGINS_SRC}/cmd// +``` + +### Build Container Images + +The dockerfiles are generated on the fly from `.in` suffixed files and `.docker` include-snippets which are stitched together with +cpp preprocessor. You need to install cpp for that, e.g. in ubuntu it is found from build-essential (sudo apt install build-essential). +Don't edit the generated dockerfiles. Edit the inputs. + +The simplest way to build all the docker images, is: +``` +$ make images +``` + +But it is very slow. You can drastically speed it up by first running once: +``` +$ make vendor +``` + +Which brings the libraries into the builder container without downloading them again and again for each plugin. + +But it is still slow. You can further speed it up by first running once: +``` +$ make licenses +``` + +Which pre-creates the go-licenses for all plugins, instead of re-creating them for each built plugin, every time. + +But it is still rather slow to build all the images, and unnecessary, if you iterate on just one. Instead, build just the one you are iterating on, example: + +``` +$ make +``` + +**Note:** All the available image build targets is roughly the output of `ls ${INTEL_DEVICE_PLUGINS_SRC}/build/docker/*.Dockerfile`. + +If you iterate on only one plugin and if you know what its target cmd is (see folder `cmd/`), you can opt to pre-create just its licenses, example: +``` +$ make licenses/ +``` + +The container image target names in the Makefile are derived from the `.Dockerfile.in` suffixed filenames under folder `build/docker/templates/`. + +Recap: +``` +$ make vendor +$ make licenses (or just make licenses/) +$ make +``` + +Repeat the last step only, unless you change library dependencies. If you pull in new sources, start again from `make vendor`. + +**Note:** The image build tool can be changed from the default `docker` by setting the `BUILDER` argument +to the [`Makefile`](Makefile): `make BUILDER=`. Supported values are `docker`, `buildah`, and `podman`. + +### Build Against a Newer Version of Kubernetes + +First, you need to update module dependencies. The easiest way is to use +`scripts/upgrade_k8s.sh` copied [from a k/k issue](https://github.com/kubernetes/kubernetes/issues/79384#issuecomment-521493597): + +Just run it inside the repo's root, e.g. + +``` +$ ${INTEL_DEVICE_PLUGINS_SRC}/scripts/upgrade_k8s.sh +``` +Finally, run: + +``` +$ make generate +$ make test +``` + +and fix all new compilation issues. + +### Work with Intel Device Plugins Operator Modifications + +There are few useful steps when working with changes to Device Plugins CRDs and controllers: + +1. Install controller-gen: `GO111MODULE=on go get -u sigs.k8s.io/controller-tools/cmd/controller-gen@, e.g, v0.4.1` +2. Generate CRD and Webhook artifacts: `make generate` +3. Test local changes using [envtest](https://book.kubebuilder.io/reference/envtest.html): `make envtest` +4. Build a custom operator image: `make intel-deviceplugin-operator` +5. (Un)deploy operator: `kubectl [apply|delete] -k deployments/operator/default` + +### Publish a New Version of the Intel Device Plugins Operator to operatorhub.io + +Check if the fields mentioned below in the [base CSV manifest file](deployments/operator/manifests/bases/intel-device-plugins-operator.clusterserviceversion.yaml) have the correct values. If not, fix them manually (operator-sdk does not support updating these fields in any other way). +- spec.version +- spec.replaces +- metadata.annotations.containerImage +- metadata.annotations.createdAT + +Fork the [Community Operators](https://github.com/k8s-operatorhub/community-operators) repo and clone it: +``` +$ git clone https://github.com//community-operators +``` + +Generate bundle and build bundle image: +``` +$ make bundle TAG=0.X.Y CHANNELS=alpha DEFAULT_CHANNEL=alpha +$ make bundle-build +``` + +Push the image to a registry: +- If pushing to the Docker hub, specify `docker.io/` in front of the image name for running bundle. +- If pushing to the local registry, put the option `--use-http` for running bundle. + +Verify the operator deployment works OK via OLM in your development cluster: +``` +$ operator-sdk olm install +$ kubectl create namespace testoperator +$ operator-sdk run bundle : -n testoperator +# do verification checks +... +# do clean up +$ operator-sdk cleanup intel-device-plugins-operator --namespace testoperator +$ kubectl delete namespace testoperator +$ operator-sdk olm uninstall +``` + +Commit files: +``` +$ cd community-operators +$ git add operators/intel-device-plugins-operator/0.X.Y +$ git commit -am 'operators intel-device-plugins-operator (0.X.Y)' -s +``` + +Submit a PR to [Community Operators](https://github.com/k8s-operatorhub/community-operators) repo. + +Check operator page +https://operatorhub.io/operator/intel-device-plugins-operator +after PR is merged. + +### Run E2E Tests + +Currently the E2E tests require having a Kubernetes cluster already configured +on the nodes with the hardware required by the device plugins. Also all the +container images with the executables under test must be available in the +cluster. If these two conditions are satisfied, run the tests with: + +```bash +$ go test -v ./test/e2e/... +``` + +In case you want to run only certain tests, e.g., QAT ones, run: + +```bash +$ go test -v ./test/e2e/... -args -ginkgo.focus "QAT" +``` + +If you need to specify paths to your custom `kubeconfig` containing +embedded authentication info then add the `-kubeconfig` argument: + +```bash +$ go test -v ./test/e2e/... -args -kubeconfig /path/to/kubeconfig +``` + +The full list of available options can be obtained with: + +```bash +$ go test ./test/e2e/... -args -help +``` + +It is also possible to run the tests which don't depend on hardware +without a pre-configured Kubernetes cluster. Just make sure you have +[Kind](https://kind.sigs.k8s.io/) installed on your host and run: + +``` +$ make test-with-kind +``` + +### Run Controller Tests with a Local Control Plane + +The controller-runtime library provides a package for integration testing by +starting a local control plane. The package is called +[envtest](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest). The +operator uses this package for its integration testing. + +For setting up the environment for testing, `setup-envtest` can be used: + +```bash +$ go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest +$ setup-envtest use +$ KUBEBUILDER_ASSETS=$(setup-envtest use -i -p path ) make envtest +``` +## How to Develop Simple Device Plugins + +To create a simple device plugin without the hassle of developing your own gRPC +server, you can use a package included in this repository called +`github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin`. + +All you have to do is instantiate a `deviceplugin.Manager` and call +its `Run()` method: + +```go +func main() { + ... + + manager := dpapi.NewManager(namespace, plugin) + manager.Run() +} +``` + +The manager's constructor accepts two parameters: + +1. `namespace` which is a string like "color.example.com". All your devices + will be exposed under this name space, e.g. "color.example.com/yellow". + Please note that one device plugin can register many such "colors". + The manager will instantiate multiple gRPC servers for every registered "color". +2. `plugin` which is a reference to an object implementing one mandatory + interface `deviceplugin.Scanner`. + +`deviceplugin.Scanner` defines one method `Scan()` which is called only once +for every device plugin by `deviceplugin.Manager` in a goroutine and operates +in an infinite loop. A `Scan()` implementation scans the host for devices and +sends all found devices to a `deviceplugin.Notifier` instance. The +`deviceplugin.Notifier` is implemented and provided by the `deviceplugin` +package itself. The found devices are organized in an instance of +`deviceplugin.DeviceTree` object. The object is filled in with its +`AddDevice()` method: + +```go +func (dp *devicePlugin) Scan(notifier deviceplugin.Notifier) error { + for { + devTree := deviceplugin.NewDeviceTree() + ... + devTree.AddDevice("yellow", devID, deviceplugin.DeviceInfo{ + State: health, + Nodes: []pluginapi.DeviceSpec{ + { + HostPath: devPath, + ContainerPath: devPath, + Permissions: "rw", + }, + }, + }) + ... + notifier.Notify(devTree) + } +} +``` + +Optionally, your device plugin may also implement the +`deviceplugin.PostAllocator` interface. If implemented, its method +`PostAllocate()` modifies `pluginapi.AllocateResponse` responses just +before they are sent to `kubelet`. To see an example, refer to the FPGA +plugin which implements this interface to annotate its responses. + +In case you want to implement the whole allocation functionality in your +device plugin, you can implement the optional `deviceplugin.Allocator` +interface. In this case `PostAllocate()` is not called. But if you decide in your +implementation of `deviceplugin.Allocator` that you need to resort to the default +implementation of the allocation functionality then return an error of the type +`deviceplugin.UseDefaultMethodError`. + +### Logging + +The framework uses [`klog`](https://github.com/kubernetes/klog) as its logging +framework. It is encouraged for plugins to also use `klog` to maintain uniformity +in the logs and command line options. + +The framework initialises `klog`, so further calls to `klog.InitFlags()` by +plugins should not be necessary. This does add a number of log configuration +options to your plugin, which can be viewed with the `-h` command line option of your +plugin. + +The framework tries to adhere to the Kubernetes +[Logging Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md). +The advise is to use the `V()` levels for `Info()` calls, as calling `Info()` +with no set level will make configuration and filtering of logging via the command +line more difficult. + +The default is to not log `Info()` calls. This can be changed using the plugin command +line `-v` parameter. The additional annotations prepended to log lines by 'klog' can be disabled +with the `-skip_headers` option. + +### Error Conventions + +The framework has a convention for producing and logging errors. Ideally plugins will also adhere +to the convention. + +Errors generated within the framework and plugins are instantiated with the `New()` and +`Errorf()` functions of the [errors package](https://golang.org/pkg/errors/): + +```golang + return errors.New("error message") +``` + +Errors generated from outside the plugins and framework are augmented with their stack dump with code such as + +```golang + return errors.WithStack(err) +``` + +or + +```golang + return errors.Wrap(err, "some additional error message") +``` + +These errors are then logged using a default struct value format like: + +```golang + klog.Errorf("Example of an internal error death: %+v", err) +``` + +at the line where it's certain that the error cannot be passed out farther nor handled gracefully. +Otherwise, they can be logged as simple values: + +```golang + klog.Warningf("Example of a warning due to an external error: %v", err) +``` + +## Checklist for New Device Plugins + +For new device plugins contributed to this repository, below is a +checklist to get the plugin on par feature and quality wise with +others: + +1. Plugin binary available in [`cmd/`](cmd), its corresponding Dockerfile in [`build/docker/`](build/docker) and deployment Kustomization/YAMLs in [`deployments/`](deployments). +2. Plugin binary Go unit tests implemented and passing with >80% coverage: `make test WHAT=./cmd/`. +3. Plugin binary linter checks passing: `make lint`. +4. Plugin e2e tests implemented in [`test/e2e/`](test/e2e) and passing: `go test -v ./test/e2e/... -args -ginkgo.focus ""`. +5. Plugin CRD API added to [`pkg/apis/deviceplugin/v1`](pkg/apis/deviceplugin/v1) and CRDs generated: `make generate`. +6. Plugin CRD validation tests implemented in [`test/envtest/`](test/envtest) and passing: `make envtest`. +7. Plugin CRD controller implemented in [`pkg/controllers/`](pkg/controllers) and added to the manager in `cmd/operator/main.go`. +8. Plugin documentation written `cmd//README.md` and optionally end to end demos created in [`demo`](demo). diff --git a/0.27/_sources/INSTALL.md.txt b/0.27/_sources/INSTALL.md.txt new file mode 100644 index 000000000..087cb6f5a --- /dev/null +++ b/0.27/_sources/INSTALL.md.txt @@ -0,0 +1,132 @@ +# Installing device plugins to cluster + +## Install device plugins via a DaemonSet + +Each plugin can be installed via a DaemonSet. The install changes slightly based on the desired plugin. See install instructions per [plugin](README.md#plugins). + +Installing plugins via DaemonSets deployes them to the ```default``` (or currently active) namespace. Use kubectl's ```--namespace``` argument to change the deployment namespace. + +## Install device plugins via device plugin operator + +A more advanced install method is via device plugin operator. Operator configures plugin deployments based on the supplied device plugin CRDs (Custom Resource Definitions). See installation instructions in the [operator README](cmd/operator/README.md#installation). + +Operator installs device plugins to the same namespace where the operator itself is deployed. The default operator namespace is ```inteldeviceplugins-system```. + +## Install with HELM charts + +Device plugins can also be installed to a cluster using the device plugin [operator Helm chart](https://github.com/intel/helm-charts/tree/main/charts/device-plugin-operator) (depending on cert-manager and NFD). Individual plugin projects are under https://github.com/intel/helm-charts/tree/main/charts/. + +These steps will install device plugin operator and plugins under ```inteldeviceplugins-system``` namespace. It's possible to change the target namespace by changing the ```--namespace``` value in the helm install command. + +### Installing HELM repositories + +```bash +helm repo add jetstack https://charts.jetstack.io # for cert-manager +helm repo add nfd https://kubernetes-sigs.github.io/node-feature-discovery/charts # for NFD +helm repo add intel https://intel.github.io/helm-charts/ # for device-plugin-operator and plugins +helm repo update +``` + +### Installing cert-manager + +```bash +helm install --wait \ + cert-manager jetstack/cert-manager \ + --namespace cert-manager \ + --create-namespace \ + --version v1.11.0 \ + --set installCRDs=true +``` + +NOTE: cert-manager install takes a while to complete. + +### Installing NFD + +```bash +helm install nfd nfd/node-feature-discovery \ + --namespace node-feature-discovery --create-namespace --version 0.12.1 \ + --set 'master.extraLabelNs={gpu.intel.com,sgx.intel.com}' \ + --set 'master.resourceLabels={gpu.intel.com/millicores,gpu.intel.com/memory.max,gpu.intel.com/tiles,sgx.intel.com/epc}' +``` + +### Installing operator + +```bash +helm install dp-operator intel/intel-device-plugins-operator --namespace inteldeviceplugins-system --create-namespace +``` + +### Installing specific plugins + +Replace PLUGIN with the desired plugin name. At least the following plugins are supported: **gpu, sgx, qat, dlb, dsa & iaa**. + +```bash +helm install intel/intel-device-plugins- --namespace inteldeviceplugins-system --create-namespace \ + --set nodeFeatureRule=true +``` + +### Listing available versions + +Use helm's search functionality to list available versions. + +```bash +helm search repo intel/intel-device-plugins-operator --versions +helm search repo intel/intel-device-plugins- --versions +``` + +For example, operator chart versions with development versions included. +```bash +$ helm search repo intel/intel-device-plugins-operator --versions --devel +NAME CHART VERSION APP VERSION DESCRIPTION +intel/intel-device-plugins-operator 0.26.0 0.26.0 A Helm chart for Intel Device Plugins Operator ... +intel/intel-device-plugins-operator 0.25.1 0.25.1 A Helm chart for Intel Device Plugins Operator ... +intel/intel-device-plugins-operator 0.25.1-helm.0 0.25.0 A Helm chart for Intel Device Plugins Operator ... +intel/intel-device-plugins-operator 0.25.0 0.25.0 A Helm chart for Intel Device Plugins Operator ... +intel/intel-device-plugins-operator 0.24.1 0.24.1 A Helm chart for Intel Device Plugins Operator ... +intel/intel-device-plugins-operator 0.24.1-helm.0 0.24.0 A Helm chart for Intel Device Plugins Operator ... +intel/intel-device-plugins-operator 0.24.0 0.24.0 A Helm chart for Intel Device Plugins Operator ... +``` + +### Customizing plugins + +To customize plugin features, see the available chart values: +```bash +helm show values intel/intel-device-plugins- +``` + +For example, qat plugin has these values: +```bash +$ helm show values intel/intel-device-plugins-qat +name: qatdeviceplugin-sample + +image: + hub: intel + tag: "" + +initImage: + hub: intel + tag: "" + +dpdkDriver: vfio-pci +kernelVfDrivers: + - c6xxvf + - 4xxxvf +maxNumDevices: 128 +logLevel: 4 + +nodeSelector: + intel.feature.node.kubernetes.io/qat: 'true' + +nodeFeatureRule: true +``` + +### Uninstall + +Uninstall each installed component with ```helm uninstall```: + +```bash +# repeat first step as many times as there are plugins installed +helm uninstall -n inteldeviceplugins-system +helm uninstall -n inteldeviceplugins-system dp-operator +helm uninstall -n node-feature-discovery nfd +helm uninstall -n cert-manager cert-manager +``` \ No newline at end of file diff --git a/0.27/_sources/README.md.txt b/0.27/_sources/README.md.txt new file mode 100644 index 000000000..8712b84d9 --- /dev/null +++ b/0.27/_sources/README.md.txt @@ -0,0 +1,310 @@ +# Overview +[![Build Status](https://github.com/intel/intel-device-plugins-for-kubernetes/workflows/CI/badge.svg?branch=main)](https://github.com/intel/intel-device-plugins-for-kubernetes/actions?query=workflow%3ACI) +[![Go Report Card](https://goreportcard.com/badge/github.com/intel/intel-device-plugins-for-kubernetes)](https://goreportcard.com/report/github.com/intel/intel-device-plugins-for-kubernetes) +[![GoDoc](https://godoc.org/github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin?status.svg)](https://godoc.org/github.com/intel/intel-device-plugins-for-kubernetes/pkg/deviceplugin) + +This repository contains a framework for developing plugins for the Kubernetes +[device plugins framework](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/), +along with a number of device plugin implementations utilizing that framework. + +The [v0.27.1 release](https://github.com/intel/intel-device-plugins-for-kubernetes/releases/latest) +is the latest feature release with its documentation available [here](https://intel.github.io/intel-device-plugins-for-kubernetes/0.27/). + +Table of Contents + +* [Prerequisites](#prerequisites) +* [Plugins](#plugins) + * [GPU device plugin](#gpu-device-plugin) + * [FPGA device plugin](#fpga-device-plugin) + * [QAT device plugin](#qat-device-plugin) + * [VPU device plugin](#vpu-device-plugin) + * [SGX device plugin](#sgx-device-plugin) + * [DSA device plugin](#dsa-device-plugin) + * [DLB device plugin](#dlb-device-plugin) + * [IAA device plugin](#iaa-device-plugin) +* [Device Plugins Operator](#device-plugins-operator) +* [XeLink XPU-Manager sidecar](#xelink-xpu-manager-sidecar) +* [Demos](#demos) +* [Workload Authors](#workload-authors) +* [Developers](#developers) +* [Supported Kubernetes versions](#supported-kubernetes-versions) +* [Pre-built plugin images](#pre-built-plugin-images) +* [License](#license) +* [Helm charts](#helm-charts) + +## Prerequisites + +Prerequisites for building and running these device plugins include: + +- Appropriate hardware and drivers +- A fully configured [Kubernetes cluster] +- A working [Go environment], of at least version v1.16. + +## Plugins + +The below sections detail existing plugins developed using the framework. + +### GPU Device Plugin + +The [GPU device plugin](cmd/gpu_plugin/README.md) provides access to +discrete and integrated Intel GPU device files. + +The demo subdirectory contains both a [GPU plugin demo video](demo/readme.md#intel-gpu-device-plugin-demo-video) +and an OpenCL sample deployment (`intelgpu-job.yaml`). + +### FPGA Device Plugin + +The [FPGA device plugin](cmd/fpga_plugin/README.md) supports FPGA passthrough for +the following hardware: + +- Intel® Arria® 10 devices +- Intel® Stratix® 10 devices + +The FPGA plugin comes as three parts. + +- the [device plugin](#device-plugin) +- the [admission controller](#admission-controller) +- the [CRIO-O prestart hook](#cri-o-prestart-hook) + +Refer to each individual sub-components documentation for more details. +Brief overviews of the sub-components are below. + +The demo subdirectory contains a +[video](demo/readme.md#intel-fpga-device-plugin-demo-video) showing deployment +and use of the FPGA plugin. Sources relating to the demo can be found in the +[opae-nlb-demo](demo/opae-nlb-demo) subdirectory. + +#### Device Plugin + +The [FPGA device plugin](cmd/fpga_plugin/README.md) is responsible for +discovering and reporting FPGA devices to `kubelet`. + +#### Admission Controller + +The [FPGA admission controller webhook](cmd/fpga_admissionwebhook/README.md) +is responsible for performing mapping from user-friendly function IDs to the +Interface ID and Bitstream ID that are required for FPGA programming. It also +implements access control by namespacing FPGA configuration information. + +#### CRI-O Prestart Hook + +The [FPGA prestart CRI-O hook](cmd/fpga_crihook/README.md) performs discovery +of the requested FPGA function bitstream and programs FPGA devices based on the +environment variables in the workload description. + +### [QAT](https://developer.intel.com/quickassist) Device Plugin + +The [QAT plugin](cmd/qat_plugin/README.md) supports device plugin for Intel QAT adapters, and includes +code [showing deployment](cmd/qat_plugin/dpdkdrv) via [DPDK](https://doc.dpdk.org/guides/cryptodevs/qat.html). + +The demo subdirectory includes details of both a +[QAT DPDK demo](demo/readme.md#intel-quickassist-technology-device-plugin-with-dpdk-demo-video) +and a [QAT OpenSSL demo](demo/readme.md#intel-quickassist-technology-device-plugin-openssl-demo-video). +Source for the OpenSSL demo can be found in the [relevant subdirectory](demo/openssl-qat-engine). + +Details for integrating the QAT device plugin into [Kata Containers](https://katacontainers.io/) +can be found in the +[Kata Containers documentation repository](https://github.com/kata-containers/kata-containers/blob/main/docs/use-cases/using-Intel-QAT-and-kata.md). + +### VPU Device Plugin + +The [VPU device plugin](cmd/vpu_plugin/README.md) supports Intel VCAC-A card +(https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/media-analytics-vcac-a-accelerator-card-by-celestica-datasheet.pdf) +the card has: +- 1 Intel Core i3-7100U processor +- 12 MyriadX VPUs +- 8GB DDR4 memory + +The demo subdirectory includes details of a OpenVINO deployment and use of the +VPU plugin. Sources can be found in [openvino-demo](demo/ubuntu-demo-openvino). + +### SGX Device Plugin + +The [SGX device plugin](cmd/sgx_plugin/README.md) allows workloads to use +Intel® Software Guard Extensions (Intel® SGX) on +platforms with SGX Flexible Launch Control enabled, e.g.,: + +- 3rd Generation Intel® Xeon® Scalable processor family, code-named “Ice Lake” +- Intel® Xeon® E3 processor +- Intel® NUC Kit NUC7CJYH + +The Intel SGX plugin comes in three parts. + +- the [device plugin](#sgx-plugin) +- the [admission webhook](#sgx-admission-webhook) +- the [SGX EPC memory registration](#sgx-epc-memory-registration) + +The demo subdirectory contains a [video](demo/readme.md#intel-sgx-device-plugin-demo-video) showing the deployment +and use of the Intel SGX device plugin. Sources relating to the demo can be found in the +[sgx-sdk-demo](demo/sgx-sdk-demo) and [sgx-aesmd-demo](demo/sgx-aesmd-demo) subdirectories. + +Brief overviews of the Intel SGX sub-components are given below. + + +#### device plugin + +The [SGX device plugin](cmd/sgx_plugin/README.md) is responsible for discovering +and reporting Intel SGX device nodes to `kubelet`. + +Containers requesting Intel SGX resources in the cluster should not use the +device plugins resources directly. + +#### Intel SGX Admission Webhook + +The Intel SGX admission webhook is responsible for performing Pod mutations based on +the `sgx.intel.com/quote-provider` pod annotation set by the user. The purpose +of the webhook is to hide the details of setting the necessary device resources +and volume mounts for using Intel SGX remote attestation in the cluster. Furthermore, +the Intel SGX admission webhook is responsible for writing a pod/sandbox +`sgx.intel.com/epc` annotation that is used by Kata Containers to dynamically +adjust its virtualized Intel SGX encrypted page cache (EPC) bank(s) size. + +The Intel SGX admission webhook is available as part of +[Intel Device Plugin Operator](cmd/operator/README.md) or +as a standalone [SGX Admission webhook image](cmd/sgx_admissionwebhook/README.md). + +#### Intel SGX EPC memory registration + +The Intel SGX EPC memory available on each node is registered as a Kubernetes extended resource using +node-feature-discovery (NFD). An NFD Node Feature Rule is installed as part of +[SGX device plugin](cmd/sgx_plugin/README.md) +operator deployment and NFD is configured to register the Intel SGX EPC memory +extended resource. + +Containers requesting Intel SGX EPC resources in the cluster use +`sgx.intel.com/epc` resource which is of +type [memory](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#meaning-of-memory). + +### DSA Device Plugin + +The [DSA device plugin](cmd/dsa_plugin/README.md) supports acceleration using +the Intel Data Streaming accelerator(DSA). + +### DLB Device Plugin + +The [DLB device plugin](cmd/dlb_plugin/README.md) supports Intel Dynamic Load +Balancer accelerator(DLB). + +### IAA Device Plugin + +The [IAA device plugin](cmd/iaa_plugin/README.md) supports acceleration using +the Intel Analytics accelerator(IAA). + +## Device Plugins Operator + +To simplify the deployment of the device plugins, a unified device plugins +operator is implemented. + +Currently the operator has support for the DSA, DLB, FPGA, GPU, IAA, QAT, and +Intel SGX device plugins. Each device plugin has its own custom resource +definition (CRD) and the corresponding controller that watches CRUD operations +to those custom resources. + +The [Device plugins operator README](cmd/operator/README.md) gives the installation and usage details for the community operator available on [operatorhub.io](https://operatorhub.io/operator/intel-device-plugins-operator). + +The [Device plugins Operator for OCP](cmd/operator/ocp_quickstart_guide/README.md) gives the installation and usage details for the operator available on [Red Hat OpenShift Container Platform](https://catalog.redhat.com/software/operators/detail/61e9f2d7b9cdd99018fc5736). + +## XeLink XPU-Manager Sidecar + +To support interconnected GPUs in Kubernetes, XeLink sidecar is needed. + +The [XeLink XPU-Manager sidecar README](cmd/xpumanager_sidecar/README.md) gives information how the sidecar functions and how to use it. + +## Demos + +The [demo subdirectory](demo/readme.md) contains a number of demonstrations for +a variety of the available plugins. + +## Workload Authors + +For workloads to get accesss to devices managed by the plugins, the +`Pod` spec must specify the hardware resources needed: + +``` +spec: + containers: + - name: demo-container + image: /: + resources: + limits: + /: X +``` + +The summary of resources available via plugins in this repository is given in the list below. + +**Device Namespace : Registered Resource(s)** + * `dlb.intel.com` : `pf` or `vf` + * [dlb-libdlb-demo-pod.yaml](demo/dlb-libdlb-demo-pod.yaml) + * `dsa.intel.com` : `wq-user-[shared or dedicated]` + * [dsa-accel-config-demo-pod.yaml](demo/dsa-accel-config-demo-pod.yaml) + * `fpga.intel.com` : custom, see [mappings](cmd/fpga_admissionwebhook/README.md#mappings) + * [intelfpga-job.yaml](demo/intelfpga-job.yaml) + * `gpu.intel.com` : `i915` + * [intelgpu-job.yaml](demo/intelgpu-job.yaml) + * `iaa.intel.com` : `wq-user-[shared or dedicated]` + * [iaa-qpl-demo-pod.yaml](demo/iaa-qpl-demo-pod.yaml) + * `qat.intel.com` : `generic` or `cy`/`dc` + * [crypto-perf-dpdk-pod-requesting-qat.yaml](deployments/qat_dpdk_app/base/crypto-perf-dpdk-pod-requesting-qat.yaml) + * `sgx.intel.com` : `epc` + * [intelsgx-job.yaml](deployments/sgx_enclave_apps/base/intelsgx-job.yaml) + * `vpu.intel.com` : `hddl` + * [intelvpu-job.yaml](demo/intelvpu-job.yaml) + +## Developers + +For information on how to develop a new plugin using the framework or work on development task in +this repository, see the [Developers Guide](DEVEL.md). + +## Supported Kubernetes Versions + +Releases are made under the github [releases area](https://github.com/intel/intel-device-plugins-for-kubernetes/releases). Supported releases and +matching Kubernetes versions are listed below: + +| Branch | Kubernetes branch/version | Status | +|:------------------|:-------------------------------|:------------| +| release-0.27 | Kubernetes 1.27 branch v1.27.x | supported | +| release-0.26 | Kubernetes 1.26 branch v1.26.x | supported | +| release-0.25 | Kubernetes 1.25 branch v1.25.x | supported | +| release-0.24 | Kubernetes 1.24 branch v1.24.x | unsupported | +| release-0.23 | Kubernetes 1.23 branch v1.23.x | unsupported | +| release-0.22 | Kubernetes 1.22 branch v1.22.x | unsupported | +| release-0.21 | Kubernetes 1.21 branch v1.21.x | unsupported | +| release-0.20 | Kubernetes 1.20 branch v1.20.x | unsupported | +| release-0.19 | Kubernetes 1.19 branch v1.19.x | unsupported | +| release-0.18 | Kubernetes 1.18 branch v1.18.x | unsupported | +| release-0.17 | Kubernetes 1.17 branch v1.17.x | unsupported | +| release-0.15 | Kubernetes 1.15 branch v1.15.x | unsupported | +| release-0.11 | Kubernetes 1.11 branch v1.11.x | unsupported | + +[Go environment]: https://golang.org/doc/install +[Kubernetes cluster]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ + +## Pre-built plugin images + +Pre-built images of the plugins are available on the Docker hub. These images +are automatically built and uploaded to the hub from the latest main branch of +this repository. + +Release tagged images of the components are also available on the Docker hub, +tagged with their release version numbers in the format x.y.z, corresponding to +the branches and releases in this repository. + +**Note:** the default deployment files and operators are configured with +[imagePullPolicy](https://kubernetes.io/docs/concepts/containers/images/#updating-images) +```IfNotPresent``` and can be changed with ```scripts/set-image-pull-policy.sh```. + +## License + +All of the source code required to build intel-device-plugins-for-kubernetes +is available under Open Source licenses. The source code files identify external Go +modules used. Binaries are distributed as container images on +DockerHub*. Those images contain license texts and source code under `/licenses`. + +### Helm Charts + +Device Plugins Helm Charts are located in Intel Helm Charts repository [Intel Helm Charts](https://github.com/intel/helm-charts). This is another way of distributing Kubernetes resources of the device plugins framework. + +To add repo: +``` +helm repo add intel https://intel.github.io/helm-charts +``` diff --git a/0.27/_sources/SECURITY.md.txt b/0.27/_sources/SECURITY.md.txt new file mode 100644 index 000000000..e342bcacb --- /dev/null +++ b/0.27/_sources/SECURITY.md.txt @@ -0,0 +1,13 @@ +**Reporting a Potential Security Vulnerability**: If you have discovered +potential security vulnerability in this project, please send an e-mail to +secure@intel.com. Encrypt sensitive information using our +[PGP public key](https://www.intel.com/content/www/us/en/security-center/pgp-public-key.html). + +Please provide as much information as possible, including: + - The projects and versions affected + - Detailed description of the vulnerability + - Information on known exploits + +A member of the Intel Product Security Team will review your e-mail and +contact you to collaborate on resolving the issue. For more information on +how Intel works to resolve security issues, see [Vulnerability Handling Guidelines](https://www.intel.com/content/www/us/en/security-center/vulnerability-handling-guidelines.html). diff --git a/0.27/_sources/cmd/dlb_plugin/README.md.txt b/0.27/_sources/cmd/dlb_plugin/README.md.txt new file mode 100644 index 000000000..d9abe2ef0 --- /dev/null +++ b/0.27/_sources/cmd/dlb_plugin/README.md.txt @@ -0,0 +1,228 @@ +# Intel DLB device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Installation](#installation) + * [Pre-built Images](#pre-built-images) + * [Verify Plugin Registration](#verify-plugin-registration) +* [Testing and Demos](#testing-and-demos) + +## Introduction + +This Intel DLB device plugin provides support for [Intel DLB](https://builders.intel.com/docs/networkbuilders/SKU-343247-001US-queue-management-and-load-balancing-on-intel-architecture.pdf) devices under Kubernetes. + +### DLB2 driver configuration for PFs +The DLB device plugin requires a Linux Kernel DLB driver to be installed and enabled to operate. Get [DLB software release](https://www.intel.com/content/www/us/en/download/686372/intel-dynamic-load-balancer.html), build and load the dlb2 driver module following the instruction of 'DLB_Driver_User_Guide.pdf' in the directory 'dlb/docs'. + +After successfully loading the module, available dlb device nodes are visible in devfs. +```bash +$ ls -1 /dev/dlb* +/dev/dlb0 /dev/dlb1 /dev/dlb2 ... +``` + +### VF configuration using a DPDK tool (but with dlb2 driver) +If you configure SR-IOV/VF (virtual functions), continue the following configurations. This instruction uses DPDK tool to check eventdev devices, unbind a VF device, and bind dlb2 driver to a VF device. + +Patch dpdk sources to work with DLB: +```bash +$ wget -q https://fast.dpdk.org/rel/dpdk-21.11.tar.xz -O- | tar -Jx +$ wget -q https://downloadmirror.intel.com/763709/dlb_linux_src_release8.0.0.txz -O- | tar -Jx +$ cd ./dpdk-*/ && patch -p1 < ../dlb/dpdk/dpdk_dlb_*_diff.patch +$ sed -i 's/270b,2710,2714/270b,2710,2711,2714/g' ./usertools/dpdk-devbind.py +``` + +List eventdev devices: +```bash +$ ./usertools/dpdk-devbind.py -s | grep -A10 ^Eventdev +Eventdev devices using kernel driver +==================================== +0000:6d:00.0 'Device 2710' drv=dlb2 unused= +0000:72:00.0 'Device 2710' drv=dlb2 unused= +... +``` + +Enable virtual functions: +```bash +$ echo 4 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/sriov_numvfs +``` +> **Note:**: If it fails saying "No such file or directory," it may be bound to vfio-pci driver. Bind the device to dlb2 driver. + +Check if new dlb device nodes appear: +```bash +$ ls -1 /dev/dlb* +/dev/dlb0 /dev/dlb1 /dev/dlb10 /dev/dlb11 ... /dev/dlb8 /dev/dlb9 +``` + +Check that new eventdev devices appear: +```bash +$ ./usertools/dpdk-devbind.py -s | grep -A14 ^Eventdev +Eventdev devices using kernel driver +==================================== +0000:6d:00.0 'Device 2710' drv=dlb2 unused= +0000:6d:00.1 'Device 2711' drv=dlb2 unused= +0000:6d:00.2 'Device 2711' drv=dlb2 unused= +0000:6d:00.3 'Device 2711' drv=dlb2 unused= +0000:6d:00.4 'Device 2711' drv=dlb2 unused= +0000:72:00.0 'Device 2710' drv=dlb2 unused= +... +``` + +Assign PF resources to VF: +> **Note:**: The process below is only for the first vf resource among 4 resources. Repeat for other vfN_resources in /sys/bus/pci/devices/0000\:6d\:00.0/, and then bind dlb2 driver to 0000:6d:00.M that corresponds to vfN_resources. + +- Unbind driver from the VF device before configuring it. +```bash +$ sudo ./usertools/dpdk-devbind.py --unbind 0000:6d:00.1 +``` + +- Assign PF resources to VF: +```bash +$ echo 2048 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_atomic_inflights && + echo 2048 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_dir_credits && + echo 64 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_dir_ports && + echo 2048 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_hist_list_entries && + echo 8192 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_ldb_credits && + echo 64 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_ldb_ports && + echo 32 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_ldb_queues && + echo 32 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_sched_domains && + echo 2 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_sn0_slots && + echo 2 | sudo tee -a /sys/bus/pci/devices/0000\:6d\:00.0/vf0_resources/num_sn1_slots +``` + +- Bind driver back to the VF device: +```bash +$ sudo ./usertools/dpdk-devbind.py --bind dlb2 0000:6d:00.1 +``` + + +### Verification of well-configured devices: +Run libdlb example app: +> **Note:**: Alternative way is to use this [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/dlb-libdlb-demo/Dockerfile) for running tests. + +```bash +$ ls +dlb dpdk-21.11 +$ cd ./dlb/libdlb/ && make && sudo LD_LIBRARY_PATH=$PWD ./examples/dir_traffic -n 128 -d 1 +# For running test for /dev/dlbN, replace 1 with N. +``` + +Run dpdk example app: +> **Note:**: Alternative way is to use this [Dockerfile](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/dlb-dpdk-demo/Dockerfile) for patching and building DPDK and running tests. + +- Install build dependencies and build dpdk: +```bash +$ sudo apt-get update && sudo apt-get install build-essential meson python3-pyelftools libnuma-dev python3-pip && sudo pip install ninja +# This configuration is based on Ubuntu/Debian distribution. For other distributions that do not use apt, install the dependencies using another way. +$ ls +dlb dpdk-21.11 +$ cd ./dpdk-* && meson setup --prefix $(pwd)/installdir builddir && ninja -C builddir install +``` + +- Run eventdev test +```bash +sudo ./builddir/app/dpdk-test-eventdev --no-huge --vdev='dlb2_event,dev_id=1' -- --test=order_queue --nb_flows 64 --nb_pkts 512 --plcores 1 --wlcores 2-7 +# For running test for /dev/dlbN, replace 1 with N. +``` + +## Installation + +The following sections detail how to obtain, build, deploy and test the DLB device plugin. + +Examples are provided showing how to deploy the plugin either using a DaemonSet or by hand on a per-node basis. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-dlb-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/dlb_plugin?ref=' +daemonset.apps/intel-dlb-plugin created +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +Nothing else is needed. See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin. + +### Verify Plugin Registration + +You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes: + +```bash +$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{" "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\( dlb\)' +master + dlb.intel.com/pf: 7 + dlb.intel.com/vf: 4 +``` + +## Testing and Demos + +We can test the plugin is working by deploying the provided example test images (dlb-libdlb-demo and dlb-dpdk-demo). + +1. Build a Docker image and create a pod running unit tests off the local Docker image: + + ```bash + $ make dlb-libdlb-demo + ... + Successfully tagged intel/dlb-libdlb-demo:devel + + $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/dlb-libdlb-demo-pod.yaml + pod/dlb-libdlb-demo-pod created + ``` + + ```bash + $ make dlb-dpdk-demo + ... + Successfully tagged intel/dlb-dpdk-demo:devel + + $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/dlb-dpdk-demo-pod.yaml + pod/dlb-dpdk-demo-pod created + ``` + +1. Wait until pod is completed: + + ```bash + $ kubectl get pods | grep dlb-.*-demo + NAME READY STATUS RESTARTS AGE + dlb-dpdk-demo 0/2 Completed 0 79m + dlb-libdlb-demo 0/2 Completed 0 18h + ``` + +1. Review the job's logs: + + ```bash + $ kubectl logs dlb-libdlb-demo + + ``` + + ```bash + $ kubectl logs dlb-dpdk-demo + + ``` + + If the pod did not successfully launch, possibly because it could not obtain the DLB + resource, it will be stuck in the `Pending` status: + + ```bash + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + dlb-dpdk-demo 0/2 Pending 0 3s + dlb-libdlb-demo 0/2 Pending 0 10s + ``` + + This can be verified by checking the Events of the pod: + + ```bash + $ kubectl describe pod dlb-libdlb-demo | grep -A3 Events: + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 85s default-scheduler 0/1 nodes are available: 1 Insufficient dlb.intel.com/pf, 1 Insufficient dlb.intel.com/vf. + ``` diff --git a/0.27/_sources/cmd/dsa_plugin/README.md.txt b/0.27/_sources/cmd/dsa_plugin/README.md.txt new file mode 100644 index 000000000..416ddf9cc --- /dev/null +++ b/0.27/_sources/cmd/dsa_plugin/README.md.txt @@ -0,0 +1,135 @@ +# Intel DSA device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Installation](#installation) + * [Pre-built Images](#pre-built-images) + * [Verify Plugin Registration](#verify-plugin-registration) +* [Testing and Demos](#testing-and-demos) + +## Introduction + +The DSA device plugin for Kubernetes supports acceleration using the Intel Data Streaming accelerator(DSA). + +The DSA plugin discovers DSA work queues and presents them as a node resources. + +The DSA plugin and operator optionally support provisioning of DSA devices and workqueues with the help of [accel-config](https://github.com/intel/idxd-config) utility through initcontainer. + +## Installation + +The following sections detail how to use the DSA device plugin. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-dsa-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/dsa_plugin?ref=' +daemonset.apps/intel-dsa-plugin created +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +Nothing else is needed. See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin. + +#### Automatic Provisioning + +There's a sample [idxd initcontainer](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/build/docker/intel-idxd-config-initcontainer.Dockerfile) included that provisions DSA devices and workqueues (1 engine / 1 group / 1 wq (user/dedicated)), to deploy: + +```bash +$ kubectl apply -k deployments/dsa_plugin/overlays/dsa_initcontainer/ +``` + +The provisioning [script](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/idxd-init.sh) and [template](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/master/demo/dsa.conf) are available for customization. + +The provisioning config can be optionally stored in the ProvisioningConfig configMap which is then passed to initcontainer through the volume mount. + +There's also a possibility for a node specific congfiguration through passing a nodename via NODE_NAME into initcontainer's environment and passing a node specific profile via configMap volume mount. + +To create a custom provisioning config: + +```bash +$ kubectl create configmap --namespace=inteldeviceplugins-system intel-dsa-config --from-file=demo/dsa.conf +``` + +### Verify Plugin Registration +You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes: + +```bash +$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{" "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\( dsa\)' +master + dsa.intel.com/wq-user-dedicated: 2 + dsa.intel.com/wq-user-shared: 8 +node1 + dsa.intel.com/wq-user-dedicated: 4 + dsa.intel.com/wq-user-shared: 20 +``` + +## Testing and Demos + +We can test the plugin is working by deploying the provided example accel-config test image. + +1. Build a Docker image with an accel-config tests: + + ```bash + $ make accel-config-demo + ... + Successfully tagged accel-config-demo:devel + ``` + +1. Create a pod running unit tests off the local Docker image: + + ```bash + $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/dsa-accel-config-demo-pod.yaml + pod/dsa-accel-config-demo created + ``` + +1. Wait until pod is completed: + + ```bash + $ kubectl get pods |grep dsa-accel-config-demo + dsa-accel-config-demo 0/1 Completed 0 31m + +1. Review the job's logs: + + ```bash + $ kubectl logs dsa-accel-config-demo | tail + [debug] PF in sub-task[6], consider as passed + [debug] PF in sub-task[7], consider as passed + [debug] PF in sub-task[8], consider as passed + [debug] PF in sub-task[9], consider as passed + [debug] PF in sub-task[10], consider as passed + [debug] PF in sub-task[11], consider as passed + [debug] PF in sub-task[12], consider as passed + [debug] PF in sub-task[13], consider as passed + [debug] PF in sub-task[14], consider as passed + [debug] PF in sub-task[15], consider as passed + ``` + + If the pod did not successfully launch, possibly because it could not obtain the DSA + resource, it will be stuck in the `Pending` status: + + ```bash + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + dsa-accel-config-demo 0/1 Pending 0 7s + ``` + + This can be verified by checking the Events of the pod: + + ```bash + + $ kubectl describe pod dsa-accel-config-demo | grep -A3 Events: + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 2m26s default-scheduler 0/1 nodes are available: 1 Insufficient dsa.intel.com/wq-user-dedicated, 1 Insufficient dsa.intel.com/wq-user-shared. + ``` diff --git a/0.27/_sources/cmd/fpga_admissionwebhook/README.md.txt b/0.27/_sources/cmd/fpga_admissionwebhook/README.md.txt new file mode 100644 index 000000000..89daa3b0d --- /dev/null +++ b/0.27/_sources/cmd/fpga_admissionwebhook/README.md.txt @@ -0,0 +1,186 @@ +# Intel FPGA admission controller for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Dependencies](#dependencies) +* [Installation](#installation) + * [Pre-requisites](#pre-requisites) + * [Mappings](#mappings) + * [Deployment](#deployment) + * [Webhook deployment](#webhook-deployment) + * [Mappings deployment](#mappings-deployment) +* [Next steps](#next-steps) + +## Introduction + +The FPGA admission controller is one of the components used to add support for Intel FPGA +devices to Kubernetes. + +> **NOTE:** Installation of the FPGA admission controller can be skipped if the +> [FPGA device plugin](../fpga_plugin/README.md) is operated with the Intel Device Plugins Operator +> since it integrates the controller's functionality. + +The FPGA admission controller webhook is responsible for performing mapping from user-friendly +function IDs to the Interface ID and Bitstream ID that are required for FPGA programming by +the [FPGA CRI-O hook](../fpga_crihook/README.md). + +Mappings are stored in namespaced custom resource definition (CRD) objects, therefore the admission +controller also performs access control, determining which bitstream can be used for which namespace. +More details can be found in the [Mappings](#mappings) section. + +The admission controller also keeps the user from bypassing namespaced mapping restrictions, +by denying admission of any pods that are trying to use internal knowledge of InterfaceID or +Bitstream ID environment variables used by the prestart hook. + +## Dependencies + +This component is one of a set of components that work together. You may also want to +install the following: + +- [FPGA device plugin](../fpga_plugin/README.md) +- [FPGA prestart CRI-O hook](../fpga_crihook/README.md) + +All components have the same basic dependencies as the +[generic plugin framework dependencies](../../README.md#about) + +## Installation + +The following sections detail how to obtain, build and deploy the admission +controller webhook plugin. + +### Pre-requisites + +The default webhook deployment depends on having [cert-manager](https://cert-manager.io/) +installed. See its installation instructions [here](https://cert-manager.io/docs/installation/kubectl/). + +Also if your cluster operates behind a corporate proxy make sure that the API +server is configured not to send requests to cluster services through the +proxy. You can check that with the following command: + +```bash +$ kubectl describe pod kube-apiserver --namespace kube-system | grep -i no_proxy | grep "\.svc" +``` + +In case there's no output and your cluster was deployed with `kubeadm` open +`/etc/kubernetes/manifests/kube-apiserver.yaml` at the control plane nodes and +append `.svc` and `.svc.cluster.local` to the `no_proxy` environment variable: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + ... +spec: + containers: + - command: + - kube-apiserver + - --advertise-address=10.237.71.99 + ... + env: + - name: http_proxy + value: http://proxy.host:8080 + - name: https_proxy + value: http://proxy.host:8433 + - name: no_proxy + value: 127.0.0.1,localhost,.example.com,10.0.0.0/8,.svc,.svc.cluster.local + ... +``` + +**Note:** To build clusters using `kubeadm` with the right `no_proxy` settings from the very beginning, +set the cluster service names to `$no_proxy` before `kubeadm init`: + +``` +$ export no_proxy=$no_proxy,.svc,.svc.cluster.local +``` + +## Mappings + +Mappings is a an essential part of the setup that gives a flexible instrument to a cluster +administrator to manage FPGA bitstreams and to control access to them. Being a set of +custom resource definitions they are used to configure the way FPGA resource requests get +translated into actual resources provided by the cluster. + +For the following mapping + +```yaml +apiVersion: fpga.intel.com/v2 +kind: AcceleratorFunction +metadata: + name: arria10.dcp1.2-nlb0-preprogrammed +spec: + afuId: d8424dc4a4a3c413f89e433683f9040b + interfaceId: 69528db6eb31577a8c3668f9faa081f6 + mode: af +``` + +requested FPGA resources are translated to AF resources. For example, +`fpga.intel.com/arria10.dcp1.2-nlb0-preprogrammed` is translated to +`fpga.intel.com/af-695.d84.aVKNtusxV3qMNmj5-qCB9thCTcSko8QT-J5DNoP5BAs` where the `af-` +prefix indicates the plugin's mode (`af`), `695` is the first three characters of +the region interface ID, `d84` is the first three characters of the accelerator function ID +and the last part `aVKNtusxV3qMNmj5-qCB9thCTcSko8QT-J5DNoP5BAs` is a base64-encoded concatenation +of the full region interface ID and accelerator function ID. +The format of resource names (e.g. `arria10.dcp1.2-nlb0-preprogrammed`) can be any and is up +to a cluster administrator. + +The same mapping, but with its mode field set to `region`, would translate +`fpga.intel.com/arria10.dcp1.2-nlb0-preprogrammed` to `fpga.intel.com/region-69528db6eb31577a8c3668f9faa081f6`, +and the corresponding AF IDs are set in environment variables for the container. +Though in this case the cluster administrator would probably want to rename +the mapping `arria10.dcp1.2-nlb0-preprogrammed` to something like `arria10.dcp1.2-nlb0-orchestrated` +to reflect its mode. The [FPGA CRI-O hook](../fpga_crihook/README.md) then loads the requested +bitstream to a region before the container is started. + +Mappings of resource names are configured with objects of `AcceleratorFunction` and +`FpgaRegion` custom resource definitions found respectively in +[`./deployment/fpga_admissionwebhook/crd/bases/fpga.intel.com_af.yaml`](/deployments/fpga_admissionwebhook/crd/bases/fpga.intel.com_acceleratorfunctions.yaml) +and [`./deployment/fpga_admissionwebhook/crd/bases/fpga.intel.com_region.yaml`](/deployments/fpga_admissionwebhook/crd/bases/fpga.intel.com_fpgaregions.yaml). + +Example mappings between 'names' and 'ID's are controlled by the admission controller mappings collection file found in +[`./deployments/fpga_admissionwebhook/mappings-collection.yaml`](/deployments/fpga_admissionwebhook/mappings-collection.yaml). + + +### Deployment + +#### Webhook deployment + +To deploy the webhook, run + +```bash +$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/fpga_admissionwebhook/default?ref=main +namespace/intelfpgawebhook-system created +customresourcedefinition.apiextensions.k8s.io/acceleratorfunctions.fpga.intel.com created +customresourcedefinition.apiextensions.k8s.io/fpgaregions.fpga.intel.com created +mutatingwebhookconfiguration.admissionregistration.k8s.io/intelfpgawebhook-mutating-webhook-configuration created +clusterrole.rbac.authorization.k8s.io/intelfpgawebhook-manager-role created +clusterrolebinding.rbac.authorization.k8s.io/intelfpgawebhook-manager-rolebinding created +service/intelfpgawebhook-webhook-service created +deployment.apps/intelfpgawebhook-webhook created +certificate.cert-manager.io/intelfpgawebhook-serving-cert created +issuer.cert-manager.io/intelfpgawebhook-selfsigned-issuer created +``` + +#### Mappings deployment + +Mappings deployment is a mandatory part of the webhook deployment. You should +prepare and deploy mappings that describe FPGA bitstreams available in your cluster. + +Example mappings collection [`./deployments/fpga_admissionwebhook/mappings-collection.yaml`](/deployments/fpga_admissionwebhook/mappings-collection.yaml) +can be used as an example for cluster mappings. This collection is not intended to be deployed as is, +it should be used as a reference and example of your own cluster mappings. + +To deploy the mappings, run + +```bash +$ kubectl apply -f + +``` + +Note that the mappings are scoped to the namespaces they were created in +and they are applicable to pods created in the corresponding namespaces. + + +## Next steps + +Continue with [FPGA prestart CRI-O hook](../fpga_crihook/README.md). diff --git a/0.27/_sources/cmd/fpga_crihook/README.md.txt b/0.27/_sources/cmd/fpga_crihook/README.md.txt new file mode 100644 index 000000000..59c26fd8a --- /dev/null +++ b/0.27/_sources/cmd/fpga_crihook/README.md.txt @@ -0,0 +1,49 @@ +# Intel FPGA prestart CRI-O webhook for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Dependencies](#dependencies) +* [Configuring CRI-O](#configuring-cri-o) + +## Introduction + +The FPGA CRI-O webhook is one of the components used to add support for Intel FPGA +devices to Kubernetes. + +The FPGA prestart CRI-O hook is triggered by container annotations, such as set by the +[FPGA device plugin](../fpga_plugin/README.md). It performs discovery of the requested FPGA +function bitstream and then programs FPGA devices based on the environment variables +in the workload description. + +The CRI-O prestart hook is only *required* when the +[FPGA admission webhook](../fpga_admissionwebhook/README.md) is configured for orchestration +programmed mode, and is benign (un-used) otherwise. + +> **Note:** The fpga CRI-O webhook is usually installed by the same DaemonSet as the +> FPGA device plugin. If building and installing the CRI-O webhook by hand, it is +> recommended you reference the +> [fpga plugin DaemonSet YAML](/deployments/fpga_plugin/base/intel-fpga-plugin-daemonset.yaml ) for +> more details. + +## Dependencies + +This component is one of a set of components that work together. You may also want to +install the following: + +- [FPGA device plugin](../fpga_plugin/README.md) +- [FPGA admission controller](../fpga_admissionwebhook/README.md) + +All components have the same basic dependencies as the +[generic plugin framework dependencies](../../README.md#about) + +See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the CRI hook. + +## Configuring CRI-O + +Recent versions of [CRI-O](https://github.com/cri-o/cri-o) are shipped with default configuration +file that prevents CRI-O to discover and configure hooks automatically. +For FPGA orchestration programmed mode, the OCI hooks are the key component. +Please ensure that your `/etc/crio/crio.conf` parameter `hooks_dir` is either unset +(to enable default search paths for OCI hooks configuration) or contains the directory +`/etc/containers/oci/hooks.d`. diff --git a/0.27/_sources/cmd/fpga_plugin/README.md.txt b/0.27/_sources/cmd/fpga_plugin/README.md.txt new file mode 100644 index 000000000..b726e461c --- /dev/null +++ b/0.27/_sources/cmd/fpga_plugin/README.md.txt @@ -0,0 +1,228 @@ +# Intel FPGA device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) + * [Component Overview](#component-overview) +* [Modes and Configuration Options](#modes-and-configuration-options) +* [Installation](#installation) + * [Prerequisites](#prerequisites) + * [Pre-built Images](#pre-built-images) + * [Verify Plugin Registration](#verify-plugin-registration) + +## Introduction + +This FPGA device plugin is part of a collection of Kubernetes components found within this +repository that enable integration of Intel FPGA hardware into Kubernetes. + +The following hardware platforms are supported: + +- Intel Arria 10 +- Intel Stratix 10 + +The components support the [Open Programmable Acceleration Engine (OPAE)](https://opae.github.io/latest/index.html) +interface. + +The components together implement the following features: + +- discovery of pre-programmed accelerator functions +- discovery of programmable regions +- orchestration of FPGA programming +- access control for FPGA hardware + +### Component Overview + +The following components are part of this repository, and work together to support Intel FPGAs under +Kubernetes: + +- [FPGA device plugin](README.md) (this component) + + A Kubernetes [device plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) + that discovers available FPGA resources on a node and advertises them to the Kubernetes control plane + via the node kubelet. + +- [FPGA admission controller webhook](../fpga_admissionwebhook/README.md) + + A Kubernetes [admission controller webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) + which can be used to dynamically convert logical resource names in pod specifications into actual FPGA + resource names, as advertised by the device plugin. + + The webhook can also set environment variables to instruct the CRI-O prestart hook to program the FPGA + before launching the container. + + > **NOTE:** Installation of the [FPGA admission controller webhook](../fpga_admissionwebhook/README.md) can be skipped if the + > FPGA device plugin is operated with the Intel Device Plugins Operator + > since it integrates the controller's functionality. + > However, [the mappings](../fpga_admissionwebhook/README.md#mappings-deployment) still must be deployed." + +- [FPGA CRI-O prestart hook](../fpga_crihook/README.md) + + A [CRI-O](https://github.com/cri-o/cri-o) prestart hook that, upon instruction from the FPGA admission + controller, allocates and programs the FPGA before the container is launched. + +The repository also contains an [FPGA helper tool](../fpga_tool/README.md) that may be useful during +development, initial deployment and debugging. + +### Modes and Configuration Options + +The FPGA plugin set can run in one of two modes: + +- `region` mode, where the plugins locate and advertise + regions of the FPGA, and facilitate programing of those regions with the + requested bistreams. +- `af` mode, where the FPGA bitstreams are already loaded + onto the FPGA, and the plugins discover and advertises the existing + Accelerator Functions (AF). + +The example YAML deployments described in this document only currently support +`af` mode. To utilise `region` mode, either modify the existing YAML appropriately, +or deploy 'by hand'. + +Overview diagrams of `af` and `region` modes are below: + +region mode: + +![Overview of `region` mode](pictures/FPGA-region.png) + +af mode: + +![Overview of `af` mode](pictures/FPGA-af.png) + +## Installation + +The below sections cover how to use this component. + +### Prerequisites + +All components have the same basic dependencies as the +[generic plugin framework dependencies](../../README.md#about) + +To obtain a fully operational FPGA enabled cluster, you must install all three +major components: + +- [FPGA device plugin](README.md) (this component) +- [FPGA admission controller webhook](../fpga_admissionwebhook/README.md) +- [FPGA prestart CRI-O hook](../fpga_crihook/README.md) + +The CRI-O hook is only *required* if `region` mode is being used, but is installed by default by the +[FPGA plugin DaemonSet YAML](/deployments/fpga_plugin/base/intel-fpga-plugin-daemonset.yaml), and is benign +in `af` mode. + +If using the `af` mode, and therefore *not* using the +CRI-O prestart hook, runtimes other than CRI-O can be used (that is, the CRI-O hook presently +*only* works with the CRI-O runtime). + +The FPGA device plugin requires a Linux Kernel FPGA driver to be installed and enabled to +operate. The plugin supports the use of either of following two drivers, and auto detects +which is present and thus to use: + +- The Linux Kernel in-tree [DFL](https://www.kernel.org/doc/html/latest/fpga/dfl.html) driver +- The out of tree [OPAE](https://opae.github.io/latest/docs/drv_arch/drv_arch.html) driver + +Install this component (FPGA device plugin) first, and then follow the links +and instructions to install the other components. + +The FPGA webhook deployment depends on having [cert-manager](https://cert-manager.io/) +installed. See its installation instructions [here](https://cert-manager.io/docs/installation/kubectl/). + +```bash +$ kubectl get pods -n cert-manager +NAME READY STATUS RESTARTS AGE +cert-manager-7747db9d88-bd2nl 1/1 Running 0 1m +cert-manager-cainjector-87c85c6ff-59sb5 1/1 Running 0 1m +cert-manager-webhook-64dc9fff44-29cfc 1/1 Running 0 1m + +``` + +### Pre-built Images + +Pre-built images of the components are available on the [Docker hub](https://hub.docker.com/u/intel). +These images are automatically built and uploaded to the hub from the latest `main` branch of +this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers (of the form `x.y.z`, matching the branch/tag release number in this repo). + +The following images are available on the Docker hub: + +- [The FPGA plugin](https://hub.docker.com/r/intel/intel-fpga-plugin) +- [The FPGA admisson webhook](https://hub.docker.com/r/intel/intel-fpga-admissionwebhook) +- [The FPGA CRI-O prestart hook (in the `initcontainer` image)](https://hub.docker.com/r/intel/intel-fpga-initcontainer) + +Depending on the FPGA mode, run either +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/fpga_plugin/overlays/af?ref=' +namespace/intelfpgaplugin-system created +customresourcedefinition.apiextensions.k8s.io/acceleratorfunctions.fpga.intel.com created +customresourcedefinition.apiextensions.k8s.io/fpgaregions.fpga.intel.com created +mutatingwebhookconfiguration.admissionregistration.k8s.io/intelfpgaplugin-mutating-webhook-configuration created +clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-manager-role created +clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-node-getter created +clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-get-nodes created +clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-manager-rolebinding created +service/intelfpgaplugin-webhook-service created +deployment.apps/intelfpgaplugin-webhook created +daemonset.apps/intelfpgaplugin-fpgadeviceplugin created +certificate.cert-manager.io/intelfpgaplugin-serving-cert created +issuer.cert-manager.io/intelfpgaplugin-selfsigned-issuer created +``` +or +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/fpga_plugin/overlays/region?ref=' +namespace/intelfpgaplugin-system created +customresourcedefinition.apiextensions.k8s.io/acceleratorfunctions.fpga.intel.com created +customresourcedefinition.apiextensions.k8s.io/fpgaregions.fpga.intel.com created +mutatingwebhookconfiguration.admissionregistration.k8s.io/intelfpgaplugin-mutating-webhook-configuration created +clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-manager-role created +clusterrole.rbac.authorization.k8s.io/intelfpgaplugin-node-getter created +clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-get-nodes created +clusterrolebinding.rbac.authorization.k8s.io/intelfpgaplugin-manager-rolebinding created +service/intelfpgaplugin-webhook-service created +deployment.apps/intelfpgaplugin-webhook created +daemonset.apps/intelfpgaplugin-fpgadeviceplugin created +certificate.cert-manager.io/intelfpgaplugin-serving-cert created +issuer.cert-manager.io/intelfpgaplugin-selfsigned-issuer created +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +The command should result in two pods running: +```bash +$ kubectl get pods -n intelfpgaplugin-system +NAME READY STATUS RESTARTS AGE +intelfpgaplugin-fpgadeviceplugin-skcw5 1/1 Running 0 57s +intelfpgaplugin-webhook-7d6bcb8b57-k52b9 1/1 Running 0 57s +``` + +If you need the FPGA plugin on some nodes to operate in a different mode then add this +annotation to the nodes: + +```bash +$ kubectl annotate node 'fpga.intel.com/device-plugin-mode=region' +``` +or +```bash +$ kubectl annotate node 'fpga.intel.com/device-plugin-mode=af' +``` +And restart the pods on the nodes. + +> **Note:** The FPGA plugin [DaemonSet YAML](/deployments/fpga_plugin/base/intel-fpga-plugin-daemonset.yaml) +> also deploys the [FPGA CRI-O hook](../fpga_crihook/README.md) `initcontainer` image, but it will be +> benign (un-used) when running the FPGA plugin in `af` mode. + +#### Verify Plugin Registration + +Verify the FPGA plugin has been deployed on the nodes. The below shows the output +you can expect in `region` mode, but similar output should be expected for `af` +mode: + +```bash +$ kubectl describe nodes | grep fpga.intel.com +fpga.intel.com/region-ce48969398f05f33946d560708be108a: 1 +fpga.intel.com/region-ce48969398f05f33946d560708be108a: 1 +``` + +> **Note:** The FPGA plugin [DaemonSet YAML](/deployments/fpga_plugin/fpga_plugin.yaml) +> also deploys the [FPGA CRI-O hook](../fpga_crihook/README.md) `initcontainer` image as well. You may +> also wish to build that image locally before deploying the FPGA plugin to avoid deploying +> the Docker hub default image. diff --git a/0.27/_sources/cmd/fpga_tool/README.md.txt b/0.27/_sources/cmd/fpga_tool/README.md.txt new file mode 100644 index 000000000..39752c890 --- /dev/null +++ b/0.27/_sources/cmd/fpga_tool/README.md.txt @@ -0,0 +1,29 @@ +# Intel FPGA test tool + +## Introduction + +This directory contains an FPGA test tool that can be used to locate, examine and program Intel +FPGAs. + +### Command line and usage + +The tool has the following command line arguments: + +```bash +info, fpgainfo, install, list, fmeinfo, portinfo, list-fme, list-port, pr, release, assign +``` + +and the following command line options: + +```bash +Usage of ./fpga_tool: + -b string + Path to bitstream file (GBS or AOCX) + -d string + Path to device node (FME or Port) + -dry-run + Don't write/program, just validate and log + -force + Force overwrite operation for installing bitstreams + -q Quiet mode. Only errors will be reported +``` \ No newline at end of file diff --git a/0.27/_sources/cmd/gpu_fakedev/README.md.txt b/0.27/_sources/cmd/gpu_fakedev/README.md.txt new file mode 100644 index 000000000..94938ea41 --- /dev/null +++ b/0.27/_sources/cmd/gpu_fakedev/README.md.txt @@ -0,0 +1,47 @@ +# Fake (GPU) device file generator + +Table of Contents +* [Introduction](#introduction) +* [Configuration](#configuration) +* [Potential improvements](#potential-improvements) +* [Related tools](#related-tools) + +## Introduction + +This is a tool for generating (large number of) fake device files for +k8s device scheduling scalability testing. But it can also be used +just to test (GPU) device plugin functionality without having +corresponding device HW. + +Its "intel-gpu-fakedev" container is intended to be run as first init +container in a device plugin pod, so that device plugin (and its NFD +labeler) see the fake (sysfs + devfs) files generated by the tool, +instead of real host sysfs and devfs content. + +## Configuration + +[Configs](configs/) subdirectory contains example JSON configuration +file(s) for the generator. Currently there's only one example JSON +file, but each new device variant adding feature(s) that have specific +support in device plugin, could have their own fake device config. + +## Potential improvements + +If support for mixed device environment is needed, tool can be updated +to use node / configuration file mapping. Such mappings could be e.g. +in configuration files themselves as node name include / exlude lists, +and tool would use first configuration file matching the node it's +running on. For now, one would need to use different pod / config +specs for different nodes to achieve that... + +Currently JSON config file options and the generated files are tied to +what GPU plugin uses, but if needed, they could be changed to fake +also sysfs + devfs device files used by other plugins. + +## Related tools + +[fakedev-exporter](#https://github.com/intel/fakedev-exporter) project +can be used to schedule suitably configured fake workloads on the fake +devices, and to provide provide fake activity metrics for them to +Prometheus, that look like they were reported by real Prometheus +metric exporters for real workloads running on real devices. diff --git a/0.27/_sources/cmd/gpu_nfdhook/README.md.txt b/0.27/_sources/cmd/gpu_nfdhook/README.md.txt new file mode 100644 index 000000000..2735f8470 --- /dev/null +++ b/0.27/_sources/cmd/gpu_nfdhook/README.md.txt @@ -0,0 +1,92 @@ +# Intel GPU NFD hook + +Table of Contents + +* [Introduction](#introduction) +* [GPU memory](#gpu-memory) +* [Default labels](#default-labels) +* [PCI-groups (optional)](#pci-groups-optional) +* [Capability labels (optional)](#capability-labels-optional) +* [Limitations](#limitations) + +## Introduction + +This is the [Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery) +binary hook implementation for the Intel GPUs. The intel-gpu-initcontainer (which +is built with the other images) can be used as part of the gpu-plugin deployment +to copy hook to the host systems on which gpu-plugin itself is deployed. + +When NFD worker runs this hook, it will add a number of labels to the nodes, +which can be used for example to deploy services to nodes with specific GPU +types. Selected numeric labels can be turned into kubernetes extended resources +by the NFD, allowing for finer grained resource management for GPU-using PODs. + +In the NFD deployment, the hook requires `/host-sys` -folder to have the host `/sys`-folder content mounted. Write access is not necessary. + +## GPU memory + +GPU memory amount is read from sysfs `gt/gt*` files and turned into a label. +There are two supported environment variables named `GPU_MEMORY_OVERRIDE` and +`GPU_MEMORY_RESERVED`. Both are supposed to hold numeric byte amounts. For systems with +older kernel drivers or GPUs which do not support reading the GPU memory +amount, the `GPU_MEMORY_OVERRIDE` environment variable value is turned into a GPU +memory amount label instead of a read value. `GPU_MEMORY_RESERVED` value will be +scoped out from the GPU memory amount found from sysfs. + +## Default labels + +Following labels are created by default. You may turn numeric labels into extended resources with NFD. + +name | type | description| +-----|------|------| +|`gpu.intel.com/millicores`| number | node GPU count * 1000. Can be used as a finer grained shared execution fraction. +|`gpu.intel.com/memory.max`| number | sum of detected [GPU memory amounts](#gpu-memory) in bytes OR environment variable value * GPU count +|`gpu.intel.com/cards`| string | list of card names separated by '`.`'. The names match host `card*`-folders under `/sys/class/drm/`. Deprecated, use `gpu-numbers`. +|`gpu.intel.com/gpu-numbers`| string | list of numbers separated by '`.`'. The numbers correspond to device file numbers for the primary nodes of given GPUs in kernel DRI subsystem, listed as `/dev/dri/card` in devfs, and `/sys/class/drm/card` in sysfs. +|`gpu.intel.com/tiles`| number | sum of all detected GPU tiles in the system. +|`gpu.intel.com/numa-gpu-map`| string | list of numa node to gpu mappings. + +If the value of the `gpu-numbers` label would not fit into the 63 character length limit, you will also get labels `gpu-numbers2`, +`gpu-numbers3`... until all the gpu numbers have been labeled. + +The tile count `gpu.intel.com/tiles` describes the total amount of tiles on the system. System is expected to be homogeneous, and thus the number of tiles per GPU can be calculated by dividing the tile count with GPU count. + +The `numa-gpu-map` label is a list of numa to gpu mapping items separated by `_`. Each list item has a numa node id combined with a list of gpu indices. e.g. 0-1.2.3 would mean: numa node 0 has gpus 1, 2 and 3. More complex example would be: 0-0.1_1-3.4 where numa node 0 would have gpus 0 and 1, and numa node 1 would have gpus 3 and 4. As with `gpu-numbers`, this label will be extended to multiple labels if the length of the value exceeds the max label length. + +## PCI-groups (optional) + +GPUs which share the same pci paths under `/sys/devices/pci*` can be grouped into a label. GPU nums are separated by '`.`' and +groups are separated by '`_`'. The label is created only if environment variable named `GPU_PCI_GROUPING_LEVEL` has a value greater +than zero. GPUs are considered to belong to the same group, if as many identical folder names are found for the GPUs, as is the value +of the environment variable. Counting starts from the folder name which starts with `pci`. + +For example, the SG1 card has 4 GPUs, which end up sharing pci-folder names under `/sys/devices`. With a `GPU_PCI_GROUPING_LEVEL` +of 3, a node with two such SG1 cards could produce a `pci-groups` label with a value of `0.1.2.3_4.5.6.7`. + +name | type | description| +-----|------|------| +|`gpu.intel.com/pci-groups`| string | list of pci-groups separated by '`_`'. GPU numbers in the groups are separated by '`.`'. The numbers correspond to device file numbers for the primary nodes of given GPUs in kernel DRI subsystem, listed as `/dev/dri/card` in devfs, and `/sys/class/drm/card` in sysfs. + +If the value of the `pci-groups` label would not fit into the 63 character length limit, you will also get labels `pci-groups2`, +`pci-groups3`... until all the pci groups have been labeled. + +## Capability labels (optional) + +Capability labels are created from information found inside debugfs, and therefore +unfortunately require running the NFD worker as root. Due to coming from debugfs, +which is not guaranteed to be stable, these are not guaranteed to be stable either. +If you do not need these, simply do not run NFD worker as root, that is also more secure. +Depending on your kernel driver, running the NFD hook as root may introduce following labels: + +name | type | description| +-----|------|------| +|`gpu.intel.com/platform_gen`| string | GPU platform generation name, typically an integer. Deprecated. +|`gpu.intel.com/media_version`| string | GPU platform Media pipeline generation name, typically a number. Deprecated. +|`gpu.intel.com/graphics_version`| string | GPU platform graphics/compute pipeline generation name, typically a number. Deprecated. +|`gpu.intel.com/platform_.count`| number | GPU count for the named platform. +|`gpu.intel.com/platform_.tiles`| number | GPU tile count in the GPUs of the named platform. +|`gpu.intel.com/platform_.present`| string | "true" for indicating the presense of the GPU platform. + +## Limitations + +For the above to work as intended, GPUs on the same node must be identical in their capabilities. diff --git a/0.27/_sources/cmd/gpu_plugin/README.md.txt b/0.27/_sources/cmd/gpu_plugin/README.md.txt new file mode 100644 index 000000000..484c09e2f --- /dev/null +++ b/0.27/_sources/cmd/gpu_plugin/README.md.txt @@ -0,0 +1,402 @@ +# Intel GPU device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Modes and Configuration Options](#modes-and-configuration-options) +* [Operation modes for different workload types](#operation-modes-for-different-workload-types) +* [Installation](#installation) + * [Prerequisites](#prerequisites) + * [Drivers for discrete GPUs](#drivers-for-discrete-gpus) + * [Kernel driver](#kernel-driver) + * [Intel DKMS packages](#intel-dkms-packages) + * [Upstream kernel](#upstream-kernel) + * [GPU Version](#gpu-version) + * [GPU Firmware](#gpu-firmware) + * [User-space drivers](#user-space-drivers) + * [Drivers for older (integrated) GPUs](#drivers-for-older-integrated-gpus) + * [Pre-built Images](#pre-built-images) + * [Install to all nodes](#install-to-all-nodes) + * [Install to nodes with Intel GPUs with NFD](#install-to-nodes-with-intel-gpus-with-nfd) + * [Install to nodes with NFD, Monitoring and Shared-dev](#install-to-nodes-with-nfd-monitoring-and-shared-dev) + * [Install to nodes with Intel GPUs with Fractional resources](#install-to-nodes-with-intel-gpus-with-fractional-resources) + * [Fractional resources details](#fractional-resources-details) + * [Verify Plugin Registration](#verify-plugin-registration) +* [Testing and Demos](#testing-and-demos) +* [Issues with media workloads on multi-GPU setups](#issues-with-media-workloads-on-multi-gpu-setups) + * [Workaround for QSV and VA-API](#workaround-for-qsv-and-va-api) + + +## Introduction + +Intel GPU plugin facilitates Kubernetes workload offloading by providing access to +discrete (including Intel® Data Center GPU Flex Series) and integrated Intel GPU devices +supported by the host kernel. + +Use cases include, but are not limited to: +- Media transcode +- Media analytics +- Cloud gaming +- High performance computing +- AI training and inference + +For example containers with Intel media driver (and components using that), can offload +video transcoding operations, and containers with the Intel OpenCL / oneAPI Level Zero +backend libraries can offload compute operations to GPU. + +## Modes and Configuration Options + +| Flag | Argument | Default | Meaning | +|:---- |:-------- |:------- |:------- | +| -enable-monitoring | - | disabled | Enable 'i915_monitoring' resource that provides access to all Intel GPU devices on the node | +| -resource-manager | - | disabled | Enable fractional resource management, [see also dependencies](#fractional-resources) | +| -shared-dev-num | int | 1 | Number of containers that can share the same GPU device | +| -allocation-policy | string | none | 3 possible values: balanced, packed, none. For shared-dev-num > 1: _balanced_ mode spreads workloads among GPU devices, _packed_ mode fills one GPU fully before moving to next, and _none_ selects first available device from kubelet. Default is _none_. Allocation policy does not have an effect when resource manager is enabled. | + +The plugin also accepts a number of other arguments (common to all plugins) related to logging. +Please use the -h option to see the complete list of logging related options. + +## Operation modes for different workload types + +Intel GPU-plugin supports a few different operation modes. Depending on the workloads the cluster is running, some modes make more sense than others. Below is a table that explains the differences between the modes and suggests workload types for each mode. Mode selection applies to the whole GPU plugin deployment, so it is a cluster wide decision. + +| Mode | Sharing | Intended workloads | Suitable for time critical workloads | +|:---- |:-------- |:------- |:------- | +| shared-dev-num == 1 | No, 1 container per GPU | Workloads using all GPU capacity, e.g. AI training | Yes | +| shared-dev-num > 1 | Yes, >1 containers per GPU | (Batch) workloads using only part of GPU resources, e.g. inference, media transcode/analytics, or CPU bound GPU workloads | No | +| shared-dev-num > 1 && resource-management | Yes and no, 1>= containers per GPU | Any. For best results, all workloads should declare their expected GPU resource usage (memory, millicores). Requires [GAS](https://github.com/intel/platform-aware-scheduling/tree/master/gpu-aware-scheduling). See also [fractional use](#fractional-resources-details) | Yes. 1000 millicores = exclusive GPU usage. See note below. | + +> **Note**: Exclusive GPU usage with >=1000 millicores requires that also *all other GPU containers* specify (non-zero) millicores resource usage. + +## Installation + +The following sections detail how to obtain, build, deploy and test the GPU device plugin. + +Examples are provided showing how to deploy the plugin either using a DaemonSet or by hand on a per-node basis. + +### Prerequisites + +Access to a GPU device requires firmware, kernel and user-space +drivers supporting it. Firmware and kernel driver need to be on the +host, user-space drivers in the GPU workload containers. + +Intel GPU devices supported by the current kernel can be listed with: +``` +$ grep i915 /sys/class/drm/card?/device/uevent +/sys/class/drm/card0/device/uevent:DRIVER=i915 +/sys/class/drm/card1/device/uevent:DRIVER=i915 +``` + +#### Drivers for discrete GPUs + +> **Note**: Kernel (on host) and user-space drivers (in containers) +> should be installed from the same repository as there are some +> differences between DKMS and upstream GPU driver uAPI. + +##### Kernel driver + +###### Intel DKMS packages + +`i915` GPU driver DKMS[^dkms] package is recommended for Intel +discrete GPUs, until their support in upstream is complete. DKMS +package(s) can be installed from Intel package repositories for a +subset of older kernel versions used in enterprise / LTS +distributions: +https://dgpu-docs.intel.com/installation-guides/index.html + +[^dkms]: [intel-gpu-i915-backports](https://github.com/intel-gpu/intel-gpu-i915-backports). + +###### Upstream kernel + +Upstream Linux kernel 6.2 or newer is needed for Intel discrete GPU +support. For now, upstream kernel is still missing support for a few +of the features available in DKMS kernels (e.g. Level-Zero Sysman API +GPU error counters). + +##### GPU Version + +PCI IDs for the Intel GPUs on given host can be listed with: +``` +$ lspci | grep -e VGA -e Display | grep Intel +88:00.0 Display controller: Intel Corporation Device 56c1 (rev 05) +8d:00.0 Display controller: Intel Corporation Device 56c1 (rev 05) +``` + +(`lspci` lists GPUs with display support as "VGA compatible controller", +and server GPUs without display support, as "Display controller".) + +Mesa "Iris" 3D driver header provides a mapping between GPU PCI IDs and their Intel brand names: +https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/include/pci_ids/iris_pci_ids.h + +###### GPU Firmware + +If your kernel build does not find the correct firmware version for +a given GPU from the host (see `dmesg | grep i915` output), latest +firmware versions are available in upstream: +https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/i915 + +##### User-space drivers + +Until new enough user-space drivers (supporting also discrete GPUs) +are available directly from distribution package repositories, they +can be installed to containers from Intel package repositories. See: +https://dgpu-docs.intel.com/installation-guides/index.html + +Example container is listed in [Testing and demos](#testing-and-demos). + +Validation status against *upstream* kernel is listed in the user-space drivers release notes: +* Media driver: https://github.com/intel/media-driver/releases +* Compute driver: https://github.com/intel/compute-runtime/releases + +#### Drivers for older (integrated) GPUs + +For the older (integrated) GPUs, new enough firmware and kernel driver +are typically included already with the host OS, and new enough +user-space drivers (for the GPU containers) are in the host OS +repositories. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-gpu-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +> **Note**: Replace `` with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +> **Note**: Add ```--dry-run=client -o yaml``` to the ```kubectl``` commands below to visualize the yaml content being applied. + +See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin. + +#### Install to all nodes + +Simplest option to enable use of Intel GPUs in Kubernetes Pods. + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin?ref=' +``` + +#### Install to nodes with Intel GPUs with NFD + +Deploying GPU plugin to only nodes that have Intel GPU attached. [Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery) is required to detect the presence of Intel GPUs. + +```bash +# Start NFD - if your cluster doesn't have NFD installed yet +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=' + +# Create NodeFeatureRules for detecting GPUs on nodes +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=' + +# Create GPU plugin daemonset +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/nfd_labeled_nodes?ref=' +``` + +#### Install to nodes with NFD, Monitoring and Shared-dev + +Same as above, but configures GPU plugin with logging, [monitoring and shared-dev](#modes-and-configuration-options) features enabled. This option is useful when there is a desire to retrieve GPU metrics from nodes. For example with [XPU-Manager](https://github.com/intel/xpumanager/) or [collectd](https://github.com/collectd/collectd/tree/collectd-6.0). + +```bash +# Start NFD - if your cluster doesn't have NFD installed yet +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=' + +# Create NodeFeatureRules for detecting GPUs on nodes +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=' + +# Create GPU plugin daemonset +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/monitoring_shared-dev_nfd/?ref=' +``` + +#### Install to nodes with Intel GPUs with Fractional resources + +With the experimental fractional resource feature you can use additional kubernetes extended +resources, such as GPU memory, which can then be consumed by deployments. PODs will then only +deploy to nodes where there are sufficient amounts of the extended resources for the containers. + +(For this to work properly, all GPUs in a given node should provide equal amount of resources +i.e. heteregenous GPU nodes are not supported.) + +Enabling the fractional resource feature isn't quite as simple as just enabling the related +command line flag. The DaemonSet needs additional RBAC-permissions +and access to the kubelet podresources gRPC service, plus there are other dependencies to +take care of, which are explained below. For the RBAC-permissions, gRPC service access and +the flag enabling, it is recommended to use kustomization by running: + +```bash +# Start NFD with GPU related configuration changes +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/gpu?ref=' + +# Create NodeFeatureRules for detecting GPUs on nodes +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=' + +# Create GPU plugin daemonset +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/overlays/fractional_resources?ref=' +``` + +##### Fractional resources details + +Usage of these fractional GPU resources requires that the cluster has node +extended resources with the name prefix `gpu.intel.com/`. Those can be created with NFD +by running the [hook](/cmd/gpu_nfdhook/) installed by the plugin initcontainer. When fractional resources are +enabled, the plugin lets a [scheduler extender](https://github.com/intel/platform-aware-scheduling/tree/master/gpu-aware-scheduling) +do card selection decisions based on resource availability and the amount of extended +resources requested in the [pod spec](https://github.com/intel/platform-aware-scheduling/blob/master/gpu-aware-scheduling/docs/usage.md#pods). + +The scheduler extender then needs to annotate the pod objects with unique +increasing numeric timestamps in the annotation `gas-ts` and container card selections in +`gas-container-cards` annotation. The latter has container separator '`|`' and card separator +'`,`'. Example for a pod with two containers and both containers getting two cards: +`gas-container-cards:card0,card1|card2,card3`. Enabling the fractional-resource support +in the plugin without running such an annotation adding scheduler extender in the cluster +will only slow down GPU-deployments, so do not enable this feature unnecessarily. + +In multi-tile systems, containers can request individual tiles to improve GPU resource usage. +Tiles targeted for containers are specified to pod via `gas-container-tiles` annotation where the the annotation +value describes a set of card and tile combinations. For example in a two container pod, the annotation +could be `gas-container-tiles:card0:gt0+gt1|card1:gt1,card2:gt0`. Similarly to `gas-container-cards`, the container +details are split via `|`. In the example above, the first container gets tiles 0 and 1 from card 0, +and the second container gets tile 1 from card 1 and tile 0 from card 2. + +> **Note**: It is also possible to run the GPU device plugin using a non-root user. To do this, +the nodes' DAC rules must be configured to device plugin socket creation and kubelet registration. +Furthermore, the deployments `securityContext` must be configured with appropriate `runAsUser/runAsGroup`. + +### Verify Plugin Registration + +You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes: + +```bash +$ kubectl get nodes -o=jsonpath="{range .items[*]}{.metadata.name}{'\n'}{' i915: '}{.status.allocatable.gpu\.intel\.com/i915}{'\n'}" +master + i915: 1 +``` + +## Testing and Demos + +The GPU plugin functionality can be verified by deploying an [OpenCL image](../../demo/intel-opencl-icd/) which runs `clinfo` outputting the GPU capabilities (detected by driver installed to the image). + +1. Make the image available to the cluster: + + Build image: + + ```bash + $ make intel-opencl-icd + ``` + + Tag and push the `intel-opencl-icd` image to a repository available in the cluster. Then modify the `intelgpu-job.yaml`'s image location accordingly: + + ```bash + $ docker tag intel/intel-opencl-icd:devel /intel/intel-opencl-icd:latest + $ docker push /intel/intel-opencl-icd:latest + $ $EDITOR ${INTEL_DEVICE_PLUGINS_SRC}/demo/intelgpu-job.yaml + ``` + + If you are running the demo on a single node cluster, and do not have your own registry, you can add image to node image cache instead. For example, to import docker image to containerd cache: + + ```bash + $ IMAGE_NAME=opencl-icd.tar + $ docker save -o $IMAGE_NAME intel/intel-opencl-icd:devel + $ ctr -n=k8s.io images import $IMAGE_NAME + $ rm $IMAGE_NAME + ``` + +1. Create a job: + + ```bash + $ kubectl apply -f ${INTEL_DEVICE_PLUGINS_SRC}/demo/intelgpu-job.yaml + job.batch/intelgpu-demo-job created + ``` + +1. Review the job's logs: + + ```bash + $ kubectl get pods | fgrep intelgpu + # substitute the 'xxxxx' below for the pod name listed in the above + $ kubectl logs intelgpu-demo-job-xxxxx + + ``` + + If the pod did not successfully launch, possibly because it could not obtain + the requested GPU resource, it will be stuck in the `Pending` status: + + ```bash + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + intelgpu-demo-job-xxxxx 0/1 Pending 0 8s + ``` + + This can be verified by checking the Events of the pod: + + ```bash + $ kubectl describe pod intelgpu-demo-job-xxxxx + ... + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling default-scheduler 0/1 nodes are available: 1 Insufficient gpu.intel.com/i915. + ``` + + +## Issues with media workloads on multi-GPU setups + +OneVPL media API, 3D and compute APIs provide device discovery +functionality for applications and work fine in multi-GPU setups. +VA-API and legacy QSV (MediaSDK) media APIs do not, and do not +provide (e.g. environment variable) override for their _default_ +device file. + +As result, media applications using VA-API or QSV, fail to locate the +correct GPU device file unless it is the first ("renderD128") one, or +device file name is explictly specified with an application option. + +Kubernetes device plugins expose only requested number of device +files, and their naming matches host device file names (for several +reasons unrelated to media). Therefore, on multi-GPU hosts, the only +GPU device file mapped to the media container can differ from +"renderD128", and media applications using VA-API or QSV need to be +explicitly told which one to use. + +These options differ from application to application. Relevant FFmpeg +options are documented here: +* VA-API: https://trac.ffmpeg.org/wiki/Hardware/VAAPI +* QSV: https://github.com/Intel-Media-SDK/MediaSDK/wiki/FFmpeg-QSV-Multi-GPU-Selection-on-Linux + + +### Workaround for QSV and VA-API + +[Render device](render-device.sh) shell script locates and outputs the +correct device file name. It can be added to the container and used +to give device file name for the application. + +Use it either from another script invoking the application, or +directly from the Pod YAML command line. In latter case, it can be +used either to add the device file name to the end of given command +line, like this: + +```bash +command: ["render-device.sh", "vainfo", "--display", "drm", "--device"] + +=> /usr/bin/vainfo --display drm --device /dev/dri/renderDXXX +``` + +Or inline, like this: + +```bash +command: ["/bin/sh", "-c", + "vainfo --device $(render-device.sh 1) --display drm" + ] +``` + +If device file name is needed for multiple commands, one can use shell variable: + +```bash +command: ["/bin/sh", "-c", + "dev=$(render-device.sh 1) && vainfo --device $dev && " + ] +``` + +With argument N, script outputs name of the Nth suitable GPU device +file, which can be used when more than one GPU resource was requested. diff --git a/0.27/_sources/cmd/iaa_plugin/README.md.txt b/0.27/_sources/cmd/iaa_plugin/README.md.txt new file mode 100644 index 000000000..d2f067a1f --- /dev/null +++ b/0.27/_sources/cmd/iaa_plugin/README.md.txt @@ -0,0 +1,120 @@ +# Intel IAA device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Installation](#installation) + * [Pre-built images](#pre-built-images) + * [Verify plugin registration](#verify-plugin-registration) +* [Testing and Demos](#testing-and-demos) + +## Introduction + +The IAA device plugin for Kubernetes supports acceleration using the Intel Analytics accelerator(IAA). + +The IAA plugin discovers IAA work queues and presents them as a node resources. + +The IAA plugin and operator optionally support provisioning of IAA devices and workqueues with the help of [accel-config](https://github.com/intel/idxd-config) utility through initcontainer. + +## Installation + +The following sections detail how to use the IAA device plugin. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-iaa-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/iaa_plugin?ref=' +daemonset.apps/intel-iaa-plugin created +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +Nothing else is needed. See [the development guide](../../DEVEL.md) for details if you want to deploy a customized version of the plugin. + +#### Automatic Provisioning + +There's a sample [idxd initcontainer](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/build/docker/intel-idxd-config-initcontainer.Dockerfile) included that provisions IAA devices and workqueues (1 engine / 1 group / 1 wq (user/dedicated)), to deploy: + +```bash +$ kubectl apply -k deployments/iaa_plugin/overlays/iaa_initcontainer/ +``` + +The provisioning [script](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/idxd-init.sh) and [template](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/demo/iaa.conf) are available for customization. + +The provisioning config can be optionally stored in the ProvisioningConfig configMap which is then passed to initcontainer through the volume mount. + +There's also a possibility for a node specific congfiguration through passing a nodename via NODE_NAME into initcontainer's environment and passing a node specific profile via configMap volume mount. + +To create a custom provisioning config: + +```bash +$ kubectl create configmap --namespace=inteldeviceplugins-system intel-iaa-config --from-file=demo/iaa.conf +``` + +### Verify Plugin Registration + +You can verify the plugin has been registered with the expected nodes by searching for the relevant +resource allocation status on the nodes: + +```bash +$ kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{" "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}' | grep '^\([^ ]\)\|\( iaa\)' +master + iaa.intel.com/wq-user-dedicated: 2 + iaa.intel.com/wq-user-shared: 10 +node1 + iaa.intel.com/wq-user-dedicated: 4 + iaa.intel.com/wq-user-shared: 30 +``` + +## Testing and Demos + +We can test the plugin is working by deploying the provided example iaa-qpl-demo test image. + +1. Build a Docker image with an accel-config tests: + + ```bash + $ make iaa-qpl-demo + ... + Successfully tagged intel/iaa-qpl-demo:devel + ``` + +1. Create a pod running unit tests off the local Docker image: + + ```bash + $ kubectl apply -f ./demo/iaa-qpl-demo-pod.yaml + pod/iaa-qpl-demo created + ``` + +1. Wait until pod is completed: + + ```bash + $ kubectl get pods |grep iaa-qpl-demo + iaa-qpl-demo 0/1 Completed 0 31m + + If the pod did not successfully launch, possibly because it could not obtain the IAA + resource, it will be stuck in the `Pending` status: + + ```bash + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + iaa-qpl-demo 0/1 Pending 0 7s + ``` + + This can be verified by checking the Events of the pod: + + ```bash + + $ kubectl describe pod iaa-qpl-demo | grep -A3 Events: + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 2m26s default-scheduler 0/1 nodes are available: 1 Insufficient iaa.intel.com/wq-user-dedicated, 1 Insufficient iaa.intel.com/wq-user-shared. + ``` diff --git a/0.27/_sources/cmd/operator/README.md.txt b/0.27/_sources/cmd/operator/README.md.txt new file mode 100644 index 000000000..22da75113 --- /dev/null +++ b/0.27/_sources/cmd/operator/README.md.txt @@ -0,0 +1,189 @@ +# Intel Device Plugins Operator + +Table of Contents + +* [Introduction](#introduction) +* [Installation](#installation) +* [Upgrade](#upgrade) +* [Limiting Supported Devices](#limiting-supported-devices) +* [Known issues](#known-issues) + +## Introduction + +Intel Device Plugins Operator is a Kubernetes custom controller whose goal is to serve the +installation and lifecycle management of Intel device plugins for Kubernetes. +It provides a single point of control for GPU, QAT, SGX, FPGA, DSA and DLB devices to a cluster +administrators. + +## Installation + +The default operator deployment depends on NFD and cert-manager. Those components have to be installed to the cluster before the operator can be deployed. + +> **Note**: Operator can also be installed via Helm charts. See [INSTALL.md](../../INSTALL.md) for details. + +### NFD + +Install NFD (if it's not already installed) and node labelling rules (requires NFD v0.13+): + +``` +# deploy NFD +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd?ref=' +# deploy NodeFeatureRules +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/nfd/overlays/node-feature-rules?ref=' +``` +Make sure both NFD master and worker pods are running: + +``` +$ kubectl get pods -n node-feature-discovery +NAME READY STATUS RESTARTS AGE +nfd-master-599c58dffc-9wql4 1/1 Running 0 25h +nfd-worker-qqq4h 1/1 Running 0 25h +``` + +Note that labelling is not performed immediately. Give NFD 1 minute to pick up the rules and label nodes. + +As a result all found devices should have correspondent labels, e.g. for Intel DLB devices the label is +`intel.feature.node.kubernetes.io/dlb`: +``` +$ kubectl get no -o json | jq .items[].metadata.labels |grep intel.feature.node.kubernetes.io/dlb + "intel.feature.node.kubernetes.io/dlb": "true", +``` + +Full list of labels can be found in the deployments/operator/samples directory: +``` +$ grep -r feature.node.kubernetes.io/ deployments/operator/samples/ +deployments/operator/samples/deviceplugin_v1_dlbdeviceplugin.yaml: intel.feature.node.kubernetes.io/dlb: 'true' +deployments/operator/samples/deviceplugin_v1_qatdeviceplugin.yaml: intel.feature.node.kubernetes.io/qat: 'true' +deployments/operator/samples/deviceplugin_v1_sgxdeviceplugin.yaml: intel.feature.node.kubernetes.io/sgx: 'true' +deployments/operator/samples/deviceplugin_v1_gpudeviceplugin.yaml: intel.feature.node.kubernetes.io/gpu: "true" +deployments/operator/samples/deviceplugin_v1_fpgadeviceplugin.yaml: intel.feature.node.kubernetes.io/fpga-arria10: 'true' +deployments/operator/samples/deviceplugin_v1_dsadeviceplugin.yaml: intel.feature.node.kubernetes.io/dsa: 'true' +``` + +### Cert-Manager + +The default operator deployment depends on [cert-manager](https://cert-manager.io/) running in the cluster. +See installation instructions [here](https://cert-manager.io/docs/installation/kubectl/). + +Make sure all the pods in the `cert-manager` namespace are up and running: + +``` +$ kubectl get pods -n cert-manager +NAME READY STATUS RESTARTS AGE +cert-manager-7747db9d88-bd2nl 1/1 Running 0 21d +cert-manager-cainjector-87c85c6ff-59sb5 1/1 Running 0 21d +cert-manager-webhook-64dc9fff44-29cfc 1/1 Running 0 21d +``` + +### Device Plugin Operator + +Finally deploy the operator itself: + +``` +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/operator/default?ref=' +``` + +Now you can deploy the device plugins by creating corresponding custom resources. +The samples for them are available [here](/deployments/operator/samples/). + +### Device Plugin Custom Resource + +Deploy your device plugin by applying its custom resource, e.g. +`GpuDevicePlugin` with + +```bash +$ kubectl apply -f https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/main/deployments/operator/samples/deviceplugin_v1_gpudeviceplugin.yaml +``` + +Observe it is up and running: + +```bash +$ kubectl get GpuDevicePlugin +NAME DESIRED READY NODE SELECTOR AGE +gpudeviceplugin-sample 1 1 5s +``` + +## Upgrade + +The upgrade of the deployed plugins can be done by simply installing a new release of the operator. + +The operator auto-upgrades operator-managed plugins (CR images and thus corresponding deployed daemonsets) to the current release of the operator. + +During upgrade the tag in the image path is updated (e.g. docker.io/intel/intel-sgx-plugin:tag), but the rest of the path is left intact. + +No upgrade is done for: +- Non-operator managed deployments +- Operator deployments without numeric tags + +## Limiting Supported Devices + +In order to limit the deployment to a specific device type, +use one of kustomizations under `deployments/operator/device`. + +For example, to limit the deployment to FPGA, use: + +```bash +$ kubectl apply -k deployments/operator/device/fpga +``` + +Operator also supports deployments with multiple selected device types. +In this case, create a new kustomization with the necessary resources +that passes the desired device types to the operator using `--device` +command line argument multiple times. + +## Known issues + +### Cluster behind a proxy + +If your cluster operates behind a corporate proxy make sure that the API +server is configured not to send requests to cluster services through the +proxy. You can check that with the following command: + +```bash +$ kubectl describe pod kube-apiserver --namespace kube-system | grep -i no_proxy | grep "\.svc" +``` + +In case there's no output and your cluster was deployed with `kubeadm` open +`/etc/kubernetes/manifests/kube-apiserver.yaml` at the control plane nodes and +append `.svc` and `.svc.cluster.local` to the `no_proxy` environment variable: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + ... +spec: + containers: + - command: + - kube-apiserver + - --advertise-address=10.237.71.99 + ... + env: + - name: http_proxy + value: http://proxy.host:8080 + - name: https_proxy + value: http://proxy.host:8433 + - name: no_proxy + value: 127.0.0.1,localhost,.example.com,10.0.0.0/8,.svc,.svc.cluster.local + ... +``` + +**Note:** To build clusters using `kubeadm` with the right `no_proxy` settings from the very beginning, +set the cluster service names to `$no_proxy` before `kubeadm init`: + +``` +$ export no_proxy=$no_proxy,.svc,.svc.cluster.local +``` + +### Leader election enabled + +When the operator is run with leader election enabled, that is with the option +`--leader-elect`, make sure the cluster is not overloaded with excessive +number of pods. Otherwise a heart beat used by the leader election code may trigger +a timeout and crash. We are going to use different clients for the controller and +leader election code to alleviate the issue. See more details in +https://github.com/intel/intel-device-plugins-for-kubernetes/issues/476. + +In case the deployment is limited to specific device type(s), +the CRDs for other device types are still created, but no controllers +for them are registered. diff --git a/0.27/_sources/cmd/operator/ocp_quickstart_guide/README.md.txt b/0.27/_sources/cmd/operator/ocp_quickstart_guide/README.md.txt new file mode 100644 index 000000000..6c6bf0ebd --- /dev/null +++ b/0.27/_sources/cmd/operator/ocp_quickstart_guide/README.md.txt @@ -0,0 +1,66 @@ +# Intel® Device Plugins Operator for Red Hat OpenShift Container Platform + +## Table of Contents +* [Introduction](#introduction) +* [Minimum Hardware Requirements](#minimum-hardware-requirements) + * [Intel SGX Enabled Server](#intel-sgx-enabled-server) +* [Installation](#installation) + * [Prerequisites](#prerequisites) + * [Install Operator using OpenShift Web Console](#install-operator-using-openshift-web-console) + * [Verify Operator installation](#verify-operator-installation) +* [Deploying Intel Device Plugins](#deploying-intel-device-plugins) + * [Intel SGX Device Plugin](#intel-sgx-device-plugin) + +## Introduction +The Intel Device Plugins Operator for OpenShift Container Platform is a collection of device plugins advertising Intel specific hardware resources to the kubelet. It provides a single point of control for Intel® Software Guard Extensions (Intel® SGX), Intel GPUs, Intel® QuickAccess Technology (Intel® QAT), Intel® Data Streaming Accelerator (Intel® DSA), and Intel® In-Memory Analytics Accelerator (Intel® IAA) devices to cluster administrators. The [`v0.24.0`](https://github.com/intel/intel-device-plugins-for-kubernetes/releases/tag/v0.24.0) release of the operator only supports Intel SGX and Intel QAT device plugins. GPU, Intel DSA, Intel IAA, and other device plugins will be supported in future releases. + +## Minimum Hardware Requirements +### Intel SGX Enabled Server +- Third Generation Intel® Xeon® Scalable Platform, code-named “Ice Lake” or later +- Configure BIOS using below details + ![SGX Server BIOS](images/SGX-BIOS.PNG) + [**Note:** The BIOS configuration shown above is just for the reference. Please contact your BIOS vendor for details] + +## Installation +### Prerequisites +- Make sure Red Hat OpenShift Cluster is ready to use and the developing machine is RHEL and `oc` command is installed and configured properly. Please note that the following operation is verified on Red Hat OpenShift Cluster 4.11 and working machine RHEL-8.6 +- Install the `oc` command to your development machine +- Follow the [link](https://docs.openshift.com/container-platform/4.11/hardware_enablement/psap-node-feature-discovery-operator.html) to install **NFD operator** (if it's not already installed). + **Note:** Please only install the NFD operator and use steps below to create the NodeFeatureDiscovery instance. + - Create the NodeFeatureDiscovery instance + ``` + $ oc apply -f https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/v0.24.0/deployments/nfd/overlays/node-feature-discovery/node-feature-discovery-openshift.yaml + ``` + - Create the NodeFeatureRule instance + ``` + $ oc apply -f https://raw.githubusercontent.com/intel/intel-device-plugins-for-kubernetes/v0.24.0/deployments/nfd/overlays/node-feature-rules/node-feature-rules-openshift.yaml + ``` +- Deploy SELinux Policy for OCP 4.10 - + The SGX device plugin and Init container run as a label `container_device_plugin_t` and `container_device_plugin_init_t` respectively. This requires a custom SELinux policy to be deployed before the SGX plugin can be run. To deploy this policy, run + ``` + $ oc apply -f https://raw.githubusercontent.com/intel/user-container-selinux/main/policy-deployment.yaml + ``` + +### Install Operator using OpenShift Web Console +1. In OpenShift web console navigate to **Operator** -> **OperatorHub** +2. Search for **Intel Device Plugins Operator ->** Click **Install** + + +### Verify Operator installation +1. Go to **Operator** -> **Installed Operators** +2. Verify the status of operator as **Succeeded** +3. Click **Intel Device Plugins Operator** to view the details + ![Verify Operator](images/verify-operator.PNG) + + +## Deploying Intel Device Plugins + +### Intel SGX Device Plugin +Follow the steps below to deploy Intel SGX Device Plugin Custom Resource +1. Go to **Operator** -> **Installed Operators** +2. Open **Intel Device Plugins Operator** +3. Navigate to tab **Intel Software Guard Extensions Device Plugin** +4. Click **Create SgxDevicePlugin ->** set correct parameters -> Click **Create** + OR for any customizations, please select `YAML view` and edit details. Once done, click **Create** +5. Verify CR by checking the status of DaemonSet **`intel-sgx-plugin`** +6. Now `SgxDevicePlugin` is ready to deploy any workloads diff --git a/0.27/_sources/cmd/qat_plugin/README.md.txt b/0.27/_sources/cmd/qat_plugin/README.md.txt new file mode 100644 index 000000000..15875480e --- /dev/null +++ b/0.27/_sources/cmd/qat_plugin/README.md.txt @@ -0,0 +1,256 @@ +# Intel QuickAssist Technology (QAT) device plugin for Kubernetes + +Table of Contents + +* [Introduction](#introduction) +* [Modes and Configuration Options](#modes-and-configuration-options) +* [Installation](#installation) + * [Prerequisites](#prerequisites) + * [Pre-built Images](#pre-built-images) + * [Verify Plugin Registration](#verify-plugin-registration) +* [Demos and Testing](#demos-and-testing) + * [DPDK QAT Demos](#dpdk-qat-demos) + * [DPDK Prerequisites](#dpdk-prerequisites) + * [Deploy the pod](#deploy-the-pod) + * [Manual test run](#manual-test-run) + * [Automated test run](#automated-test-run) + * [OpenSSL QAT Demo](#openssl-qat-demo) +* [Checking for Hardware](#checking-for-hardware) + +## Introduction + +This Intel QAT device plugin provides support for Intel QAT devices under Kubernetes. +The supported devices are determined by the VF device drivers available in your Linux +Kernel. See the [Prerequisites](#prerequisites) section for more details. + +Supported Devices include, but may not be limited to, the following: + +- [Intel® Xeon® with Intel® C62X Series Chipset][1] +- Intel® Xeon® with Intel® QAT Gen4 devices +- [Intel® Atom™ Processor C3000][2] +- [Intel® Communications Chipset 8925 to 8955 Series][3] + +The QAT device plugin provides access to QAT hardware accelerated cryptographic and compression features. +Demonstrations are provided utilising [DPDK](https://doc.dpdk.org/) and [OpenSSL](https://www.openssl.org/). + +[Kata Containers](https://katacontainers.io/) QAT integration is documented in the +[Kata Containers documentation repository][6]. + +## Modes and Configuration Options + +The QAT plugin can take a number of command line arguments, summarised in the following table: + +| Flag | Argument | Meaning | +|:---- |:-------- |:------- | +| -dpdk-driver | string | DPDK Device driver for configuring the QAT device (default: `vfio-pci`) | +| -kernel-vf-drivers | string | Comma separated VF Device Driver of the QuickAssist Devices in the system. Devices supported: DH895xCC, C62x, C3xxx, 4xxx/401xx/402xx, C4xxx and D15xx (default: `c6xxvf,4xxxvf`) | +| -max-num-devices | int | maximum number of QAT devices to be provided to the QuickAssist device plugin (default: `64`) | +| -mode | string | plugin mode which can be either `dpdk` or `kernel` (default: `dpdk`) | +| -allocation-policy | string | 2 possible values: balanced and packed. Balanced mode spreads allocated QAT VF resources balanced among QAT PF devices, and packed mode packs one QAT PF device full of QAT VF resources before allocating resources from the next QAT PF. (There is no default.) | + +The plugin also accepts a number of other arguments related to logging. Please use the `-h` option to see +the complete list of logging related options. + +For more details on the `-dpdk-driver` choice, see +[DPDK Linux Driver Guide](http://dpdk.org/doc/guides/linux_gsg/linux_drivers.html). + +> **Note:**: With Linux 5.9+ kernels the `vfio-pci` module must be loaded with +> `disable_denylist=1` parameter for the QAT device plugin to work correctly with +> devices prior to Gen4 (`4xxx`). + +For more details on the available options to the `-kernel-vf-drivers` option, see the list of +vf drivers available in the [Linux Kernel](https://github.com/torvalds/linux/tree/master/drivers/crypto/qat). + +If the `-mode` parameter is set to `kernel`, no other parameter documented above are valid, +except the `klog` logging related parameters. +`kernel` mode implements resource allocation based on system configured [logical instances][7]. + +> **Note**: `kernel` mode is excluded by default from all builds (including those hosted on the Docker hub), +> by default. See the [Build the plugin image](#build-the-plugin-image) section for more details. + +The `kernel` mode does not guarantee full device isolation between containers +and therefore it's not recommended. This mode will be deprecated and removed once `libqat` +implements non-UIO based device access. + +## Installation + +The below sections cover how to obtain, build and install this component. + +The component can be installed either using a DaemonSet or running 'by hand' on each node. + +### Prerequisites + +The component has the same basic dependancies as the +[generic plugin framework dependencies](../../README.md#about). + +You will also need [appropriate hardware installed](#checking-for-hardware). + +The QAT plugin requires Linux Kernel VF QAT drivers to be available. These drivers +are available via two methods. One of them must be installed and enabled: + +- [Linux Kernel upstream drivers](https://github.com/torvalds/linux/tree/master/drivers/crypto/qat) +- [Intel QuickAssist Technology software for Linux][9] + +The demonstrations have their own requirements, listed in their own specific sections. + +### Pre-built Images + +[Pre-built images](https://hub.docker.com/r/intel/intel-qat-plugin) +of this component are available on the Docker hub. These images are automatically built and uploaded +to the hub from the latest main branch of this repository. + +Release tagged images of the components are also available on the Docker hub, tagged with their +release version numbers in the format `x.y.z`, corresponding to the branches and releases in this +repository. Thus the easiest way to deploy the plugin in your cluster is to run this command + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_plugin?ref=' +``` + +Where `` needs to be substituted with the desired [release tag](https://github.com/intel/intel-device-plugins-for-kubernetes/tags) or `main` to get `devel` images. + +An alternative kustomization for deploying the plugin is with the debug mode switched on: + +```bash +$ kubectl apply -k 'https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/qat_plugin/overlays/debug?ref=' +``` + +> **Note**: It is also possible to run the QAT device plugin using a non-root user. To do this, +> the nodes' DAC rules must be configured to allow PCI driver unbinding/binding, device plugin +> socket creation and kubelet registration. Furthermore, the deployments `securityContext` must +> be configured with appropriate `runAsUser/runAsGroup`. + +#### Automatic Provisioning + +There's a sample [qat initcontainer](https://github.com/intel/intel-device-plugins-for-kubernetes/blob/main/build/docker/intel-qat-initcontainer.Dockerfile). Regardless of device types, the script running inside the initcontainer enables QAT SR-IOV VFs. + +To deploy, run as follows: + +```bash +$ kubectl apply -k deployments/qat_plugin/overlays/qat_initcontainer/ +``` + +In addition to the default configuration, you can add device-specific configurations via ConfigMap. + +| Device | Possible Configuration | How To Customize | Options | Notes | +|:-------|:-----------------------|:-----------------|:--------|:------| +| 4xxx, 401xx,402xx | [cfg_services](https://github.com/torvalds/linux/blob/42e66b1cc3a070671001f8a1e933a80818a192bf/Documentation/ABI/testing/sysfs-driver-qat) reports the configured services (crypto services or compression services) of the QAT device. | `ServicesEnabled=` | compress:`dc`, crypto:`sym;asym` | Linux 6.0+ kernel is required. | + +To create a provisioning `configMap`, run the following command before deploying initcontainer: + +```bash +$ kubectl create configmap --namespace=inteldeviceplugins-system qat-config --from-file=/path/to/qat.conf +``` +or +```bash +$ kubectl create configmap --namespace=inteldeviceplugins-system --from-literal "qat.conf=ServicesEnabled=