Merging to release-4.3: Custom plugins Rewrite (#2135) (#2230)

Custom plugins Rewrite (#2135)

Re-word the custom plugins bit to further declutter.

Simplify by making the navigation geared toward new starters, with an
easy-walk-through tutorial of building and deploying their first custom
plugin.

Also, recommend Go plugins as the preferred plugin type.

tyk-docs/content/plugins.md

@@ -9,20 +9,28 @@ aliases:
     - /customise-tyk/plugins/
 ---
 
-Tyk supports the use of the following plugins to extend Tyk functionality:
+Tyk supports the use of custom plugins to extend Tyk functionality.
 
-*   [Python, Lua, gRPC (Rich Plugins)]({{< ref "plugins/supported-languages/rich-plugins" >}})
-*   [JavaScript Plugins]({{< ref "plugins/supported-languages/javascript-middleware" >}}) (JSVM Middleware)
-*   [Golang native plugins]({{< ref "plugins/supported-languages/golang" >}})
+Plugins can be executed in the **following order** inside the following areas of the [API Request Lifecycle]({{< ref "concepts/middleware-execution-order" >}}):
 
-inside the following areas of the API Request Lifecycle
-
-*   [Authentication Plugins]({{< ref "plugins/auth-plugins" >}})
-*   [Request Plugins]({{< ref "plugins/request-plugins" >}})
-*   [Response Plugins]({{< ref "plugins/response-plugins" >}})
+*   [Pre (Request) Plugin]({{< ref "plugins/plugin-types/request-plugins" >}})
+*   [Authentication Plugin]({{< ref "plugins/plugin-types/auth-plugins/auth-plugins" >}})
+*   [Post-Auth (Request) Plugin]({{< ref "plugins/plugin-types/request-plugins" >}})
+*   [Post (Request) Plugin]({{< ref "plugins/plugin-types/request-plugins" >}})
+*   [Response Plugin]({{< ref "plugins/plugin-types/response-plugins" >}})
+*   [Analytics Plugin]({{< ref "plugins/plugin-types/analytics-plugins" >}})
 
 ### Plugin Caveats
 
 *   They must run as a single process.
-*   The plugins used *must* be specified in an API definition and are not global across APIs.
+*   To apply a custom plugin to an API you must modify the API definition and add the plugin information to one of the areas of the API Request Lifecycle mentioned above.
 *   They must manage API-specific cases in the same process, only one CoProcess will be managed by a Tyk Instance.
+
+### Language Support
+
+Tyk recommends using Go plugins for performance, flexibility, and nativity reasons (All Tyk components are written in Go).
+
+The following languages are supported for custom plugins:
+*   [Golang native plugins]({{< ref "plugins/supported-languages/golang" >}}) - fast, native performance
+*   [JavaScript Plugins]({{< ref "plugins/supported-languages/javascript-middleware" >}}) (JSVM Middleware) - simple with limited direct API when performance not important (same with Python / Lua)
+*   [Python, Lua, gRPC (Rich Plugins)]({{< ref "plugins/supported-languages/rich-plugins" >}}) - ultimate flexibility in the language of implementation, however, there are some performance and management overheads when compared to native GoLang plugins

tyk-docs/content/plugins/get-started-selfmanaged/deploy.md

@@ -0,0 +1,120 @@
+---
+date: 2017-03-24T15:45:13Z
+title: Task 3 - Automating Your Build Process
+menu:
+  main:
+    parent: "Get Started with Custom Plugins"
+weight: 10
+---
+
+It's very important to automate the deployment of your infrastructure.  
+
+Ideally, you store your configurations and code in version control, and then through a trigger, have the ability to deploy everything automatically into higher environments.
+
+With custom plugins, this is no different.
+
+To illustrate this, we can look at the GitHub Actions of the [example repo][0].
+
+We see that upon every pull request, a section of steps are taken to "Build, [Bundle]({{< ref "plugins/how-to-serve-plugins/plugin-bundles" >}}), Release Go Plugin".
+
+Let's break down the [workflow file][1]:
+
+
+## 1. Compiling the Plugin
+
+We can see the first few steps replicate our first task, bootstrapping the environment and compiling the plugin into a binary format.
+
+```make
+ steps:
+    - uses: actions/checkout@v3
+
+    - name: Copy Env Files
+      run: cp tyk/confs/tyk_analytics.env.example tyk/confs/tyk_analytics.env
+
+    - name: Build Go Plugin
+      run: make go-build
+```
+
+We can look at the [Makefile][2] to further break down the last `go-build` command.
+
+## 2. Bundle The Plugin
+
+The next step of the workflow is to "[bundle]({{< ref "plugins/how-to-serve-plugins/plugin-bundles" >}})" the plugin.
+
+```
+- name: Bundle Go Plugin
+      run: docker-compose run --rm --user=1000 --entrypoint "bundle/bundle-entrypoint.sh" tyk-gateway
+```
+
+This command generates a "bundle" from the sample Go plugin in the repo.
+
+{{< note success >}}
+**Note**  
+
+For added security, please consider signing your [bundles]({{< ref "plugins/how-to-serve-plugins.md#global-parameters" >}}), especially if the connection between the Gateways and the Bundler server traverses the internet.
+
+{{< /note >}}
+
+
+Custom plugins can be "bundled", (zipped/compressed) into a standard format, and then uploaded to some server so that they can be downloaded by the Gateways in real time.
+
+This process allows us to decouple the building of our custom plugins from the runtime of the Gateways.
+
+In other words, Gateways can be scaled up and down, and pointed at different plugin repos very easily.  This makes it easier to deploy Custom plugins especially in containerized environments such as Kubernetes, where we don't have to worry about persistent volumes.
+
+You can read more about plugin bundles [here][3].
+
+## 3. Deploy The Plugin
+
+Next step of the workflow is to publish our bundle to a server that's reachable by the Gateways.
+
+```make
+- name: Upload Bundle
+      uses: actions/upload-artifact@v3
+      with:
+        name: customgoplugin.zip
+        path: tyk/bundle/bundle.zip
+
+    - uses: jakejarvis/s3-sync-action@master
+      with:
+        args: --acl public-read --follow-symlinks
+      env:
+        AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
+        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        AWS_REGION: 'us-east-1'   
+        SOURCE_DIR: 'tyk/bundle'
+```
+
+This step uploads the Bundle to both GitHub and an AWS S3 bucket.  Obviously, your workflow will look slightly different here.
+
+{{< note success >}}
+**Note**  
+
+For seamless deployments, take a look at multi-version [plugin support]({{< ref "plugins/supported-languages/golang.md#upgrading-tyk" >}}) to enable zero downtime deployments of your Tyk Gateway installs
+
+{{< /note >}}
+
+### 4. Configure the Gateway
+
+In order to instruct the Gateway to download the bundle, we need two things:
+
+1. The root server - The server which all bundles will be downloaded from.  This is set globally in the Tyk conf file [here]({{< ref "tyk-oss-gateway/configuration#enable_bundle_downloader" >}}).
+
+2. The name of the bundle - this is generated during your workflow usually.  This is defined at the API level (this is where you declare Custom plugins, as evident in task 2)
+
+The field of the API Definition that needs to be set is `custom_middleware_bundle`.
+
+## Summary
+
+That's it!  
+
+We've set up our dev environment, built, compiled, a Custom Go plugin, loaded onto a Tyk Gateway, and tested it by sending an API request.  Finally, we've talked about deploying a Bundle in a production grade set up.
+
+Have a look at our [examples repo][4] for more inspiration.
+
+[0]: https://github.com/TykTechnologies/custom-go-plugin/actions
+[1]: https://github.com/TykTechnologies/custom-go-plugin/blob/master/.github/workflows/makefile.yml
+[2]: https://github.com/TykTechnologies/custom-go-plugin/blob/master/Makefile#L59
+[3]: https://github.com/TykTechnologies/custom-go-plugin#deploying-the-go-plugin
+[4]: https://github.com/TykTechnologies/custom-plugin-examples

tyk-docs/content/plugins/get-started-selfmanaged/get-started.md

@@ -0,0 +1,22 @@
+---
+date: 2017-03-24T15:45:13Z
+title: Get Started with Custom Plugins
+menu:
+  main:
+    parent: "Custom Plugins"
+weight: 10
+---
+
+This section takes you through the development process of creating a Custom Go Plugin.
+
+At the end of this process you will have a Tyk environment running locally and a simple Go plugin executing on each API request.
+
+Go plugins are the recommended plugin type and suitable for most use cases.
+## Prerequisites
+
+* docker & docker-compose
+* [A Tyk license](https://tyk.io/sign-up/#self) (if using Self-Managed Tyk, which will make the process easier via UI)
+* Make
+* OSX (Intel)  -> Not a prerequisite, though these steps are tested on OSX Intel/ARM
+
+This tutorial will take between 15-20 minutes.

tyk-docs/content/plugins/get-started-selfmanaged/run.md

@@ -0,0 +1,81 @@
+---
+date: 2017-03-24T15:45:13Z
+title: Task 1 - Set up your environment
+menu:
+  main:
+    parent: "Get Started with Custom Plugins"
+weight: 10
+---
+
+We have a getting started repo to help you set up your development environment.
+
+### 1.  Clone the getting started repo
+
+Please clone the [getting started repo][0].
+
+
+
+{{< tabs_start >}}
+{{< tab_start "Self-Managed" >}}
+
+```bash
+git clone https://github.com/TykTechnologies/custom-go-plugin
+
+```
+
+{{< tab_end >}}
+{{< tab_start "Open Source" >}}
+
+```bash
+git clone https://github.com/TykTechnologies/custom-go-plugin
+git checkout opensource
+```
+
+
+{{< tab_end >}}
+{{< tabs_end >}}
+
+### 2. Run the stack
+
+Navigate to the cloned repository and run the following command:
+
+
+{{< tabs_start >}}
+{{< tab_start "Self-Managed" >}}
+
+```shell
+# Make a copy of the example .env file for the Tyk-Dashboard 
+cp tyk/confs/tyk_analytics.env.example tyk/confs/tyk_analytics.env
+```
+Edit the file `tyk/confs/tyk_analytics.env` with your Tyk-Dashboard license key.  
+
+Finally, run the `make` command:
+
+```bash
+make
+```
+
+{{< tab_end >}}
+{{< tab_start "Open Source" >}}
+
+run the `make` command:
+
+```bash
+make
+```
+
+{{< tab_end >}}
+{{< tabs_end >}}
+
+
+This will take a few minutes to run as it compiles the plugin for the first time and downloads all the necessary Docker images.
+
+### What have we done?
+
+1. We've run a local Tyk stack.
+2. We compiled the sample Go plugin and loaded it onto the Gateway's file system.
+
+Next, let's create an API definition and test our Go plugin in the next task.
+
+
+[0]: https://github.com/TykTechnologies/custom-go-plugin