Remove nginx config files from cloud proxy container in favor of Conf…

…igmaps for easier runtime overrides (#2018)

Summary: Remove nginx config files from cloud proxy container in favor
of Configmaps for easier runtime overrides

This is an alternative approach to #2014 and #2016. While this doesn't
provide an environment variable for configuring the intended behavior,
this approach is more flexible since many Nginx directives don't work
with variables (`server_name`, `resolver`, among others ).

Because nginx prohibits variables in these directives, it makes it very
difficult to provide environment variable based settings without our
previous `sed` approach. The `sed` approach also has its problems since
it requires
[hacks](https://github.com/pixie-io/pixie/pull/2014/files#diff-5ec7ca8d0f624fe1f4eb3778cc96dcee2f999bf39bad422807b67b15ce2f8e7bR27)
to support configuration removals. Rather than trying to solve all
potential use cases, this PR opts to make the configuration easy to swap
out via the `pl-proxy-nginx-config` Configmap.

I plan to update the self hosted cloud docs to call out that this
Configmap exists and should be used if custom nginx configuration is
needed outside of the upstream defaults.

Relevant Issues: #2017

Type of change: /kind feature

Test Plan: Deployed to a cloud environment and verified that the
upstream defaults and `PL_DOMAIN_NAME` apply as expected

Changelog Message: Removed nginx configuration from the container image
into `pl-proxy-nginx-config` Configmap for easier runtime overrides

---------

Signed-off-by: Dom Del Nano <[email protected]>

k8s/cloud/base/kustomization.yaml

@@ -44,3 +44,4 @@ resources:
 - cron_script_deployment.yaml
 - cron_script_service.yaml
 - support_access_config.yaml
+- proxy_nginx_config.yaml

k8s/cloud/base/proxy_deployment.yaml

@@ -78,6 +78,8 @@ spec:
         volumeMounts:
         - name: certs
           mountPath: /certs
+        - name: nginx-config
+          mountPath: /usr/local/openresty/nginx/conf
         securityContext:
           allowPrivilegeEscalation: false
           capabilities:
@@ -146,3 +148,6 @@ spec:
       - name: certs
         secret:
           secretName: cloud-proxy-tls-certs
+      - name: nginx-config
+        configMap:
+          name: pl-proxy-nginx-config