Build OPA docker image with builtin policies & Run OPA as a side car

[t83714]

Motivation

• Build OPA docker image with builtin policies
  • Currently, built-in policies are deployed as part of helm chart content
  • Due to k8s etcd 1m limit, unless use other storage targets, helm chart deployment manifest (after compression) can't be larger than 1M.
  • We haven't reached the limit yet but it's still good to move built-in policies to prebuilt docker image to spare the room
  • the user should still be able to supply their own policies files (as part of helm config) to overwrite built-in policy files
• Move OPA container into auth API pod as a sidecar container
  • Performance will be better & ensure it's highly available to auth API
  • In Magda, we never access auth API directly. Instead, we access it via auth API's decision endpoint to make sure the user profile has been prefilled during the policy evaluation.
  • To run opa as a sidecar in auth API pod will make management & scale up / down easier as we don't need to manage two pods anymore.

Acceptance Criteria

• Build OPA docker image with builtin policies
• The docker image should be multi-arch images that support linux/arm64
• the user should still be able to supply their own policies files (as part of helm config) to overwrite built-in policy files
  • https://github.com/magda-io/magda-configmap-dir-loader might need to be adjusted for this
  • see Multi-arch docker images & upgrade to node 14 magda-configmap-dir-loader#1
• Move OPA container into auth API pod as a sidecar container
• Make sure liveness readiness probs covers opa container as well

[t83714]

closed via #3327