Skip to content

Security and Authz

Jump to bottom
Andrew Azores edited this page Jan 19, 2024 · 31 revisions

General Application Architecture w.r.t Security

The Cryostat 3.0 application as a whole consists of:

  • Cryostat Deployment
    • Service + Route → Auth container
    • Cryostat Pod
      • Auth Proxy container instance
      • Cryostat container instance
      • cryostat-db container instance
        • PersistentVolumeClaim for Postgres Database data
      • (optional) Grafana container instance
      • (optional) jfr-datasource container instance
  • Cryostat Storage Deployment
    • Service + Route → Auth container
    • Cryostat Storage Pod
      • Auth Proxy container instance
      • cryostat-storage container instance
        • PersistentVolumeClaim for cryostat-storage data
  • (optional) Cryostat Report Generator Deployment
    • Service (no Route) → Auth container
    • Cryostat Report Generator Pod(s)
      • Auth Proxy container instance
      • cryostat-report container instance
  • Operator Pod
    • cryostat-operator instance, containing various controllers

The Routes are configured with TLS Re-Encryption so all connections from outside the cluster use HTTPS/WSS using the OpenShift cluster's TLS cert externally. Internally, Service connections between Cryostat components use HTTPS with cert-manager (described in more detail below) to ensure that connections are private even within the cluster namespace. Each Auth Proxy container is either an oauth2-proxy configured with htpasswd Basic authentication, or an openshift-oauth-proxy delegating to the cluster's internal authentication/authorization server.

Scenario

Network topology of Cryostat, its containers, and target applications

In this scenario, the Cryostat Operator is installed into its own namespace. It runs here separately with its privileged serviceaccount. Cryostat CR objects are created to request the Operator to create Cryostat instances. The CR has a field for a list of namespace names that the associated Cryostat instance should be deployed across. When the Cryostat instances are created, they are supplied with an environment variable informing them which namespaces should be monitored. These Cryostat instances are deployed into their own separate install namespaces as well and run with their own lower privileged serviceaccounts. Using these privileges they perform an Endpoints query to discover target applications across each of the listed namespaces. Cryostat will only automatically discover those target applications (potentially including itself) that are located within this namespace. Cryostat queries the k8s/OpenShift API server for Endpoints objects within each namespace, then filters them for ports with either the name jfr-jmx or the number 9091. Other applications, within the namespace or otherwise, may be registered via the Custom Targets API or the Discovery Plugin API (ex. using the Cryostat Agent), but Cryostat will not be aware that these applications may be in other namespaces.

With this setup, the target applications are not able to assume the privileges associated with the serviceaccounts for the Cryostat Operator or each of the Cryostat instances. Each Cryostat instance can discover and become aware of target JVM applications across any of the namespaces that this particular instance is monitoring. The separated namespaces also ease administration and access management, so cluster administrators can assign roles to users that allow them to work on projects within namespaces, and assign other roles to other users that allow them to acces Cryostat instances that may have visibility into those namespaces.

Flow of JFR Data

Cryostat connects to other JVM applications within its cluster using remote JMX, using cluster-internal URLs so that no traffic will leave the cluster. Cryostat supports connecting to target JVMs with JMX auth credentials enabled ("Basic" style authentication). When a connection attempt to a target fails due to a SecurityException, Cryostat responds to the requesting client with an HTTP 427 status code and the header X-JMX-Authenticate: Basic. The client is then expected to retry the request with a header like X-JMX-Authorization: Basic base64(user:pass), which is a protocol modelled after typical HTTP Authenticate/Authorization for Basic auth. When deployed in OpenShift the requests are already encrypted using OpenShift TLS re-encryption as mentioned above, so the credentials are never transmitted in cleartext. It is also possible to configure Cryostat to trust SSL certificates used by target JVMs by adding the certificate to a volume mounted at /truststore on the Cryostat container, which will add the certificate to the SSL trust store used by Cryostat. The Operator also uses cert-manager to generate a self-signed CA and provides Cryostat and Grafana with certificates as a mounted volume, which Cryostat also uses to secure its own JMX connections. Cryostat also enables JMX connection auth using Operator-generated credentials to ensure that only authorized users can connect to Cryostat, including when attempting to have it perform introspective operations. When deployed by the Operator, the JMX username is simply "cryostat" and the password is generated by the Operator, stored in a Secret, and then "mounted" in an environment variable when the Cryostat container is created.

Cryostat and the associated Operator will only monitor the OpenShift namespace(s) that they are deployed within (see Scenarios above), and can only connect to target JVMs within this namespace - this is enforced by OpenShift's networking setup. This way, end user administrators or developers can be sure of which set of JVMs they are running which are visible to Cryostat and thus which JVMs' data they should be mindful of.

Once Cryostat has established a JMX connection to a target application its primary purpose is to enable JFR recordings on the target JVM and expose them to the end user. These recordings can be transferred from the target JVM back to Cryostat over the JMX connection. Cryostat does this for four purposes:

  1. to generate Automated Rules Reports of the JFR contents, served to clients over HTTPS. These may be generated by the Cryostat container itself or by cryostat-reports sidecar container(s) depending on the configuration.
  2. to copy the JFR contents into a file saved in its OpenShift PersistentVolumeClaim ("archive")
  3. to stream a snapshot of the JFR contents over HTTPS to a requesting client's GET request
  4. to upload a snapshot of the JFR contents using HTTPS POST to the jfr-datasource

("archived" JFR copies can also be streamed to clients over HTTPS or POSTed to jfr-datasource, and Automated Rules Reports can also be made of them)

Here, "the client" may refer to an end user's browser when using Cryostat's web interface, or may be the end user using a direct WebSocket or HTTP(S) client (ex. websocat or curl), or may be an OpenShift Operator controller acting as an automated client. All of these cases are handled identically by Cryostat.

jfr-datasource receives file uploads by POST request from the Cryostat container. Cryostat and jfr-datasource run together within the same Pod and use the local loopback network interface, so the file contents do not travel across the network outside of the Pod. These files are held in transient storage by the jfr-datasource container and the parsed JFR data contents held in-memory to make available for querying by the Grafana dashboard container, which also runs within the same Pod and communicates over the local loopback network interface.

Cryostat Authz Specifics

When deployed in OpenShift, the Cryostat container instance detects this scenario and expects clients to provide a Bearer token on all Command Channel (WSS) connections as well as on any HTTPS API connections that can provide information about target applications within the cluster (ie authz are not checked only for requests for things like web-client assets). These tokens are the ones provided by OpenShift OAuth itself, ie. the user's account for that OpenShift instance/cluster. On each HTTPS request, Cryostat receives the token and sends its own request to the internal OpenShift OAuth server to validate the token. If OpenShift OAuth validates the token the request is accepted. If OpenShift OAuth does not validate the token, or the user does not provide a token, then the request is rejected with a 401. Likewise, for each new WSS WebSocket connection, Cryostat expects the client to provide a token as part of the WebSocket SubProtocol header. This token is then passed to the OpenShift OAuth server in the same way previously described. If the token validation fails, the server will reply with an appropriate closure status code and message after the client sends its first message frame.

The specific method for verifying an OAuth token is to receive the client's provided token and construct a new OpenShift client instance using this token, to perform a request while Cryostat masquerades as the actual client. Currently, the request performed is to attempt to list the Routes within the Namespace. This is likely to change in the future to a more robust criterion.

TODO describe non-OpenShift cases

Grafana Authz

The Operator configures the Grafana container to use the default admin username, but the default password is overridden. The Operator generates a random password as below (at the time of writing):

func NewGrafanaSecretForCR(cr *rhjmcv1alpha1.Cryostat) *corev1.Secret {
	return &corev1.Secret{
		ObjectMeta: metav1.ObjectMeta{
			Name:      cr.Name + "-grafana-basic",
			Namespace: cr.Namespace,
		},
		StringData: map[string]string{
			"GF_SECURITY_ADMIN_USER":     "admin",
			"GF_SECURITY_ADMIN_PASSWORD": GenPasswd(20),
		},
	}
}

func GenPasswd(length int) string {
	rand.Seed(time.Now().UnixNano())
	chars := "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_"
	b := make([]byte, length)
	for i := range b {
		b[i] = chars[rand.Intn(len(chars))]
	}
	return string(b)
}

(ie: [a-zA-Z0-9\-_]{20})

This generated password is stored in a Kubernetes Secret, which is then "mounted" into the Grafana container as an environment variable at startup time. This Secret is also re-read by another controller within the Operator at a later time after Grafana container startup so that the Operator can perform API requests to the Grafana container to configure it with a default dashboard and to add the jfr-datasource datasource definition/URL.

Operator Connections to Cryostat

Operator connections to its "child" Cryostat instance are solely via HTTPS, identically as outlined above for all client connections in general. The Operator passes its ServiceAccount API token to Cryostat via bearer authentication. Cryostat then uses this token to masquerade as the ServiceAccount and verify its permissions within the cluster and namespace.

Once the Operator has obtained information about the target JVM(s) within the namespace, this information is copied into Custom Resources owned by the Operator. This information only includes more basic details such as the names, durations, and states of any recordings active in the target JVM(s), as well as the URL to download the recording. This URL is a direct link to the Cryostat Route at a path that allows the recording to be downloaded by the client. This path is also secured using HTTPS Bearer token authentication, so the end user client must supply their own account's token in order to retrieve the recording file. Any information contained within the Custom Resources is secured using OpenShift RBAC policy, similar to other built-in Resource types. Important to note is that the Operator itself does not receive the JFR file itself in whole or in part, so the only information available via the API is information obtained via the Cryostat HTTP API, which a user would also be able to view using the web-client or direct HTTP queries.

Clone this wiki locally