The ElasticSearch operator is designed to manage one or more elastic search clusters. Included in the project (initially) is the ability to create the Elastic cluster, deploy the data nodes across zones in your Kubernetes cluster, and snapshot indexes to AWS S3.
The operator was built and tested on a 1.5.X Kubernetes cluster and is the only version supported currently. This is because it is utilizing StatefulSets which are not available in previous versions of Kubernetes.
The operator was also currently designed to leverage Amazon AWS S3 for snapshot / restore to the elastic cluster. The goal of this project is to extend to support additional clouds and scenarios to make it fully featured.
By swapping out the storage types, this can be used in GKE, but snapshots won't work at the moment.
Watch a demo here:
https://www.youtube.com/watch?v=3HnV7NfgP6A
The operator is built using the controller + third party resource model. Once the controller is deployed to your cluster, it will automatically create the ThirdPartyResource. Next create a Kubernetes object type elasticsearchCluster to deploy the elastic cluster based upon the TPR.
Following parameters are available to customize the elastic cluster:
- client-node-replicas: Number of client node replicas
- master-node-replicas: Number of master node replicas
- data-node-replicas: Number of data node replicas
- zones: Define which zones to deploy data nodes to for high availability (Note: Zones are evenly distributed based upon number of data-node-replicas defined)
- data-volume-size: Size of persistent volume to attach to data nodes
- elastic-search-image: Override the elasticsearch image (e.g.
upmcenterprises/docker-elasticsearch-kubernetes:5.3.1) - snapshot
- scheduler-enabled: If the cron scheduler should be running to enable snapshotting
- bucket-name: Name of S3 bucket to dump snaptshots
- cron-schedule: Cron task definition for intervals to do snapshots
- storage
- type: Defines the type of storage to provision based upon cloud (e.g.
gp2) - storage-class-provisioner: Defines which type of provisioner to use (e.g.
kubernetes.io/aws-ebs)
- type: Defines the type of storage to provision based upon cloud (e.g.
The default image used adds TLS to the Elastic cluster. This secret is used to volume in certs to the cluster dynamically. Provided in this repo are sample certificates, but you are encouraged to change them for your cluster.
$ kubectl create secret generic es-certs --from-file=./certs/node-keystore.jks --from-file=./certs/truststore.jksThe base image used is upmcenterprises/docker-elasticsearch-kubernetes:5.3.1 which can be overriden by addeding to the custom cluster you create (See: ThirdPartyResource above).
NOTE: If no image is specified, the default noted previously is used.
If you are using a private repository you can add a pull secret under spec in your ElasticsearchCluster manifest
spec:
client-node-replicas: 3
data-node-replicas: 3
data-volume-size: 10Gi
java-options: -Xms256m -Xmx256m
master-node-replicas: 2
image-pull-secrets:
- name: pull-secret-name
snapshot:
bucket-name: elasticsnapshots99
cron-schedule: '@every 2m'
scheduler-enabled: false
storage:
storage-class-provisioner: kubernetes.io/aws-ebs
type: gp2
zones:
- us-east-1a
- us-east-1b
- us-east-1c
To deploy the operator simply deploy to your cluster:
$ kubectl create -f https://raw.githubusercontent.com/upmc-enterprises/elasticsearch-operator/master/example/controller.yaml$ kubectl create -f https://raw.githubusercontent.com/upmc-enterprises/elasticsearch-operator/master/example/example-es-cluster.jsonNOTE: Creating a custom cluster requires the creation of a ThirdPartyResource. This happens automatically after the controller is created.
To run the operator on minikube, this sample file is setup to do that. It sets lower Java memory contraints as well as uses the default storage class in Minikube which writes to hostPath.
$ kubectl create -f https://raw.githubusercontent.com/upmc-enterprises/elasticsearch-operator/master/example/example-es-cluster-minikube.jsonNOTE: Creating a custom cluster requires the creation of a ThirdPartyResource. This happens automatically after the controller is created.
kubectl apply doesn't work for TPR for the moment. See kubernetes/#29542. As a workaround, we use curl to resize the cluster.
First update the default example configuration, then send a PUT request to the Kubernetes API server. NOTE: The API is acesssed the API service in this example via kubectl proxy.
curl -H 'Content-Type: application/json' -X PUT --data @example/example-es-cluster.json http://127.0.0.1:8001/apis/enterprises.upmc.com/v1/namespaces/default/elasticsearchclusters/example-es-clusterElasticsearch can snapshot it's indexes for easy backup / recovery of the cluster. Currently there's an integration to Amazon S3 as the backup respository for snapshots. The upmcenterprises docker images include the S3 Plugin which enables this feature in AWS.
Snapshots can be scheduled via a Cron syntax by defining the cron schedule in your elastic cluster. See: https://godoc.org/github.com/robfig/cron
NOTE: Be sure to enable the scheduler as well by setting scheduler-enabled=true
To enable the snapshots create a bucket in S3, then apply the following IAM permissions to your EC2 instances replacing {!YOUR_BUCKET!} with the correct bucket name.
{
"Statement": [
{
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::{!YOUR_BUCKET!}"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::{!YOUR_BUCKET!}/*"
]
}
],
"Version": "2012-10-17"
}
If you are using an elasticsearch image that requires authentication for the snapshot url, you can specify basic auth credentials.
spec:
client-node-replicas: 3
data-node-replicas: 3
data-volume-size: 10Gi
java-options: -Xms256m -Xmx256m
master-node-replicas: 2
snapshot:
bucket-name: elasticsnapshots99
cron-schedule: '@every 2m'
scheduler-enabled: false
authentication:
password: betterpasswordneeded
username: basicauthadmin
storage:
storage-class-provisioner: kubernetes.io/aws-ebs
type: gp2
zones:
- us-east-1a
- us-east-1b
- us-east-1c
Once deployed and all pods are running, the cluster can be accessed internally via https://elasticsearch:9200/ or https://${ELASTICSEARCH_SERVICE_HOST}:9200/
To run the Operator locally:
$ kubectl proxy
$ go run cmd/operator/main.go --kubecfg-file=${HOME}/.kube/config
Built by UPMC Enterprises in Pittsburgh, PA. http://enterprises.upmc.com/