Skip to content

Commit

Permalink
Adding feature/tolerations and node selector to PodOptions (#107)
Browse files Browse the repository at this point in the history
  • Loading branch information
@sepulworld
sepulworld authored Apr 8, 2020
1 parent b0cf11e commit 6bd5727
Show file tree
Hide file tree
Showing 14 changed files with 357 additions and 21 deletions.
6 changes: 3 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -375,16 +375,16 @@ Two Docker images are published to [DockerHub](https://hub.docker.com/r/bloomber

#### Building

Building and releasing a test operator image with a custom namespace.
Building and releasing a test operator image with a custom Docker namespace.

```bash
$ NAMESPACE=your-namespace make docker-build docker-push
$ NAMESPACE=your-namespace/ make docker-build docker-push
```

You can test the vendor docker container by running

```bash
$ NAMESPACE=your-namespace make docker-vendor-build docker-vendor-push
$ NAMESPACE=your-namespace/ make docker-vendor-build docker-vendor-push
```

### Docker for Mac Local Development Setup
Expand Down
11 changes: 10 additions & 1 deletion api/v1beta1/common_types.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@ limitations under the License.
package v1beta1

import (
corev1 "k8s.io/api/core/v1"
"strings"

corev1 "k8s.io/api/core/v1"
)

// StatefulSetOptions defines custom options for StatefulSets
Expand Down Expand Up @@ -74,6 +75,14 @@ type PodOptions struct {
// Labels to be added for pods.
// +optional
Labels map[string]string `json:"labels,omitempty"`

// Tolerations to be added for the StatefulSet.
// +optional
Tolerations []corev1.Toleration `json:"tolerations,omitempty"`

// Node Selector to be added for the StatefulSet.
// +optional
NodeSelector map[string]string `json:"nodeSelector,omitempty"`
}

// ServiceOptions defines custom options for services
Expand Down
14 changes: 14 additions & 0 deletions api/v1beta1/zz_generated.deepcopy.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

47 changes: 47 additions & 0 deletions config/crd/bases/solr.bloomberg.com_solrclouds.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2069,6 +2069,11 @@ spec:
type: string
description: Labels to be added for pods.
type: object
nodeSelector:
additionalProperties:
type: string
description: Node Selector to be added for the StatefulSet.
type: object
podSecurityContext:
description: PodSecurityContext is the security context for
the pod.
Expand Down Expand Up @@ -2217,6 +2222,48 @@ spec:
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
type: object
type: object
tolerations:
description: Tolerations to be added for the StatefulSet.
items:
description: The pod this Toleration is attached to tolerates
any taint that matches the triple <key,value,effect> using
the matching operator <operator>.
properties:
effect:
description: Effect indicates the taint effect to match.
Empty means match all taint effects. When specified,
allowed values are NoSchedule, PreferNoSchedule and
NoExecute.
type: string
key:
description: Key is the taint key that the toleration
applies to. Empty means match all taint keys. If the
key is empty, operator must be Exists; this combination
means to match all values and all keys.
type: string
operator:
description: Operator represents a key's relationship
to the value. Valid operators are Exists and Equal.
Defaults to Equal. Exists is equivalent to wildcard
for value, so that a pod can tolerate all taints of
a particular category.
type: string
tolerationSeconds:
description: TolerationSeconds represents the period of
time the toleration (which must be of effect NoExecute,
otherwise this field is ignored) tolerates the taint.
By default, it is not set, which means tolerate the
taint forever (do not evict). Zero and negative values
will be treated as 0 (evict immediately) by the system.
format: int64
type: integer
value:
description: Value is the taint value the toleration matches
to. If the operator is Exists, the value should be empty,
otherwise just a regular string.
type: string
type: object
type: array
volumes:
description: Additional non-data volumes to load into the default
container.
Expand Down
47 changes: 47 additions & 0 deletions config/crd/bases/solr.bloomberg.com_solrprometheusexporters.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -833,6 +833,11 @@ spec:
type: string
description: Labels to be added for pods.
type: object
nodeSelector:
additionalProperties:
type: string
description: Node Selector to be added for the StatefulSet.
type: object
podSecurityContext:
description: PodSecurityContext is the security context for
the pod.
Expand Down Expand Up @@ -981,6 +986,48 @@ spec:
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/'
type: object
type: object
tolerations:
description: Tolerations to be added for the StatefulSet.
items:
description: The pod this Toleration is attached to tolerates
any taint that matches the triple <key,value,effect> using
the matching operator <operator>.
properties:
effect:
description: Effect indicates the taint effect to match.
Empty means match all taint effects. When specified,
allowed values are NoSchedule, PreferNoSchedule and
NoExecute.
type: string
key:
description: Key is the taint key that the toleration
applies to. Empty means match all taint keys. If the
key is empty, operator must be Exists; this combination
means to match all values and all keys.
type: string
operator:
description: Operator represents a key's relationship
to the value. Valid operators are Exists and Equal.
Defaults to Equal. Exists is equivalent to wildcard
for value, so that a pod can tolerate all taints of
a particular category.
type: string
tolerationSeconds:
description: TolerationSeconds represents the period of
time the toleration (which must be of effect NoExecute,
otherwise this field is ignored) tolerates the taint.
By default, it is not set, which means tolerate the
taint forever (do not evict). Zero and negative values
will be treated as 0 (evict immediately) by the system.
format: int64
type: integer
value:
description: Value is the taint value the toleration matches
to. If the operator is Exists, the value should be empty,
otherwise just a regular string.
type: string
type: object
type: array
volumes:
description: Additional non-data volumes to load into the default
container.
Expand Down
27 changes: 26 additions & 1 deletion controllers/controller_utils_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ limitations under the License.
package controllers

import (
"reflect"
"testing"

solr "github.com/bloomberg/solr-operator/api/v1beta1"
"github.com/onsi/gomega"
"github.com/stretchr/testify/assert"
Expand All @@ -30,7 +33,6 @@ import (
"k8s.io/apimachinery/pkg/types"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
"testing"
)

func expectStatefulSet(t *testing.T, g *gomega.GomegaWithT, requests chan reconcile.Request, expectedRequest reconcile.Request, statefulSetKey types.NamespacedName) *appsv1.StatefulSet {
Expand Down Expand Up @@ -173,6 +175,10 @@ func testPodEnvVariables(t *testing.T, expectedEnvVars map[string]string, foundE
assert.Equal(t, len(expectedEnvVars), matchCount, "Not all expected env variables found in podSpec")
}

func testPodTolerations(t *testing.T, expectedTolerations []corev1.Toleration, foundTolerations []corev1.Toleration) {
assert.True(t, reflect.DeepEqual(expectedTolerations, foundTolerations), "Expected tolerations and found tolerations don't match")
}

func testMapsEqual(t *testing.T, mapName string, expected map[string]string, found map[string]string) {
assert.Equal(t, expected, found, "Expected and found %s are not the same", mapName)
}
Expand Down Expand Up @@ -295,6 +301,25 @@ var (
"testS3": "valueS3",
"testS4": "valueS4",
}
testNodeSelectors = map[string]string{
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"solrclouds": "true",
}
testTolerations = []corev1.Toleration{
{
Effect: "NoSchedule",
Key: "node-restriction.kubernetes.io/dedicated",
Value: "solrclouds",
Operator: "Exists",
},
}
testTolerationsPromExporter = []corev1.Toleration{
{
Effect: "NoSchedule",
Operator: "Exists",
},
}
extraVars = []corev1.EnvVar{
{
Name: "VAR_1",
Expand Down
11 changes: 8 additions & 3 deletions controllers/solrcloud_controller_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,10 @@ limitations under the License.
package controllers

import (
"testing"

"github.com/bloomberg/solr-operator/controllers/util"
"github.com/stretchr/testify/assert"
"testing"

solr "github.com/bloomberg/solr-operator/api/v1beta1"
"github.com/onsi/gomega"
Expand Down Expand Up @@ -171,8 +172,10 @@ func TestCloudReconcileWithIngress(t *testing.T) {
SolrGCTune: "gc Options",
CustomSolrKubeOptions: solr.CustomSolrKubeOptions{
PodOptions: &solr.PodOptions{
Annotations: testPodAnnotations,
Labels: testPodLabels,
Annotations: testPodAnnotations,
Labels: testPodLabels,
Tolerations: testTolerations,
NodeSelector: testNodeSelectors,
},
StatefulSetOptions: &solr.StatefulSetOptions{
Annotations: testSSAnnotations,
Expand Down Expand Up @@ -257,6 +260,8 @@ func TestCloudReconcileWithIngress(t *testing.T) {
testMapsEqual(t, "statefulSet annotations", util.MergeLabelsOrAnnotations(expectedStatefulSetAnnotations, testSSAnnotations), statefulSet.Annotations)
testMapsEqual(t, "pod labels", util.MergeLabelsOrAnnotations(expectedStatefulSetLabels, testPodLabels), statefulSet.Spec.Template.ObjectMeta.Labels)
testMapsEqual(t, "pod annotations", testPodAnnotations, statefulSet.Spec.Template.Annotations)
testMapsEqual(t, "pod node selectors", testNodeSelectors, statefulSet.Spec.Template.Spec.NodeSelector)
testPodTolerations(t, testTolerations, statefulSet.Spec.Template.Spec.Tolerations)

// Check the client Service
service := expectService(t, g, requests, expectedCloudRequest, cloudCsKey, statefulSet.Spec.Selector.MatchLabels)
Expand Down
17 changes: 13 additions & 4 deletions controllers/solrprometheusexporter_controller_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ limitations under the License.
package controllers

import (
"testing"

solr "github.com/bloomberg/solr-operator/api/v1beta1"
"github.com/bloomberg/solr-operator/controllers/util"
"github.com/onsi/gomega"
Expand All @@ -28,7 +30,6 @@ import (
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/manager"
"sigs.k8s.io/controller-runtime/pkg/reconcile"
"testing"
)

var _ reconcile.Reconciler = &SolrPrometheusExporterReconciler{}
Expand Down Expand Up @@ -121,9 +122,11 @@ func TestMetricsReconcileWithExporterConfig(t *testing.T) {
Config: testExporterConfig,
CustomKubeOptions: solr.CustomExporterKubeOptions{
PodOptions: &solr.PodOptions{
Annotations: testPodAnnotations,
Labels: testPodLabels,
Volumes: extraVolumes,
Annotations: testPodAnnotations,
Labels: testPodLabels,
Volumes: extraVolumes,
Tolerations: testTolerationsPromExporter,
NodeSelector: testNodeSelectors,
},
DeploymentOptions: &solr.DeploymentOptions{
Annotations: testDeploymentAnnotations,
Expand Down Expand Up @@ -161,6 +164,8 @@ func TestMetricsReconcileWithExporterConfig(t *testing.T) {
mgrStopped.Wait()
}()

cleanupTest(g, instance.Namespace)

// Create the SolrPrometheusExporter object and expect the Reconcile and Deployment to be created
err = testClient.Create(context.TODO(), instance)
// The instance object may not be a valid object because it might be missing some required fields.
Expand All @@ -184,6 +189,10 @@ func TestMetricsReconcileWithExporterConfig(t *testing.T) {
testMapsEqual(t, "pod labels", util.MergeLabelsOrAnnotations(expectedDeploymentLabels, testPodLabels), deployment.Spec.Template.ObjectMeta.Labels)
testMapsEqual(t, "pod annotations", testPodAnnotations, deployment.Spec.Template.ObjectMeta.Annotations)

// Test tolerations and node selectors
testMapsEqual(t, "pod node selectors", testNodeSelectors, deployment.Spec.Template.Spec.NodeSelector)
testPodTolerations(t, testTolerationsPromExporter, deployment.Spec.Template.Spec.Tolerations)

// Other Pod Options
extraVolumes[0].DefaultContainerMount.Name = extraVolumes[0].Name
assert.Equal(t, len(extraVolumes)+1, len(deployment.Spec.Template.Spec.Containers[0].VolumeMounts), "Container has wrong number of volumeMounts")
Expand Down
11 changes: 9 additions & 2 deletions controllers/util/prometheus_exporter_util.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -217,12 +217,20 @@ func GenerateSolrPrometheusExporterDeployment(solrPrometheusExporter *solr.SolrP
if customPodOptions.PodSecurityContext != nil {
deployment.Spec.Template.Spec.SecurityContext = customPodOptions.PodSecurityContext
}

if customPodOptions.Tolerations != nil {
deployment.Spec.Template.Spec.Tolerations = customPodOptions.Tolerations
}

if customPodOptions.NodeSelector != nil {
deployment.Spec.Template.Spec.NodeSelector = customPodOptions.NodeSelector
}
}

return deployment
}

// GenerateConfigMap returns a new corev1.ConfigMap pointer generated for the Solr Prometheus Exporter instance solr-prometheus-exporter.xml
// GenerateMetricsConfigMap returns a new corev1.ConfigMap pointer generated for the Solr Prometheus Exporter instance solr-prometheus-exporter.xml
// solrPrometheusExporter: SolrPrometheusExporter instance
func GenerateMetricsConfigMap(solrPrometheusExporter *solr.SolrPrometheusExporter) *corev1.ConfigMap {
labels := solrPrometheusExporter.SharedLabelsWith(solrPrometheusExporter.GetLabels())
Expand Down Expand Up @@ -293,7 +301,6 @@ func GenerateSolrMetricsService(solrPrometheusExporter *solr.SolrPrometheusExpor

// CreateMetricsIngressRule returns a new Ingress Rule generated for the solr metrics endpoint
// This is not currently used, as an ingress is not created for the metrics endpoint.

// solrCloud: SolrCloud instance
// nodeName: string Name of the node
// ingressBaseDomain: string base domain for the ingress controller
Expand Down
20 changes: 20 additions & 0 deletions controllers/util/solr_util.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -348,6 +348,14 @@ func GenerateStatefulSet(solrCloud *solr.SolrCloud, solrCloudStatus *solr.SolrCl
if customPodOptions.PodSecurityContext != nil {
stateful.Spec.Template.Spec.SecurityContext = customPodOptions.PodSecurityContext
}

if customPodOptions.Tolerations != nil {
stateful.Spec.Template.Spec.Tolerations = customPodOptions.Tolerations
}

if customPodOptions.NodeSelector != nil {
stateful.Spec.Template.Spec.NodeSelector = customPodOptions.NodeSelector
}
}

return stateful
Expand Down Expand Up @@ -445,6 +453,18 @@ func CopyStatefulSetFields(from, to *appsv1.StatefulSet) bool {
to.Spec.Template.Spec.SecurityContext = from.Spec.Template.Spec.SecurityContext
}

if !DeepEqualWithNils(to.Spec.Template.Spec.NodeSelector, from.Spec.Template.Spec.NodeSelector) {
requireUpdate = true
log.Info("Update required because:", "Spec.Template.Spec.NodeSelector changed from", to.Spec.Template.Spec.NodeSelector, "To:", from.Spec.Template.Spec.NodeSelector)
to.Spec.Template.Spec.NodeSelector = from.Spec.Template.Spec.NodeSelector
}

if !DeepEqualWithNils(to.Spec.Template.Spec.Tolerations, from.Spec.Template.Spec.Tolerations) {
requireUpdate = true
log.Info("Update required because:", "Spec.Template.Spec.Tolerations changed from", to.Spec.Template.Spec.Tolerations, "To:", from.Spec.Template.Spec.Tolerations)
to.Spec.Template.Spec.Tolerations = from.Spec.Template.Spec.Tolerations
}

return requireUpdate
}

Expand Down
Loading

0 comments on commit 6bd5727

Please sign in to comment.