Conformance Testing

[n3wscott]

@vaikas and @n3wscott have been trying to convert the Broker/Trigger spec into the Reconciler Test style tests.

The core ideas around Reconciler Test is that test infra setup inside the target testing environment could be defined using YAML if you wish. So there are a bunch of helpers to convert YAML into resources, and some templating features have been added to allow for some things to not be hard coded. An example

apiVersion: v1
kind: Service
metadata:
  name: {{ .name }}
  namespace: {{ .namespace }}
spec:
  selector:
    {{ .selectorKey }}: {{ .selectorValue }}
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

The idea is you provide this plus some base configuration and you end up with something like:

	images := map[string]string{}
	cfg := map[string]interface{}{
		"name":          "foo",
		"namespace":     "bar",
		"selectorKey":   "baz",
		"selectorValue": "baf",
	}

	files, err := manifest.ExecuteLocalYAML(images, cfg)
	if err != nil {
		panic(err)
	}

	manifest.OutputYAML(os.Stdout, files)
	// Output:
	// apiVersion: v1
	// kind: Service
	// metadata:
	//   name: foo
	//   namespace: bar
	// spec:
	//   selector:
	//     baz: baf
	//   ports:
	//     - protocol: TCP
	//       port: 80
	//       targetPort: 8080

And then leveraging some helpers in eventing, the Install function:

func Install(name, selectorKey, selectorValue string) feature.StepFn {
	cfg := map[string]interface{}{
		"name":          name,
		"selectorKey":   selectorKey,
		"selectorValue": selectorValue,
	}

	return func(ctx context.Context, t feature.T) {
		if _, err := manifest.InstallLocalYaml(ctx, cfg); err != nil {
			t.Fatal(err)
		}
	}
}

And then when writing a feature:

f.Setup("install sink", svc.Install(sink, "some", "label"))

This all results in augmentation to the target test environment.

One oversight we had in writing the ReconcilerTest was for conformance testing, the plan was to copy the top level test entry points, and use that as a place to do configuration and customization for downstream integrators. This is not wanted for Conformance. So in the last couple weeks we have been looking at what it might take to solve this problem.

The new question are attempting to answer is the following: Can the conformance test be run from the eventing repo with no source code changes for any other "Unit Under Test"?

We are answering this by defining env vars that adjust how the test is run in very specific ways we directly support. At the moment we are focused on the broker/trigger spec, and have mostly finished the control plane tests, and are starting on the data plane tests. In the code, at the moment we can do this (always subject to change):

export BROKER_CLASS=MyCustomClass

This changes the Broker Class that the test will use when creating Broker resources.

BROKER_TEMPLATES"`

This feature will change a little, at the moment it replaces the broker template, but in the near future you will have the ability to replace the "invarrents" in the test template based on resource kinds. This will let you hardwire a configuration that you know is needed in your broker or special labels/annotations.

At the moment we are still trying to figure out a good UX for how test authors should use eventshub. At the moment it is very verbose to configure. It might be able to do anything we need, but getting it to do that is still more work than we would really like test authors to do. Finding the balance between too much magic and too much water chopping is always hard.

These updates are also being logged daily at https://hackmd.io/6Xk0wo8pT82Dt3ezCfKvVg?view

---

We are really close to what we think developers should be writing for conformance, the end result is defining a TestSet which holds all of the conformance tests.

func ControlPlaneConformance(brokerName string) *feature.FeatureSet {
	fs := &feature.FeatureSet{
		Name: "Knative Broker Specification - Control Plane",
		Features: []feature.Feature{
			*ControlPlaneBroker(brokerName),
			*ControlPlaneTrigger_GivenBroker(brokerName),
			*ControlPlaneTrigger_GivenBrokerTriggerReady(brokerName),
			*ControlPlaneTrigger_WithBrokerLifecycle(),
			*ControlPlaneTrigger_WithValidFilters(brokerName),
			*ControlPlaneTrigger_WithInvalidFilters(brokerName),
			*ControlPlaneDelivery(brokerName),
		},
	}
	// TODO: This is not a control plane test, or at best it is a blend with data plane.
	// Must("Events that pass the attributes filter MUST include context or extension attributes that match all key-value pairs exactly.", todo)

	return fs
}

Then each test looks something like:

func ControlPlaneBroker(brokerName string) *feature.Feature {
	f := feature.NewFeatureNamed("Broker")

	f.Setup("Set Broker Name", setBrokerName(brokerName))

	f.Stable("Conformance").
		Should("Broker objects SHOULD include a Ready condition in their status",
			brokerHasReadyInConditions).
		Should("The Broker SHOULD indicate Ready=True when its ingress is available to receive events.",
			readyBrokerHasIngressAvailable).
		Should("While a Broker is Ready, it SHOULD be a valid Addressable and its `status.address.url` field SHOULD indicate the address of its ingress.",
			readyBrokerIsAddressable).
		Should("The class of a Broker object SHOULD be immutable.",
			brokerClassIsImmutable)
	return f
}

These are broken down into a smaller test step that looks like:

func brokerClassIsImmutable(ctx context.Context, t feature.T) {
	broker := getBroker(ctx, t)

	if broker.Annotations == nil {
		broker.Annotations = map[string]string{}
	}
	// update annotations
	broker.Annotations[eventingv1.BrokerClassAnnotationKey] = "Rekt.brokerClassIsImmutable"

	if _, err := Client(ctx).Brokers.Update(ctx, broker, metav1.UpdateOptions{}); err != nil {
		// Success!
		t.Log("broker class is immutable")
	} else {
		t.Errorf("broker class is mutable")
	}
}

You can look at the entire test here: https://github.com/knative/eventing/blob/main/test/rekt/features/broker/control_plane.go

As we get closer to stamping this as ready for others to make more, we wanted to open the floor for questions or gotchyas we have not thought about. So... what is on your mind? This seem like an ok plan so far? what is needed in terms of docs for folks to come in and start helping this effort when we have this first example finished?

[slinkydeveloper]

This feature will change a little, at the moment it replaces the broker template, but in the near future you will have the ability to replace the "invarrents" in the test template based on resource kinds.

Can you be more specific on what you foresee as invariants? And how this replace should work? Who needs to take care of implementing the replace feature? Is this something that will be in the test framework itself?

At the moment we are still trying to figure out a good UX for how test authors should use eventshub.

Please open an issue for that on reconciler-test repo, I have some ideas :)

These are broken down into a smaller test step that looks like:

I would prefer these to look like assertions, eg:

require.UpdateFails(ctx, broker)

export BROKER_CLASS=MyCustomClass

Envs and cli params are very shortcoming in this use case. Configuring a test suite is not trivial, more and more specific knobs will come out that the conformance test user (e.g. the downstream project that wants to run them), and my fear is that going with cli args and envs will end up in a pile of undocumented arguments, and complex shell scripts to execute the test suite. IMO we need to invest on configuration files: the conformance tests user (e.g. a downstream project) writes a single yaml file where he/she can specify:

• components to test
• the features to run
• the expected conformance level
• some other fancy confs specific to some features

And so on.

I drafted this approach a while ago knative-extensions/reconciler-test#82 and I think it's worth retrying it.

So... what is on your mind? This seem like an ok plan so far? what is needed in terms of docs for folks to come in and start helping this effort when we have this first example finished?

I think before investing additional time on configuration and test authoring, we need to make sure the framework itself works and the "core concepts" are documented, well explained and behave as expected:

• Test execution model knative-extensions/reconciler-test#106
• Environment.Prerequisite semantics knative-extensions/reconciler-test#122
• The entire testing framework needs unit tests. knative-extensions/reconciler-test#36
• Can step fn panic? is the panic recovered and recorded as failure? knative-extensions/reconciler-test#32

We also need to look at reconciler-test from the "e2e tests" POV. In the initial discussions I recall it clearly come up that e2e tests and conformance tests, in our use cases, have similar requirements: we want to be able to reuse e2e tests downstream like for conformance tests. Hence the configuration stuff, like several other "components" about re-usability, should live in the framework itself and not in eventing, ofc having the right abstractions.
Examples of e2e tests developed with reconciler-test: https://github.com/knative-sandbox/eventing-kafka-broker/blob/380258a781c28af8be5a677e746a5213a4625b4c/test/e2e_new/ordered_test.go

[n3wscott]

the config is for eventing, not the framework. As this evolves we can get more advanced config, but at the moment it is a waste of effort to work on that.