Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix flaky TestFQDN #3097

Merged
merged 3 commits into from
Jul 20, 2023
Merged

Fix flaky TestFQDN #3097

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e6dda34
Pass namespace to FQDN ES query
ycombinator Jul 18, 2023
59495fb
Add documentation on namespace usage
ycombinator Jul 19, 2023
e303015
Add warning
ycombinator Jul 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion docs/test-framework-dev-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,25 @@ the `github.com/elastic/elastic-agent/pkg/testing/define` package for the test
framework's API and the `github.com/elastic/elastic-agent/pkg/testing/tools`
package for helper utilities.

### Test namespaces

Every test has access to its own unique namespace (a string value). This namespace can
be accessed from the `info.Namespace` field, where `info` is the struct value returned
from the `define.Require(...)` call made at the start of the test.

Namespaces should be used whenever test data is being written to or read from a persistent store that's
shared across all tests. Most commonly, this store will be the Elasticsearch cluster that Agent
components may index their data into. All tests share a single stack deployment and, therefore,
a single Elasticsearch cluster as well.

Some examples of where namespaces should be used:
* When creating a policy in Fleet. The Create Policy and Update Policy APIs takes a namespace parameter.
* When searching for documents in `logs-*` or `metrics-*` data streams. Every document in these
data streams has a `data_stream.namespace` field.

:warning: Not using namespaces when accessing data in a shared persistent store can cause tests to
be flaky.

## Troubleshooting Tips

### Error: GCE service token missing; run 'mage integration:auth'
Expand Down Expand Up @@ -99,4 +118,4 @@ If we need to use a different version between agent and stack we can specify the
using a separate env variable `AGENT_STACK_VERSION` like in this example (we used a
custom package version for the agent):

```AGENT_VERSION="8.10.0-testpkgversion.1-SNAPSHOT" AGENT_STACK_VERSION="8.10.0-SNAPSHOT" mage integration:test```
```AGENT_VERSION="8.10.0-testpkgversion.1-SNAPSHOT" AGENT_STACK_VERSION="8.10.0-SNAPSHOT" mage integration:test```
43 changes: 35 additions & 8 deletions testing/integration/fqdn_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
package integration

import (
"bytes"
"context"
"encoding/json"
"fmt"
Expand Down Expand Up @@ -98,8 +99,8 @@ func TestFQDN(t *testing.T) {
agent := verifyAgentName(t, shortName, info.KibanaClient)

t.Log("Verify that hostname in `logs-*` and `metrics-*` is short hostname")
verifyHostNameInIndices(t, "logs-*", shortName, info.ESClient)
verifyHostNameInIndices(t, "metrics-*", shortName, info.ESClient)
verifyHostNameInIndices(t, "logs-*", shortName, info.Namespace, info.ESClient)
verifyHostNameInIndices(t, "metrics-*", shortName, info.Namespace, info.ESClient)

t.Log("Update Agent policy to enable FQDN")
policy.AgentFeatures = []map[string]interface{}{
Expand Down Expand Up @@ -129,8 +130,8 @@ func TestFQDN(t *testing.T) {
verifyAgentName(t, fqdn, info.KibanaClient)

t.Log("Verify that hostname in `logs-*` and `metrics-*` is FQDN")
verifyHostNameInIndices(t, "logs-*", fqdn, info.ESClient)
verifyHostNameInIndices(t, "metrics-*", fqdn, info.ESClient)
verifyHostNameInIndices(t, "logs-*", fqdn, info.Namespace, info.ESClient)
verifyHostNameInIndices(t, "metrics-*", fqdn, info.Namespace, info.ESClient)

t.Log("Update Agent policy to disable FQDN")
policy.AgentFeatures = []map[string]interface{}{
Expand Down Expand Up @@ -185,7 +186,34 @@ func verifyAgentName(t *testing.T, hostname string, kibClient *kibana.Client) *k
return agent
}

func verifyHostNameInIndices(t *testing.T, indices, hostname string, esClient *elasticsearch.Client) {
func verifyHostNameInIndices(t *testing.T, indices, hostname, namespace string, esClient *elasticsearch.Client) {
queryRaw := map[string]interface{}{
"query": map[string]interface{}{
"bool": map[string]interface{}{
"must": []map[string]interface{}{
{
"term": map[string]interface{}{
"host.name": map[string]interface{}{
"value": hostname,
},
},
},
{
"term": map[string]interface{}{
"data_stream.namespace": map[string]interface{}{
"value": namespace,
},
},
},
},
},
},
}

var buf bytes.Buffer
err := json.NewEncoder(&buf).Encode(queryRaw)
require.NoError(t, err)

search := esClient.Search

require.Eventually(
Expand All @@ -196,6 +224,7 @@ func verifyHostNameInIndices(t *testing.T, indices, hostname string, esClient *e
search.WithSort("@timestamp:desc"),
search.WithFilterPath("hits.hits"),
search.WithSize(1),
search.WithBody(&buf),
)
require.NoError(t, err)
require.False(t, resp.IsError())
Expand All @@ -216,9 +245,7 @@ func verifyHostNameInIndices(t *testing.T, indices, hostname string, esClient *e
err = decoder.Decode(&body)
require.NoError(t, err)

require.Len(t, body.Hits.Hits, 1)
hit := body.Hits.Hits[0]
return hostname == hit.Source.Host.Name
return len(body.Hits.Hits) == 1
},
2*time.Minute,
5*time.Second,
Expand Down
Loading