Add basic serverless support for elasticsearch, beats index management #36587

fearful-symmetry · 2023-09-13T20:39:58Z

Proposed commit message

The tests that are a part of elastic/elastic-agent#3258 will pass with this PR.

There's a few parts to this PR:

Adds an IsServerless() method to the ES client
Changes the Ping() method in the ES client
Adds some checks to the index management setup process to bypass ILM setup under serverless
Adds some doc comments where needed
Changes a handful of ES interfaces to support the IsServerless() method

If there's anything weird in this PR, it's probably because I spent a day just sorting through ancient ILM code.

So, I'm not sure if we want to merge this without DSL support, or add DSL support in a follow-up PR, or merge elastic/elastic-agent#3258 before or after this PR. Currently open to suggestions.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

mergify · 2023-09-13T20:40:38Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @fearful-symmetry? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2023-09-13T20:46:53Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-09-20T16:36:30.074+0000
Duration: 101 min 21 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28277
Skipped	2013
Total	30290

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2023-09-13T21:32:47Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b serverless-aware upstream/serverless-aware
git merge upstream/main
git push upstream serverless-aware

faec

Looks good as far as the code here (thank you for adding more docs!)... I don't have the context to say much about inter-repo order of operations for this feature but it looks to me like merging this one wouldn't break anything, so it's probably a reasonable incremental step?

faec · 2023-09-14T18:21:40Z

libbeat/esleg/eslegclient/connection.go

+	if versionData.Version.BuildFlavor == "serverless" {
+		conn.isServerless = true
+	} else if versionData.Version.BuildFlavor == "default" {
+		conn.isServerless = false


Can this ever actually change, or is this just defensive programming? If the latter can you add a comment saying something like "this should already be false but let's be extra careful"?

faec · 2023-09-14T18:23:29Z

libbeat/idxmgmt/std.go

+		}
+		err = selCfg.SetChild("indices", -1, sub)
+		if err != nil {
+			return nil, fmt.Errorf("error setting child 'indicies': %w", err)


typo (indicies)

cmacknz

A few suggestions to improve error text to use official names, otherwise looks good.

libbeat/cmd/instance/beat.go

libbeat/dashboards/kibana_loader.go

Co-authored-by: Craig MacKenzie <[email protected]>

elastic#36587) * make serverless integration tests run * update deps * linter, error handling * still fixing error handling * fixing old formatting verbs * still finding format verbs * add docs, fix typos * Update libbeat/cmd/instance/beat.go Co-authored-by: Craig MacKenzie <[email protected]> * Update libbeat/dashboards/kibana_loader.go Co-authored-by: Craig MacKenzie <[email protected]> * Update libbeat/dashboards/kibana_loader.go Co-authored-by: Craig MacKenzie <[email protected]> --------- Co-authored-by: Craig MacKenzie <[email protected]>

make serverless integration tests run

14b8dcd

fearful-symmetry added enhancement Team:Elastic-Agent Label for the Agent team labels Sep 13, 2023

fearful-symmetry requested a review from a team as a code owner September 13, 2023 20:39

fearful-symmetry self-assigned this Sep 13, 2023

fearful-symmetry requested review from ycombinator and faec September 13, 2023 20:39

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 13, 2023

update deps

9f86184

fearful-symmetry requested a review from a team as a code owner September 13, 2023 21:31

fearful-symmetry added 5 commits September 13, 2023 14:35

Merge remote-tracking branch 'upstream/main' into serverless-aware

ae975c5

linter, error handling

dc9d1a3

still fixing error handling

be2eb52

fixing old formatting verbs

e14d7fa

still finding format verbs

afe73e8

faec approved these changes Sep 14, 2023

View reviewed changes

fearful-symmetry and others added 3 commits September 14, 2023 11:49

add docs, fix typos

bbb8072

Merge branch 'main' into serverless-aware

b44c96a

Merge branch 'main' into serverless-aware

3d55c29

cmacknz approved these changes Sep 19, 2023

View reviewed changes

libbeat/cmd/instance/beat.go Outdated Show resolved Hide resolved

libbeat/dashboards/kibana_loader.go Outdated Show resolved Hide resolved

libbeat/dashboards/kibana_loader.go Outdated Show resolved Hide resolved

fearful-symmetry and others added 3 commits September 20, 2023 09:33

Update libbeat/cmd/instance/beat.go

347ef3f

Co-authored-by: Craig MacKenzie <[email protected]>

Update libbeat/dashboards/kibana_loader.go

0509d41

Co-authored-by: Craig MacKenzie <[email protected]>

Update libbeat/dashboards/kibana_loader.go

88d5ce3

Co-authored-by: Craig MacKenzie <[email protected]>

fearful-symmetry merged commit bd088b8 into elastic:main Sep 20, 2023
114 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add basic serverless support for elasticsearch, beats index management #36587

Add basic serverless support for elasticsearch, beats index management #36587

fearful-symmetry commented Sep 13, 2023 •

edited

Loading

mergify bot commented Sep 13, 2023

elasticmachine commented Sep 13, 2023 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

mergify bot commented Sep 13, 2023

faec left a comment

faec Sep 14, 2023

faec Sep 14, 2023

cmacknz left a comment

Add basic serverless support for elasticsearch, beats index management #36587

Add basic serverless support for elasticsearch, beats index management #36587

Conversation

fearful-symmetry commented Sep 13, 2023 • edited Loading

Proposed commit message

Checklist

mergify bot commented Sep 13, 2023

elasticmachine commented Sep 13, 2023 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Sep 13, 2023

faec left a comment

Choose a reason for hiding this comment

faec Sep 14, 2023

Choose a reason for hiding this comment

faec Sep 14, 2023

Choose a reason for hiding this comment

cmacknz left a comment

Choose a reason for hiding this comment

fearful-symmetry commented Sep 13, 2023 •

edited

Loading

elasticmachine commented Sep 13, 2023 •

edited by jenkins-beats-ci bot

Loading