[Fleet] Settings Framework API and UI #179795

nchaulet · 2024-04-02T08:25:19Z

Summary

Follow up on #170539
Related to https://github.com/elastic/ingest-dev/issues/2471 (Phase 1)

Dynamically creating settings fields from configuration.
These settings are saved in the agent policy SO's advanced_settings field.
In the current pr the agent policy read/create/update works including the UI.
It still has to be extended to support a few more type of settings: e.g. dropdown values, settings consisting of multiple fields.

These settings are added to the full agent policy agents section:

Old description:

Add support for saved object mapping and api field name,

Example API calls and full policy generation

Open questions/Issues

Saved objects

*I think we will still have to do some work to add a new model version when adding a new saved object field, I do not see an easy way to programatically generate that. In a first time it probably could be a manual action to add those migration

API

Open api generation, I think as a first iteration it could be a manual operation to update openAPI spec, but we should be able to programatically generate that with a script in the future

apmmachine · 2024-04-02T08:25:35Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
/oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

nchaulet · 2024-04-02T08:50:31Z

x-pack/plugins/fleet/server/services/form_settings.ts

@@ -0,0 +1,81 @@
+/*


Created that service that return mappings, api schema validation, and full policy values for a given section (right now we only have AGENT_POLICY_ADVANCED_SETTINGS )

nchaulet · 2024-04-02T09:44:31Z

x-pack/plugins/fleet/common/settings/types.ts

+import type { SavedObjectsFieldMapping } from '@kbn/core-saved-objects-server';
+import type { z } from 'zod';
+
+export interface SettingsConfig {


I added the saved_object_field and api_field definition here

nchaulet · 2024-04-02T14:08:23Z

/ci

nchaulet · 2024-04-03T01:03:57Z

x-pack/plugins/fleet/common/settings/types.ts

+    name: string;
+    mapping: SavedObjectsFieldMapping;
+  };
+  api_field: {


used an object here so it easily extendable in the future,

nchaulet · 2024-04-03T06:57:46Z

/ci

…fix'

juliaElastic · 2024-04-03T13:15:46Z

x-pack/plugins/fleet/server/saved_objects/index.ts

@@ -156,6 +157,7 @@ export const getSavedObjectTypes = (): { [key: string]: SavedObjectsType } => ({
        is_protected: { type: 'boolean' },
        overrides: { type: 'flattened', index: false },
        keep_monitoring_alive: { type: 'boolean' },
+        ...getSettingsSavedObjectMappings('AGENT_POLICY_ADVANCED_SETTINGS'),


Saved objects
*I think we will still have to do some work to add a new model version when adding a new saved object field, I do not see an easy way to programatically generate that. In a first time it probably could be a manual action to add those migration

I see what you mean, without adding model versions, I am getting an error when trying to set the new field

{ "statusCode": 400, "error": "Bad Request", "message": "[1:349] mapping set to strict, dynamic introduction of [agent_limits_go_max_procs] within [ingest-agent-policies] is not allowed: strict_dynamic_mapping_exception\n\tRoot causes:\n\t\tstrict_dynamic_mapping_exception: [1:349] mapping set to strict, dynamic introduction of [agent_limits_go_max_procs] within [ingest-agent-policies] is not allowed" }

Added this modelVersions locally to get it working:

modelVersions: { '1': { changes: [ { type: 'mappings_addition', addedMappings: { ...getSettingsSavedObjectMappings('AGENT_POLICY_ADVANCED_SETTINGS'), }, }, ], }, },

I am wondering if we could add a generic mapping here, since these fields are usually not indexed anyway, something like:

advanced_settings: { type: 'flattened', index: false }

One caveat with this would be that we can't set a field type mapping on specific fields e.g. go_max_procs to be an integer. But I think it doesn't matter since the fields are not indexed.

I made this work locally, to save these settings under a generic advanced_settings mapping in the SO. The full agent policy hasn't changed.

PUT kbn:/api/fleet/agent_policies/b180cbb0-b276-4f96-97a7-e2acb97c6a36 { "name": "Agent policy 2", "namespace": "default", "advanced_settings": { "agent_limits_go_max_procs": 6 } }

This simplifies the settings config that we don't need saved_object_field.

x-pack/plugins/fleet/common/settings/agent_policy_settings.ts

juliaElastic · 2024-04-04T07:19:45Z

/ci

juliaElastic · 2024-04-04T09:50:01Z

/ci

juliaElastic · 2024-04-04T11:27:40Z

/ci

juliaElastic · 2024-04-04T11:29:40Z

x-pack/plugins/fleet/public/applications/fleet/components/form_settings/index.tsx

+      }
+    >
+      <EuiFormRow fullWidth key={fieldKey} error={error} isInvalid={!!error}>
+        <EuiFieldText


there is a lot of duplication between number and text type, I wonder if there could be a common component as a wrapper

refactored this component to use a common SettingsFieldWrapper

…te --fix'

juliaElastic · 2024-04-04T12:13:14Z

/ci

juliaElastic · 2024-04-04T12:14:39Z

/ci

… src/core/server/integration_tests/ci_checks'

elasticmachine · 2024-04-08T08:52:04Z

Pinging @elastic/fleet (Team:Fleet)

juliaElastic · 2024-04-08T08:52:29Z

/ci

juliaElastic · 2024-04-08T11:54:45Z

x-pack/plugins/fleet/common/settings/types.ts

+    name: string;
+  };
+  // form error state is passed up to the form
+  updateAdvancedSettingsHasErrors?: (hasErrors: boolean) => void;


I don't really like this to add a function to the SettingsConfig, haven't found a better way to pass down this function to the dynamic components.
It is needed to make the submit button disabled if one of the advanced fields are invalid.

EDIT: I found a better solution to put this function in AgentPolicyFormContext

kibana-ci · 2024-04-09T12:14:21Z

💚 Build Succeeded

Buildkite Build
Commit: ddb0925

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	971	984	+13

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1116	1117	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1.2MB	1.3MB	+69.3KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	158.6KB	158.7KB	+26.0B

Unknown metric groups

API count

id	before	after	diff
`fleet`	1236	1237	+1

History

💛 Build #202046 was flaky ab47145
💛 Build #201788 was flaky 779acf5
💔 Build #201714 failed bc62731
💛 Build #201427 was flaky ba7a56f
💚 Build #201363 succeeded a52d6bb
💔 Build #201339 failed 01c9d53

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kpollich

LGTM 🚀 - thanks for picking up where Nicolas and I left off 🙂

juliaElastic · 2024-04-09T12:57:33Z

x-pack/plugins/fleet/common/openapi/components/schemas/agent_policy.yaml

@@ -69,6 +69,10 @@ properties:
    type: object
    description: Override settings that are defined in the agent policy. Input settings cannot be overridden. The override option should be used only in unusual circumstances and not as a routine procedure.
    nullable: true
+  advanced_settings:


We decided to go with a generic advanced_settings in openapi for now. We will look at generating the openapi spec from the settings config later.

kilfoyle

LGTM for the descriptions! 👍

amitkanfer · 2024-04-09T22:13:04Z

🚀

nchaulet commented Apr 2, 2024

View reviewed changes

nchaulet force-pushed the feature-settings-configurable branch from 78a4355 to 64d3598 Compare April 2, 2024 09:40

nchaulet commented Apr 2, 2024

View reviewed changes

[Fleet] Settings Framework API and SO

459ed22

nchaulet force-pushed the feature-settings-configurable branch from 64d3598 to 459ed22 Compare April 2, 2024 09:47

nchaulet added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team labels Apr 3, 2024

nchaulet commented Apr 3, 2024

View reviewed changes

Add unit tests

b9b1d89

kibanamachine and others added 2 commits April 3, 2024 07:05

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

e9da8bf

…fix'

Merge branch 'main' into feature-settings-configurable

137b40e

juliaElastic reviewed Apr 3, 2024

View reviewed changes

x-pack/plugins/fleet/common/settings/agent_policy_settings.ts Outdated Show resolved Hide resolved

fix lint

ba36f2a

juliaElastic added 2 commits April 4, 2024 13:16

storing dynamic settings under advanced_settings in SO

c319d0c

remove saved_object_field

cc9e820

juliaElastic reviewed Apr 4, 2024

View reviewed changes

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

c87e0d6

…te --fix'

remove unused export

4e639a2

kibanamachine and others added 3 commits April 4, 2024 12:44

[CI] Auto-commit changed files from 'node scripts/jest_integration -u…

07130c4

… src/core/server/integration_tests/ci_checks'

added openapi spec for advanced_settings

6761aa1

move i18n to form settings component

5223904

juliaElastic and others added 2 commits April 8, 2024 12:54

Merge branch 'main' into feature-settings-configurable

b3a83a7

added regex validation on duration fields

5020c56

juliaElastic reviewed Apr 8, 2024

View reviewed changes

juliaElastic added 2 commits April 8, 2024 14:00

changed regex to allow empty value

c47b966

add unit test on form_settings

6879094

juliaElastic force-pushed the feature-settings-configurable branch from 547f9ed to 6879094 Compare April 8, 2024 12:17

juliaElastic added 2 commits April 8, 2024 14:24

move updateAdvancedSettingsHasErrors to AgentPolicyFormContext

7a78b57

add more unit test

779acf5

juliaElastic force-pushed the feature-settings-configurable branch from 6935271 to 779acf5 Compare April 8, 2024 12:52

pgayvallet approved these changes Apr 9, 2024

View reviewed changes

juliaElastic and others added 2 commits April 9, 2024 11:08

Merge branch 'main' into feature-settings-configurable

ab47145

add unit test on full_agent_policy

ddb0925

juliaElastic changed the title ~~[Fleet] Settings Framework API and SO~~ [Fleet] Settings Framework API and UI Apr 9, 2024

juliaElastic requested a review from a team April 9, 2024 11:00

kpollich mentioned this pull request Apr 9, 2024

[Fleet] Add proof-of-concept for Fleet settings framework #170539

Closed

juliaElastic added release_note:enhancement and removed release_note:skip Skip the PR/issue when compiling release notes labels Apr 9, 2024

kpollich approved these changes Apr 9, 2024

View reviewed changes

juliaElastic reviewed Apr 9, 2024

View reviewed changes

juliaElastic requested a review from kilfoyle April 9, 2024 12:58

kilfoyle approved these changes Apr 9, 2024

View reviewed changes

SiddharthMantri approved these changes Apr 9, 2024

View reviewed changes

juliaElastic merged commit c88b3bd into elastic:main Apr 9, 2024
22 checks passed

kibanamachine added v8.14.0 backport:skip This commit does not require backporting labels Apr 9, 2024

jlind23 mentioned this pull request Apr 9, 2024

[Fleet] The agent upgrade download timeout should be configurable. elastic/elastic-agent#4580

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Settings Framework API and UI #179795

[Fleet] Settings Framework API and UI #179795

nchaulet commented Apr 2, 2024 •

edited by juliaElastic

Loading

apmmachine commented Apr 2, 2024

nchaulet Apr 2, 2024

nchaulet Apr 2, 2024

nchaulet commented Apr 2, 2024

nchaulet Apr 3, 2024 •

edited

Loading

nchaulet commented Apr 3, 2024

juliaElastic Apr 3, 2024 •

edited

Loading

juliaElastic Apr 4, 2024 •

edited

Loading

juliaElastic commented Apr 4, 2024

juliaElastic commented Apr 4, 2024

juliaElastic commented Apr 4, 2024

juliaElastic Apr 4, 2024

juliaElastic Apr 5, 2024

juliaElastic commented Apr 4, 2024

juliaElastic commented Apr 4, 2024

elasticmachine commented Apr 8, 2024

juliaElastic commented Apr 8, 2024

juliaElastic Apr 8, 2024 •

edited

Loading

kibana-ci commented Apr 9, 2024

API count

kpollich left a comment

juliaElastic Apr 9, 2024

kilfoyle left a comment

amitkanfer commented Apr 9, 2024

[Fleet] Settings Framework API and UI #179795

[Fleet] Settings Framework API and UI #179795

Conversation

nchaulet commented Apr 2, 2024 • edited by juliaElastic Loading

Summary

Old description:

Example API calls and full policy generation

Open questions/Issues

Saved objects

API

apmmachine commented Apr 2, 2024

🤖 GitHub comments

nchaulet Apr 2, 2024

Choose a reason for hiding this comment

nchaulet Apr 2, 2024

Choose a reason for hiding this comment

nchaulet commented Apr 2, 2024

nchaulet Apr 3, 2024 • edited Loading

Choose a reason for hiding this comment

nchaulet commented Apr 3, 2024

juliaElastic Apr 3, 2024 • edited Loading

Choose a reason for hiding this comment

juliaElastic Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

juliaElastic commented Apr 4, 2024

juliaElastic commented Apr 4, 2024

juliaElastic commented Apr 4, 2024

juliaElastic Apr 4, 2024

Choose a reason for hiding this comment

juliaElastic Apr 5, 2024

Choose a reason for hiding this comment

juliaElastic commented Apr 4, 2024

juliaElastic commented Apr 4, 2024

elasticmachine commented Apr 8, 2024

juliaElastic commented Apr 8, 2024

juliaElastic Apr 8, 2024 • edited Loading

Choose a reason for hiding this comment

kibana-ci commented Apr 9, 2024

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

History

kpollich left a comment

Choose a reason for hiding this comment

juliaElastic Apr 9, 2024

Choose a reason for hiding this comment

kilfoyle left a comment

Choose a reason for hiding this comment

amitkanfer commented Apr 9, 2024

nchaulet commented Apr 2, 2024 •

edited by juliaElastic

Loading

nchaulet Apr 3, 2024 •

edited

Loading

juliaElastic Apr 3, 2024 •

edited

Loading

juliaElastic Apr 4, 2024 •

edited

Loading

juliaElastic Apr 8, 2024 •

edited

Loading