Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SavedObjects] Allow migration path for single namespace to multiple #197097

Closed
shahzad31 opened this issue Oct 21, 2024 · 7 comments
Closed

[SavedObjects] Allow migration path for single namespace to multiple #197097

shahzad31 opened this issue Oct 21, 2024 · 7 comments
Labels
Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@shahzad31
Copy link
Contributor

shahzad31 commented Oct 21, 2024

In Synthetics we have synthetics monitors saved objects declared as single.

We have been receiving tons of requests from our users to allow monitors sharing across spaces.

Currently there is no path for migrating for single namespace monitors.

Ideally we would like to have the ability to at least share new/edited monitor saved objects to be of multiple spaces. Migration of existing saved objects is not required.

Workaround

Workaround that i can think of is to register a new saved objects and only add/edit monitors in that type but query both. And whenever write happens we delete from existing SO type and only write them in new if monitor is of single namespace.

@botelastic botelastic bot added the needs-team Issues missing a team label label Oct 21, 2024
@shahzad31 shahzad31 added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Oct 21, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Oct 21, 2024
@TinaHeiligers
Copy link
Contributor

TinaHeiligers commented Oct 21, 2024

@shahzad31 You are correct, migration from single namespace to multiple is not supported after 8.0.
Your workaround is the recommended approach:
introduce a new saved object with the desired namespace that will allow sharing and sunset the old saved object.

And whenever write happens we delete from existing and only write them in new if monitor is of single namespace.

I'm not sure I follow. A new shared saved object can be shared to other spaces at any time after creation. Even if it only exists in one space and isn't shared to any other spaces, it will still be possible to share later.
Could you clarify what "only write them in new if monitor is of single namespace" means?

@shahzad31
Copy link
Contributor Author

@TinaHeiligers i meant when editing, if monitor if of previous saved object type, we delete it from existing SO type and create it in new SO type.

@rudolf
Copy link
Contributor

rudolf commented Oct 22, 2024

Just as some historical background, it was possible to convert types from single to multinamespace in 8.0. But supporting this use case meant reindexing saved objects on upgrade since the saved objects need a new regenerated _id field. This reindexing step was the root cause of most upgrade failures since reindexing requires more shards and a fair amount of our users have unhealthy clusters where the cluster has exceeded the max open shards limit. So an RFC was accepted to stop supporting this behavior in 8.8 (internal link https://github.com/elastic/dev/issues/2189)

More recently with serverless and zero downtime migrations we can no longer perform a reindexing step as there's no feasible way to do this without downtime. So we have a hard blocker that would make single to multiple migrations impossible.

More recently we investigated supporting migration from agnostic to multiple, this is a much simpler case since we wouldn't need to regenerate _ids or reindex documents. But even in this case our conclusion was that the overall complexity was a lot lower by just doing this with custom code in the plugin (summary of discussion:
#188420 (comment))

Here is what the final implementation looks like: #189387

This is somewhat similar to your idea of "sunsetting" the old saved object type, but worth considering if a user opt-in migration could simplify the solution.

@shahzad31
Copy link
Contributor Author

@rudolf yeah basically our use case don't need reindexing AFAIK, we just need the ability to edit/create new saved object in multiple spaces, existing can remain as is, do you think, that can be supported?

@rudolf
Copy link
Contributor

rudolf commented Oct 28, 2024

there is quite a lot of different logic for single vs multinamespace saved object, and the decision on which code path to follow happens at the type level not an individual saved object. So it's not possible to have a mix of single and multinamespace saved objects of the same type.

One example of the different behaviour is that single namespace saved object types have _id fields like: coreteamspace:dashboard:1234 (space, type, id). Whereas multi namespace saved objects don't have the space in the id but only dashboard:1234. So when we do something like savedObjectsClient.get('dashboard', '1234') the actual request to Elasticsearch needs to be changed depending on whether 'dashboard' is single or multi namespace.

@shahzad31
Copy link
Contributor Author

Closing based on discussion, it seems like the best course of action is to register a new Saved object type.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Development

No branches or pull requests

4 participants