Add pipeline to clean docs during data stream reindex #121617
Conversation
parkertimmins
added
:Data Management/Data streams
.../plugin/migrate/src/main/java/org/elasticsearch/xpack/migrate/ReindexDataStreamPipeline.java
Outdated
Show resolved
Hide resolved
.../plugin/migrate/src/main/java/org/elasticsearch/xpack/migrate/ReindexDataStreamPipeline.java
Outdated
Show resolved
Hide resolved
elasticsearchmachine
added
the
Team:Data Management
|
Pinging @elastic/es-data-management (Team:Data Management) |
|
Hi @parkertimmins, I've created a changelog YAML for you. |
| final BytesReference pipeline = BytesReference.bytes(currentPipelineDefinition()); | ||
| client.execute( | ||
| PutPipelineTransportAction.TYPE, | ||
| new PutPipelineRequest( |
There was a problem hiding this comment.
We probably ought to set the parent task id here so that it's cancellable (although I'm not 100% sure it's worth it).
| { | ||
| builder.startObject(); | ||
| { | ||
| builder.startObject("set"); |
There was a problem hiding this comment.
Is this going to set every @timestamp to 0? Shouldn't this be a script processor that checks if it exists first?
There was a problem hiding this comment.
Oh never mind! This is what override is for.
There was a problem hiding this comment.
It would be interesting to compare which performs better: a set with override: false or a set with an if condition checking for a null @timestamp
There was a problem hiding this comment.
It's hard to imagine that a script would be faster, but it would be interesting. We could always change that later though.
There was a problem hiding this comment.
And if the script is faster, that would probably shame @joegallo into action, making the set processor faster.
|
Looks pretty good, but a couple of things remain:
|
.../plugin/migrate/src/main/java/org/elasticsearch/xpack/migrate/ReindexDataStreamPipeline.java
Outdated
Show resolved
Hide resolved
.../plugin/migrate/src/main/java/org/elasticsearch/xpack/migrate/ReindexDataStreamPipeline.java
Outdated
Show resolved
Hide resolved
| { | ||
| builder.startObject(); | ||
| { | ||
| builder.startObject("set"); |
There was a problem hiding this comment.
It would be interesting to compare which performs better: a set with override: false or a set with an if condition checking for a null @timestamp
| } | ||
|
|
||
| @Override | ||
| protected String getOrigin() { |
There was a problem hiding this comment.
Since this no longer needs to be run with the user permissions, it seemed better to not require the user to have put-pipeline, and to make new user with system perms (or something like that) and only give it to this registry.
| } | ||
| } | ||
| ], | ||
| "version": 1000 |
There was a problem hiding this comment.
Since version is now handled by the index template registry, the way to keep it from overwriting a custom template is to use a higher version number.
| @@ -196,6 +196,7 @@ private static String maybeRewriteSingleAuthenticationHeaderForVersion( | |||
| public static final String APM_ORIGIN = "apm"; | |||
| public static final String OTEL_ORIGIN = "otel"; | |||
| public static final String REINDEX_DATA_STREAM_ORIGIN = "reindex_data_stream"; | |||
| public static final String MIGRATE_ORIGIN = "migrate"; | |||
There was a problem hiding this comment.
Let's use the existing STACK_ORIGIN origin, I think there's an aversion to adding too many of these if I remember correctly.
It is possible for documents in a data stream to not have a
@timestampfield. For example, if an existing index is added to a data stream. But, this will cause the reindex operation to fail as the destination index will already contained the_data_stream_timestampvalue from the source mappings, causing each document to be checked for an@timestampfield.This change adds a pipeline, tentatively called
reindex-data-stream, which adds a@timestampfield with a value of0to destination docs if a timestamp is missing. If a user creates a pipeline with this name, and without aversionfield, the user's pipeline will be used instead of the built in pipeline.