Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

start adding documentation on updating mapping #5290

Merged
merged 10 commits into from
Jan 6, 2025

Conversation

trinity-1686a
Copy link
Contributor

Description

add documentation on the known limitations when updating a doc mapping

@rdettai i know you were surprised by a few things, please step in if you think I missed anything important
@fmassot you might have opinions on things to add too.

this is based on the expect stat of updates once all currently existing PR on the subject are merged

@trinity-1686a trinity-1686a added the documentation Improvements or additions to documentation label Aug 1, 2024
@trinity-1686a trinity-1686a mentioned this pull request Aug 1, 2024
9 tasks
Copy link

github-actions bot commented Aug 1, 2024

On SSD:

Average search latency is 1.01x that of the reference (lower is better).
Ref run id: 3629, ref commit: 450f26a
Link

On GCS:

Average search latency is 0.986x that of the reference (lower is better).
Ref run id: 3630, ref commit: 450f26a
Link

docs/reference/updating-mapper.md Show resolved Hide resolved
docs/reference/updating-mapper.md Outdated Show resolved Hide resolved
docs/reference/updating-mapper.md Outdated Show resolved Hide resolved
docs/reference/updating-mapper.md Outdated Show resolved Hide resolved
docs/reference/updating-mapper.md Outdated Show resolved Hide resolved
docs/reference/updating-mapper.md Show resolved Hide resolved
@trinity-1686a trinity-1686a force-pushed the trinity/update-mapper-doc branch from 36528e1 to 02f2edb Compare September 27, 2024 10:46
docs/reference/rest-api.md Outdated Show resolved Hide resolved
docs/reference/rest-api.md Outdated Show resolved Hide resolved
docs/reference/updating-mapper.md Outdated Show resolved Hide resolved

## Indexing

When you update a doc mapping for an index, Quickwit will restart indexing pipelines to take the changes into account. As both this operation and the document ingestion are asynchronous, you can't assume documents sent immediately after the update will necessarily use the new mapping nor that documents sent immediately before the update won't be indexed using the new doc mapping.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
When you update a doc mapping for an index, Quickwit will restart indexing pipelines to take the changes into account. As both this operation and the document ingestion are asynchronous, you can't assume documents sent immediately after the update will necessarily use the new mapping nor that documents sent immediately before the update won't be indexed using the new doc mapping.
When you update a doc mapping for an index, Quickwit will restart indexing pipelines to take the changes into account. As both this operation and the document ingestion are asynchronous, you can't assume documents sent immediately after the update will necessarily use the new mapping or documents sent immediately before the update will be indexed using the new doc mapping.

Removing the double negative.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think won't was what i meant, if you send a document while there is a large queue in the wal, and then update the mapping, it is possible for the pipeline to be restarted before reaching that document, so it may be indexed with the newer mapping (or it may not, if the wal was short/empty).

docs/reference/updating-mapper.md Outdated Show resolved Hide resolved
docs/reference/updating-mapper.md Outdated Show resolved Hide resolved
docs/reference/updating-mapper.md Outdated Show resolved Hide resolved
docs/reference/updating-mapper.md Outdated Show resolved Hide resolved
docs/reference/updating-mapper.md Outdated Show resolved Hide resolved
@trinity-1686a trinity-1686a merged commit ec20059 into main Jan 6, 2025
4 of 5 checks passed
@trinity-1686a trinity-1686a deleted the trinity/update-mapper-doc branch January 6, 2025 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants