RFC: use versioning for edge repair in HNSW #35
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yairgott
requested changes
Feb 11, 2025
Comment on lines
+22
to
+24
| - **Tracks Node Reuse**: Each node has a version counter incremented every time the node is reused (after being deleted). | ||
| - **Detects Stale Edges**: Adjacency lists store both an identifier and the version of the target node at link creation time. During search, edges whose stored version doesn’t match the node’s current version are treated as invalid (stale). | ||
| - **Supports Lazy Repair**: An asynchronous background process progressively removes or updates invalid edges, preventing them from repeatedly impacting query results. |
There was a problem hiding this comment.
Can you clarify what you aim to achieve with node versioning?
Just a reminder: ValkeySearch uses a time-slice mechanism with mutual exclusion between reads and writes to ensure consistency. This guarantees that a mutation is either applied to all indexes or to none, ensuring atomicity. This consistency model applies to all indexes, not just vector indexes.
|
|
||
| Hierarchical Navigable Small World (HNSW) graphs are widely used for Approximate Nearest Neighbor (ANN) search due to their efficient graph-based structure. However, in production systems that support dynamic updates (e.g., new data points arriving or existing ones being removed), deleted nodes are sometimes reused (via `allow_replace_deleted_` in hnswlib). This reuse introduces two major problems: | ||
|
|
||
| - **Stale Inbound Links**: When a node is deleted and subsequently reused, any existing inbound links from other nodes’ adjacency lists point to a different “state.” These stale references mislead search queries. |
There was a problem hiding this comment.
Today reuse is not enabled.
| Hierarchical Navigable Small World (HNSW) graphs are widely used for Approximate Nearest Neighbor (ANN) search due to their efficient graph-based structure. However, in production systems that support dynamic updates (e.g., new data points arriving or existing ones being removed), deleted nodes are sometimes reused (via `allow_replace_deleted_` in hnswlib). This reuse introduces two major problems: | ||
|
|
||
| - **Stale Inbound Links**: When a node is deleted and subsequently reused, any existing inbound links from other nodes’ adjacency lists point to a different “state.” These stale references mislead search queries. | ||
| - **Recall Degradation**: Searches that traverse stale edges can produce suboptimal or incorrect nearest-neighbor results—an issue with potentially severe impact on high-recall vector search applications. |
There was a problem hiding this comment.
I'm uncertain about this given the current implementation, which doesn't reuse deleted nodes. On the other hand, since HNSWlib handles deletion by marking nodes as removed without reclaiming them, a large number of deleted nodes could lead to increased query runtimes.
| Each encounter with a stale edge enqueues a repair task (e.g., `(owner_node_id, edge_index)` or `(neighbor_id, stored_version)`). | ||
|
|
||
| - **Batch Repairs**: | ||
| A background process periodically locks node adjacency lists, removes or fixes stale entries, and moves on. |
There was a problem hiding this comment.
I think you should provide more details on how the node removal algorithm would work. Specifically:
- Handling Removal of an Entry Point at a Higher Level: How would you accommodate the scenario where the removed node is an entry point at a higher layer?
- Finding Incoming Links: How would you identify all nodes that have links pointing to the removed node, given that HNSW does not maintain reverse links?
- Fixing Stale Entries: Could you clarify what you mean by "stale" entries? Additionally, how would you handle and clean up these entries to maintain graph integrity and avoid performance degradation?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding an RFC for improving hnsw library to allow better handling of deleted nodes in the graph.