feat: add a LRU-based cache module for preheat jobs #945

SouthWest7 · 2025-01-16T09:34:23Z

Description

This PR introduces a Cache module designed specifically for preheating jobs, providing an LRU-based mechanism for storing and managing piece content. The purpose of the cache is to enable other peers within the cluster to access user-space maintained cached pieces when downloading the same tasks.

Dependency Updates:

Updates the version of dragonfly-api from 2.1.3 to 2.1.4 in Cargo.toml.

Refactoring and Method Changes:

Refactored the upload_piece method in dragonfly-client-storage/src/lib.rs. This change aims to support uploading piece content from cache when hitting cache.
Changed the return type from Result<impl AsyncRead> to Result<Pin<Box<dyn AsyncRead + Send>>> because reading cache or disk will return different types implementing AsyncRead.
Refactored the download_from_local_into_async_read and upload_from_local_into_async_read method in dragonfly-client/src/resource/piece.rs.
Modified their return types, because they both call upload_piece and return its result.
Refactored the download_from_source and download_from_parent method in dragonfly-client/src/resource/piece.rs. Add a new parameter load_to_cache on both. This change is aim to support loading piece content to cache when downloading piece content from parent peer or source.
Refactored the download_partial_from_local in dragonfly-client/src/resource/persistent_cache_task.rs. Add a new parameter load_to_cache. Added the code for loading content into cache.
Add a new method upload_piece_from_cache in dragonfly-client-storage/src/lib.rs. This method is used by the storage to provide reading cache.
Add a new method load_piece_to_cache in dragonfly-client-storage/src/lib.rs. This method is used by the storage to provide writing cache.
Add a new parameter load_to_cache in some methods.

Metadata and Struct Changes:

Add a new file cache in Storage in dragonfly-client-storage/src/lib.rs.

Module Changes:

Add a new module Cache with unit tests in dragonfly-client-storage/src/cache.rs.
For detailed design ideas and method definition, please refer to related issue.

Related Issue

dragonflyoss/dragonfly#3742

Motivation and Context

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation Update (if none of the other choices apply)

Checklist

My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.

Signed-off-by: southwest <[email protected]>

…tion Signed-off-by: southwest <[email protected]>

Signed-off-by: southwest <[email protected]>

…with added annotations. Signed-off-by: southwest <[email protected]>

Signed-off-by: southwest <[email protected]>

…he' into feature/storage-cache

Signed-off-by: southwest <[email protected]>

codecov · 2025-01-16T10:10:02Z

Codecov Report

Attention: Patch coverage is 96.36364% with 10 lines in your changes missing coverage. Please review.

Project coverage is 34.40%. Comparing base (d4096db) to head (fe27f7e).

Files with missing lines	Patch %	Lines
dragonfly-client/src/resource/task.rs	0.00%	6 Missing ⚠️
dragonfly-client-storage/src/cache.rs	99.25%	2 Missing ⚠️
dragonfly-client-storage/src/lib.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #945      +/-   ##
==========================================
+ Coverage   32.66%   34.40%   +1.73%     
==========================================
  Files          60       61       +1     
  Lines        9792    10067     +275     
==========================================
+ Hits         3199     3464     +265     
- Misses       6593     6603      +10

Files with missing lines	Coverage Δ
dragonfly-client/src/resource/piece.rs	`55.51% <ø> (ø)`
dragonfly-client-storage/src/cache.rs	`99.25% <99.25%> (ø)`
dragonfly-client-storage/src/lib.rs	`1.38% <0.00%> (-0.04%)`	⬇️
dragonfly-client/src/resource/task.rs	`0.00% <0.00%> (ø)`

Signed-off-by: southwest <[email protected]>

dragonfly-client-storage/Cargo.toml

dragonfly-client-storage/src/cache.rs

Signed-off-by: southwest miao <[email protected]>

gaius-qi · 2025-01-24T02:48:11Z

dragonfly-client-storage/src/cache.rs

+#[derive(Clone)]
+pub struct Cache {
+    /// pieces stores the pieces with their piece id and content.
+    pieces: Arc<Mutex<LruCache<String, bytes::Bytes>>>,


Use RWMutex.

gaius-qi · 2025-01-24T02:53:57Z

dragonfly-client-storage/src/cache.rs

+    }
+
+    #[tokio::test]
+    async fn test_cache_initialization() {


Add more cases.

gaius-qi · 2025-01-24T02:55:42Z

dragonfly-client-storage/src/cache.rs

+/// Cache implements the cache for storing piece content by LRU algorithm.
+impl Cache {
+    /// new creates a new cache with the specified capacity.
+    pub fn new(config: Arc<Config>) -> Result<Self> {


pub fn new(capacity: usize) -> Result<Self> {

gaius-qi · 2025-01-24T02:56:48Z

dragonfly-client-storage/src/lib.rs

@@ -405,7 +414,7 @@ impl Storage {
        piece_id: &str,
        task_id: &str,
        range: Option<Range>,
-    ) -> Result<impl AsyncRead> {
+    ) -> Result<Pin<Box<dyn AsyncRead + Send>>> {


Return Result<impl AsyncRead>.

gaius-qi · 2025-01-24T02:59:56Z

dragonfly-client-storage/src/lib.rs

@@ -415,6 +424,24 @@ impl Storage {
        // Get the piece metadata and return the content of the piece.
        match self.metadata.get_piece(piece_id) {
            Ok(Some(piece)) => {
+                // Try to upload piece content form cache.
+                if !self.cache.is_empty().await && self.cache.contains_piece(piece_id).await {


if self.cache.contains_piece(piece_id).await

gaius-qi · 2025-01-24T03:03:21Z

dragonfly-client/src/resource/piece.rs

@@ -520,6 +522,14 @@ impl Piece {
                };
            })?;

+        // Load piece content to cache from parent
+        if load_to_cache {


Move to download_piece_from_parent_finished and download_piece_from_source_finished.

gaius-qi · 2025-01-24T03:05:59Z

dragonfly-client/src/resource/piece.rs

+        if load_to_cache {
+            // Load piece content to cache from source.
+            self.storage
+                .load_piece_to_cache(piece_id, &mut reader, length)


Move to download_piece_from_parent_finished and download_piece_from_source_finished.

gaius-qi · 2025-01-24T03:08:46Z

dragonfly-client/src/resource/task.rs

-            // If need_piece_content is true, read the piece content from the local.
-            if need_piece_content {
+            // If need_piece_content or load_to_cache is true, read the piece content from the local.
+            if need_piece_content || load_to_cache {


SouthWest7 and others added 18 commits January 6, 2025 15:39

feat(storage): add cache configuration support for preheat tasks

0f54cb0

Signed-off-by: southwest <[email protected]>

feat(storage): added cache initialization, according to the configura…

d319dc4

…tion Signed-off-by: southwest <[email protected]>

feat(storage): add cache configuration support for preheat tasks

af21663

Signed-off-by: southwest <[email protected]>

merge with main branch

15e3455

Signed-off-by: southwest <[email protected]>

feat: 'cache' definition and implementation

04a4a2a

Signed-off-by: southwest <[email protected]>

refactor: change 'Cache' configurationand default value setting

373072d

Signed-off-by: southwest <[email protected]>

chore: modify comments of Cache

aa81cc8

Signed-off-by: southwest <[email protected]>

chore: add workflow diagram of Cache

8b45619

Signed-off-by: southwest <[email protected]>

Merge branch 'dragonflyoss:main' into feature/storage-cache

5128719

Merge branch 'dragonflyoss:main' into feature/storage-cache

2925faa

Signed-off-by: southwest <[email protected]>

feat: cache is enabled by default, and the diagram has been modified …

f306041

…with added annotations. Signed-off-by: southwest <[email protected]>

chore: modify the comment of 'default_storage_cache_capacity()'

b807755

Signed-off-by: southwest <[email protected]>

Merge remote-tracking branch 'refs/remotes/origin/feature/storage-cac…

987cbb8

…he' into feature/storage-cache

merge with branch feature/config

80cbe94

Signed-off-by: southwest <[email protected]>

feat: add def & impl of Cache and cache access logic

826c78a

Signed-off-by: southwest <[email protected]>

chore: add diagram and comment for Cache

e6f0d82

Signed-off-by: southwest <[email protected]>

feat: wrting cache

1b3ade7

Signed-off-by: southwest <[email protected]>

feat: add unit tests and code optimization

3cdde85

Signed-off-by: southwest <[email protected]>

SouthWest7 requested a review from a team as a code owner January 16, 2025 09:34

SouthWest7 requested review from bergwolf, anjia0532 and gaius-qi January 16, 2025 09:34

SouthWest7 added 4 commits January 16, 2025 18:00

chore: resolve conlicts

436687e

Signed-off-by: southwest <[email protected]>

chore: resolve conlicts

7ef46a7

Signed-off-by: southwest <[email protected]>

chore: resolve conlicts

0a0c516

Signed-off-by: southwest <[email protected]>

chore: resolve conlicts

69d01f3

Signed-off-by: southwest <[email protected]>

SouthWest7 added 3 commits January 16, 2025 18:26

chore: fixed fmt issues.

04f0702

Signed-off-by: southwest <[email protected]>

Merge branch 'main' into feature/storage-cache

17a9657

Signed-off-by: southwest <[email protected]>

fix: upload_piece

13500b0

Signed-off-by: southwest <[email protected]>

gaius-qi requested changes Jan 21, 2025

View reviewed changes

dragonfly-client-storage/src/cache.rs Show resolved Hide resolved

gaius-qi requested changes Jan 21, 2025

View reviewed changes

dragonfly-client-storage/src/cache.rs Outdated Show resolved Hide resolved

SouthWest7 added 2 commits January 23, 2025 21:07

Merge branch 'main' into feature/storage-cache

be859af

Signed-off-by: southwest miao <[email protected]>

fix: fmt

fe27f7e

Signed-off-by: southwest miao <[email protected]>

gaius-qi requested changes Jan 24, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add a LRU-based cache module for preheat jobs #945

feat: add a LRU-based cache module for preheat jobs #945

SouthWest7 commented Jan 16, 2025 •

edited

Loading

codecov bot commented Jan 16, 2025 •

edited

Loading

gaius-qi Jan 24, 2025

gaius-qi Jan 24, 2025

gaius-qi Jan 24, 2025

gaius-qi Jan 24, 2025

gaius-qi Jan 24, 2025

gaius-qi Jan 24, 2025

gaius-qi Jan 24, 2025

gaius-qi Jan 24, 2025

feat: add a LRU-based cache module for preheat jobs #945

Are you sure you want to change the base?

feat: add a LRU-based cache module for preheat jobs #945

Conversation

SouthWest7 commented Jan 16, 2025 • edited Loading

Description

Dependency Updates:

Refactoring and Method Changes:

Metadata and Struct Changes:

Module Changes:

Related Issue

Motivation and Context

Screenshots (if appropriate)

Types of changes

Checklist

codecov bot commented Jan 16, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SouthWest7 commented Jan 16, 2025 •

edited

Loading

codecov bot commented Jan 16, 2025 •

edited

Loading