Introduced data freshness with the MongoDB implementation at connection-level and at query-level #209

suresh-prakash · 2024-09-13T10:28:30Z

This PRs gives the flexibility to redirect reads optionally to secondary MongoDB nodes.
This configuration can be tweaked at the connection-level while instantiating a connection-config object. It can also be configured at a query level. By default, all the reads will go to the primary node for backward compatibility.

Preference order:

Query level configuration
Connection level configuration if query level configuration is default (unspecified)
Document store defaults to reads from primary (if both query-level and connection-level read preferences are unspecified/default).

In this PR, the Postgres implementation simply ignores the configuration.

…on-level and at query-level

codecov · 2024-09-13T10:30:35Z

Codecov Report

Attention: Patch coverage is 83.33333% with 8 lines in your changes missing coverage. Please review.

Project coverage is 79.69%. Comparing base (7e161f5) to head (e85b786).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...tore/model/config/mongo/MongoConnectionConfig.java	0.00%	0 Missing and 2 partials ⚠️
...mongo/collection/MongoReadPreferenceConverter.java	81.81%	2 Missing ⚠️
...core/documentstore/model/options/QueryOptions.java	75.00%	1 Missing ⚠️
...race/core/documentstore/mongo/MongoCollection.java	0.00%	1 Missing ⚠️
...ongo/collection/MongoCollectionOptionsApplier.java	91.66%	0 Missing and 1 partial ⚠️
...ore/documentstore/postgres/PostgresCollection.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #209      +/-   ##
============================================
- Coverage     79.70%   79.69%   -0.01%     
- Complexity     1022     1036      +14     
============================================
  Files           194      198       +4     
  Lines          4879     4921      +42     
  Branches        408      411       +3     
============================================
+ Hits           3889     3922      +33     
- Misses          706      713       +7     
- Partials        284      286       +2

Flag	Coverage Δ
integration	`79.69% <83.33%> (-0.01%)`	⬇️
unit	`57.50% <81.25%> (+0.20%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-09-13T10:33:19Z

Test Results

39 files +1 39 suites +1 34s ⏱️ +2s
242 tests +7 242 ✅ +7 0 💤 ±0 0 ❌ ±0
495 runs +7 495 ✅ +7 0 💤 ±0 0 ❌ ±0

Results for commit e85b786. ± Comparison against base commit 7e161f5.

♻️ This comment has been updated with latest results.

kotharironak · 2024-09-18T08:21:10Z

build.gradle.kts

@@ -6,7 +6,7 @@ plugins {
  id("org.hypertrace.ci-utils-plugin") version "0.3.0"
  id("org.hypertrace.publish-plugin") version "1.0.2" apply false
  id("org.hypertrace.code-style-plugin") version "1.1.0" apply false
-  id("org.owasp.dependencycheck") version "8.2.1"
+  id("org.owasp.dependencycheck") version "8.4.3"


Any reason for not moving to latest version - 10.0.4?

I tried that version. It's failing (mostly due to Java version incompatibility).

kotharironak · 2024-09-18T08:22:19Z

document-store/src/integrationTest/java/org/hypertrace/core/documentstore/DocStoreTest.java

@@ -71,7 +71,7 @@ public class DocStoreTest {
  public static void init() {
    datastoreMap = Maps.newHashMap();
    mongo =
-        new GenericContainer<>(DockerImageName.parse("mongo:4.4.0"))
+        new GenericContainer<>(DockerImageName.parse("mongo:7.0.14"))


Is the MongoDB version 7.0.14? Is this the same version that has been deployed?

kotharironak · 2024-09-18T08:26:03Z

document-store/src/main/java/org/hypertrace/core/documentstore/Collection.java

+   * @param queryOptions The query options
+   * @return {@link CloseableIterator} of matching documents
+   */
+  CloseableIterator<Document> query(


Q: the existing query api deprecated?

Don't think that's required. If clients want to query with the default configurations, the older method can be used.

kotharironak · 2024-09-18T08:29:18Z

document-store/src/main/java/org/hypertrace/core/documentstore/model/options/DataFreshness.java

+@SuppressWarnings("UnnecessarySemicolon")
+public enum DataFreshness {
+  SYSTEM_DEFAULT,
+  REALTIME_FRESHNESS,


Q: What does the freshness mean?

Freshness means the liveliness of the data.
REALTIME_FRESHNESS means we want the most up-to-date data.
NEAR_REALTIME_FRESHNESS means a little bit of stale data is acceptable.

Here NEAR_REALTIME_FRESHNESS means fetching data from secondary, right?

Yes. Basically "SECONDARY" is preferred. In case "SECONDARY" is not available, it'll fetch from "PRIMARY".

kotharironak · 2024-09-18T08:41:53Z

...n/java/org/hypertrace/core/documentstore/mongo/collection/MongoCollectionOptionsApplier.java

+    final CacheKey cacheKey =
+        CacheKey.builder().readPreference(readPreference(connectionConfig, queryOptions)).build();
+    return collectionCache.computeIfAbsent(
+        cacheKey, key -> collection.withReadPreference(key.readPreference()));


How is the CacheKey associated with the collection? Since ReadPreference can be either primary or secondary, does this mean that this.collectionCache contains a maximum of 2 values?

Since ReadPreference can be either primary or secondary, does this mean that this.collectionCache contains a maximum of 2 values?

Yes.

kotharironak · 2024-09-19T06:27:58Z

document-store/src/main/java/org/hypertrace/core/documentstore/model/options/DataFreshness.java

+@SuppressWarnings("UnnecessarySemicolon")
+public enum DataFreshness {
+  SYSTEM_DEFAULT,
+  REALTIME_FRESHNESS,


Here NEAR_REALTIME_FRESHNESS means fetching data from secondary, right?

Introduced data freshness with the MongoDB implementation at connecti…

8077db8

…on-level and at query-level

suresh-prakash requested review from avinashkolluru, kotharironak and skjindal93 as code owners September 13, 2024 10:28

Minor fixes

b3ab9a0

suresh-prakash added 3 commits September 13, 2024 18:33

Made proper fixes with unit tests

8d1037f

Renaming

a97c11a

Add a missing implementation link

21b8783

suresh-prakash requested a review from aaron-steinfeld September 13, 2024 13:33

suresh-prakash and others added 6 commits September 16, 2024 13:10

Test for builder and getter

78503c2

Upgrade MongoDB version in integration tests

38d3b6c

Update MongoDB version in integration tests

a16f4d5

Merge branch 'main' into introduce_data_freshness_parameter

fd2b3f4

Upgrade dependency check version

4e0b29c

Update version

e85b786

kotharironak reviewed Sep 18, 2024

View reviewed changes

kotharironak approved these changes Sep 19, 2024

View reviewed changes

suresh-prakash merged commit 72cc0d2 into main Sep 19, 2024
6 of 7 checks passed

suresh-prakash deleted the introduce_data_freshness_parameter branch September 19, 2024 06:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduced data freshness with the MongoDB implementation at connection-level and at query-level #209

Introduced data freshness with the MongoDB implementation at connection-level and at query-level #209

suresh-prakash commented Sep 13, 2024 •

edited

Loading

codecov bot commented Sep 13, 2024 •

edited

Loading

github-actions bot commented Sep 13, 2024 •

edited

Loading

kotharironak Sep 18, 2024

suresh-prakash Sep 18, 2024

kotharironak Sep 18, 2024

kotharironak Sep 18, 2024

suresh-prakash Sep 18, 2024 •

edited

Loading

kotharironak Sep 18, 2024

suresh-prakash Sep 18, 2024

kotharironak Sep 19, 2024

suresh-prakash Sep 19, 2024

kotharironak Sep 18, 2024

suresh-prakash Sep 18, 2024

kotharironak Sep 19, 2024

Introduced data freshness with the MongoDB implementation at connection-level and at query-level #209

Introduced data freshness with the MongoDB implementation at connection-level and at query-level #209

Conversation

suresh-prakash commented Sep 13, 2024 • edited Loading

codecov bot commented Sep 13, 2024 • edited Loading

Codecov Report

github-actions bot commented Sep 13, 2024 • edited Loading

Test Results

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suresh-prakash Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suresh-prakash commented Sep 13, 2024 •

edited

Loading

codecov bot commented Sep 13, 2024 •

edited

Loading

github-actions bot commented Sep 13, 2024 •

edited

Loading

suresh-prakash Sep 18, 2024 •

edited

Loading