Add bandit support #25

aarsilv · 2024-08-06T03:38:10Z

Eppo Internal:
🎟️ Ticket: FF-2743 - Common Shared Bandits Tests Passing

This pull request adds support for contextual multi-armed bandits to the common SDK for future use by upstream Universal Flag Configuration (UFC)-powered Java and--perhaps in the future--Android SDKs.

Much of the classes, tests, and logic have been ported from the existing Java bandits implementation, updated to account for new decisions and edge-case handling as bandit development has taken place.

Bandit functionality is verified against our shared SDK test data bandit tests. Additional tests ensure bandits log and handle errors as expected.

aarsilv · 2024-08-06T03:39:07Z

build.gradle

@@ -23,6 +23,7 @@ dependencies {
  testImplementation 'commons-io:commons-io:2.11.0'
  testImplementation "com.google.truth:truth:1.4.4"
  testImplementation 'org.mockito:mockito-core:4.11.0'
+  testImplementation 'org.mockito:mockito-inline:4.11.0'


Needed for mocking static methods (such as evaluateBandit())

aarsilv · 2024-08-06T03:50:18Z

src/main/java/cloud/eppo/BanditEvaluator.java

+      // The coefficient implementation knows how to score
+      double attributeScore = attributeCoefficients.scoreForAttributeValue(contextValue);


This is a bit different than other SDKs; the logic for how to score a given attribute value is collocated with the type of attribute (i.e. BanditNumericAttributeCoefficients and BanditCategoricalAttributeCoefficients)

aarsilv · 2024-08-06T03:56:11Z

src/main/java/cloud/eppo/ConfigurationRequestor.java

+    Set<String> neededModelVersions = configurationStore.banditModelVersions();
+    boolean needBanditParameters = !loadedBanditModelVersions.containsAll(neededModelVersions);


Taking advantage of the latest and greatest UFC feature: bandit model versions.
We will only request bandit parameters if we are missing model versions at play (as opposed to every time we poll if there are one or more bandits).

Shooting for bleeding edge all at once eh? I like your style 🤠

aarsilv · 2024-08-06T04:01:34Z

src/main/java/cloud/eppo/ConfigurationStore.java

    } else {
      // TODO: atomic flags to prevent clobbering like android does
      // Record that flags were set from a response so we don't later clobber them with a
      // slow cache read
      flags = new ConcurrentHashMap<>(config.getFlags());
-      log.debug("Loaded " + flags.size() + " flags from configuration response");
+      banditReferences = new ConcurrentHashMap<>(config.getBanditReferences());


banditReferences is a top-level UFC property containing some information on any bandits at play (but not their model data, which is provided separately)

aarsilv · 2024-08-06T04:04:49Z

src/main/java/cloud/eppo/ConfigurationStore.java

+      for (BanditFlagVariation banditFlagVariation : banditReference.getFlagVariations()) {
+        if (banditFlagVariation.getFlagKey().equals(flagKey)


While we could use the shortcut of assuming the variation key and value and bandit key are all the same, since the UFC explicitly provides the information to be more specific, I'll take advantage of it. We can have other SDKs do this once needed.

aarsilv · 2024-08-06T04:41:29Z

src/test/java/cloud/eppo/EppoClientBanditTest.java

+  private static final BanditLogger mockBanditLogger = mock(BanditLogger.class);
+  private static final Date testStart = new Date();
+
+  // TODO: possibly consolidate code between this and the non-bandit test


Thinking of leaving this as a TODO for now until everything is said and done. I don't want to prematurely combine parts of tests only to have things diverge once non-bandit-only (for now) things like caching is layered on.

aarsilv · 2024-08-06T04:42:00Z

src/test/java/cloud/eppo/EppoClientBanditTest.java

+    EppoClient.getInstance().setIsGracefulFailureMode(false);
+  }
+
+  @ParameterizedTest


Shared UFC Bandit Tests! 💪

aarsilv · 2024-08-06T04:44:50Z

src/test/java/cloud/eppo/EppoClientTest.java

-import static org.junit.Assert.assertThrows;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
+import static org.junit.jupiter.api.Assertions.assertEquals;


Consolidating on the jupiter flavors to match the imported @Test

aarsilv · 2024-08-06T04:47:37Z

src/test/java/cloud/eppo/EppoClientTest.java

@@ -201,26 +192,26 @@ private <T> void assertAssignment(

    if (assignment instanceof Boolean) {
      assertEquals(
-          failureMessage, expectedSubjectAssignment.getAssignment().booleanValue(), assignment);
+          expectedSubjectAssignment.getAssignment().booleanValue(), assignment, failureMessage);


jupiter flavored asserts have the error message at the end (which I like more anyways 🤷‍♂️ )

aarsilv · 2024-08-06T04:52:42Z

src/test/java/cloud/eppo/helpers/BanditTestCase.java

+  private final String flag;
+  private final String defaultValue;
+  private final List<SubjectBanditAssignment> subjects;
+  private String fileName;


Set later (after deserialization) to make helpful test case "names" that are displayed in IDE/terminal

aarsilv · 2024-08-06T04:59:42Z

src/test/resources/bandits-parameters-1.json

@@ -0,0 +1,101 @@
+{


For now, our deserializer tests use their own files (as opposed to the shared UFC test file). I think this is ok, as it makes the deserializer tests less fragile. The shared test data should ensure that the SDK correctly handles anything new thrown at it. Nevertheless, we could consider having the deserializer tests also use shared test data.

just my 0.02 (or 0.0144USD after adjusting for ForEx)
For deserialization, I think using these files is the best way to go. Imo, it doesn't make a lot of sense to pull in dynamic data, parse it, and then assert on hard-coded values. Regardless of whether the underlying data is stable, the mismatch of dynamic vs coded feels awkward and hangs a lampshade on the fragility.

aarsilv · 2024-08-06T05:05:01Z

src/test/java/cloud/eppo/EppoClientTest.java

-      throw new RuntimeException(ex);
-    }
+    assertTrue(capturedAssignment.getTimestamp().after(testStart));
+    Date inTheNearFuture = new Date(System.currentTimeMillis() + 1);


Deflake the test for when things are particularly speedy

giorgiomartini0

Impeccable! Big THANK YOU for the detailed comments explaining what changed and why 🙏

giorgiomartini0 · 2024-08-06T16:35:53Z

src/main/java/cloud/eppo/BanditEvaluationResult.java

+
+  private final String flagKey;
+  private final String subjectKey;
+  private final DiscriminableAttributes subjectAttributes;


Fancy word! I like it

giorgiomartini0 · 2024-08-06T16:55:59Z

src/test/java/cloud/eppo/helpers/AssignmentTestCaseDeserializer.java

@@ -60,7 +60,7 @@ private List<SubjectAssignment> deserializeSubjectAssignments(JsonNode jsonNode)
  }

  private TestCaseValue deserializeTestCaseValue(JsonNode jsonNode) {
-    if (jsonNode != null && jsonNode.isObject()) {
+    if (jsonNode != null && (jsonNode.isObject() || jsonNode.isArray())) {


Is this defensive or is there a specific test that needs it?

We have a test case with JSON array now! https://github.com/Eppo-exp/sdk-test-data/blob/main/ufc/tests/test-json-array-config-flag.json#L4

Added four days ago (PR) when a customer caught it wasn't working for them 😞

We're still deciding as a company if we actually want to support this.

I actually rolled back that test case earlier after talking with Leo. Some of the SDKs would need a fair amount of work to support the JSON Array of Objects as a variation value. More work, I think, than is agreeable while it's questionable whether support for this is desired.

Rolling back the test case sounds good. Any harm in leaving support in SDKs that can handle it (this one, node, etc.?)

no hard that I can think of - especially on the test parsing side.

giorgiomartini0 · 2024-08-06T17:00:19Z

src/main/java/cloud/eppo/ConfigurationStore.java

+    // Note: In practice this double loop should be quite quick as the number of bandits and bandit
+    // variations will be small. Should this ever change, we can optimize things.


Agreed - no need to prematurely optimize this.

typotter

what a monster of a pull 💪
Requesting changes but only because of the math mismatch here

typotter · 2024-08-07T05:13:07Z

src/main/java/cloud/eppo/BanditEvaluator.java

+      // Compute weight (probability)
+      double unboundedProbability =
+          1 / (actionScores.size() + (gamma * (highestScore - actionScore.getValue())));
+      double boundedProbability = Math.max(unboundedProbability, actionProbabilityFloor);


In the other SDKs, we divide actionProbabilityFloor by the number of actions to weigh;

Comment 🧵

Yikes! GREAT catch 🎉 Dividing by actions is what we should do.

This logic was part of what was copied over from the original Java implementation, before we changed up our strategy, and I failed to update it.

What this does mean is that we don't have a test enforcing that logic. It may be hard to put one together, but I'll see if I can. 🤔

The reason why this doesn't get teased out in a test is because we're using small min probability floors and typically a gamma of 1 with only a couple of actions. With some tweaking of scoring parameters, should be able to create a test case that pushes it over the edge

PR for test cases that cover this (and more!) here: Eppo-exp/sdk-test-data#52

typotter · 2024-08-07T05:13:39Z

src/main/java/cloud/eppo/BanditEvaluator.java

+                            ShardUtils.getShard(
+                                flagKey + "-" + subjectKey + "-" + actionKey,
+                                BANDIT_ASSIGNMENT_SHARDS))
+                    .thenComparing(actionKey -> actionKey))


typotter · 2024-08-07T05:14:41Z

src/main/java/cloud/eppo/ConfigurationRequestor.java

+    Set<String> neededModelVersions = configurationStore.banditModelVersions();
+    boolean needBanditParameters = !loadedBanditModelVersions.containsAll(neededModelVersions);


Shooting for bleeding edge all at once eh? I like your style 🤠

typotter · 2024-08-07T05:16:01Z

src/main/java/cloud/eppo/ConfigurationRequestor.java

  public void load() {
    log.debug("Fetching configuration");
-    Response response = client.get("/api/flag-config/v1/config");
+    String flagConfigurationJsonString = requestBody("/api/flag-config/v1/config");
+    configurationStore.setFlagsFromJsonString(flagConfigurationJsonString);


Do you need to worry about threading here and lock reads until the bandit models are set below?

Good thinking! The underlying implementation is a thread-safe ConcurrentHashMap, which is what we are using in the Android SDK without (known) issues

The ConcurrentHashMap underlying will keep reads and writes happening in an orderly fashion, but there's no lock around the setting of the flags and the subsequent setting of the bandit parameters below, so, there's a brief instant the config store could be accessed while it has incomplete configuration set. If a caller manages to evaluate a bandit between those two instructions, the SDK will have incomplete data and may not have the required bandit. The scenario is probably unlikely, contrived and probably nearly impossible to test but I'm curious how wrong things would go if this happened (user would get returned the bandit ket as the variation with action=null which the devs should already have handling for in their app).

typotter · 2024-08-07T05:18:30Z

src/main/java/cloud/eppo/ConfigurationRequestor.java

+    String flagConfigurationJsonString = requestBody("/api/flag-config/v1/config");
+    configurationStore.setFlagsFromJsonString(flagConfigurationJsonString);
+
+    Set<String> neededModelVersions = configurationStore.banditModelVersions();


Is it possible for more than one bandit to have the same model version string?

No, model versions are unique to each bandit.

Under the hood, this version ID--for now--is the auto incrementing primary key in a model_versions table that has versions for all bandits (as it includes the bandit id)

Thanks for the clarification

typotter · 2024-08-07T05:24:43Z

src/main/java/cloud/eppo/ConfigurationStore.java

+    // Note: In practice this double loop should be quite quick as the number of bandits and bandit
+    // variations will be small. Should this ever change, we can optimize things.


typotter · 2024-08-07T05:30:50Z

src/main/java/cloud/eppo/EppoClient.java

+        }
+
+        // Update result to reflect that we've been assigned an action
+        result = new BanditResult(assignedVariation, banditResult.getActionKey());


result should get set above the call to banditLogger.logBanditAssignment. This ensures that if the user-provided bandit logger throws an exception, the throwIfNotGraceful call below has an accurate result to return.

Great point. That should be done and all logger calls should be wrapped in try-catches so they are non fatal

typotter · 2024-08-07T05:34:22Z

src/main/java/cloud/eppo/logging/BanditAssignment.java

+import java.util.Date;
+import java.util.Map;
+
+public class BanditAssignment {


Would you consider public final fields instead of getters? I'm not sure what the java wisdom would say here.

My impression is that Java has a weird obsession with getters 😅 The way you access fields for Records seems to confirm this.

Avoiding directly referencing fields will also make interacting with this class consistent with all the others, which I like.

Works for me. In the case of the SDKs, my preference is going to always be to "do it like the community does it" regardless of how neat and tidy I might find it.

typotter · 2024-08-07T05:36:31Z

src/main/java/cloud/eppo/ufc/dto/BanditActions.java

+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class BanditActions extends HashMap<String, DiscriminableAttributes> implements Actions {


🧑‍🍳

aarsilv · 2024-08-08T06:11:37Z

build.gradle

-  sign publishing.publications.mavenJava
-  if (System.env['CI']) {
-    useInMemoryPgpKeys(System.env.GPG_PRIVATE_KEY, System.env.GPG_PASSPHRASE)
+if (!project.gradle.startParameter.taskNames.contains('publishToMavenLocal')) {


Don't sign the maven package if publishing to local maven repository (for multi-repository development)

aarsilv · 2024-08-08T06:15:13Z

src/main/java/cloud/eppo/EppoClient.java

+                flagKey, subjectKey, subjectAttributes, actions, banditParameters.getModelData());
+
+        // Update result to reflect that we've been assigned an action
+        result = new BanditResult(assignedVariation, banditResult.getActionKey());


@typotter we now set the action part of the result as soon as we get it, and surround bandit logging with a try-catch like we do assignment logging.

Non-fatal logging error tests were added as well.

awesomesauce

aarsilv · 2024-08-08T06:15:54Z

src/test/java/cloud/eppo/EppoClientTest.java

+  @Test
+  public void testAssignmentLogErrorNonFatal() {


Adding a test to ensure errors with logging assignments are non-fatal

aarsilv · 2024-08-08T06:17:08Z

src/test/java/cloud/eppo/EppoClientBanditTest.java

+  }
+
+  @Test
+  public void testBanditLogErrorNonFatal() {


@typotter test case to ensure any issues with the user-provided bandit logging method do not interfere with returning the bandit result.

setting the bar higher and higher. 💪

aarsilv · 2024-08-08T06:17:55Z

src/main/java/cloud/eppo/BanditEvaluator.java

+      // Compute weight (probability)
+      double unboundedProbability =
+          1 / (actionScores.size() + (gamma * (highestScore - actionScore.getValue())));
+      double minimumProbability = actionProbabilityFloor / actionScores.size();


@typotter bandit probability floor now normalized

typotter

⛵

typotter · 2024-08-08T16:44:05Z

src/main/java/cloud/eppo/BanditEvaluator.java

+      // Compute weight (probability)
+      double unboundedProbability =
+          1 / (actionScores.size() + (gamma * (highestScore - actionScore.getValue())));
+      double minimumProbability = actionProbabilityFloor / actionScores.size();


typotter · 2024-08-08T16:47:08Z

src/main/java/cloud/eppo/ConfigurationRequestor.java

+    String flagConfigurationJsonString = requestBody("/api/flag-config/v1/config");
+    configurationStore.setFlagsFromJsonString(flagConfigurationJsonString);
+
+    Set<String> neededModelVersions = configurationStore.banditModelVersions();


Thanks for the clarification

typotter · 2024-08-08T16:52:41Z

src/main/java/cloud/eppo/ConfigurationRequestor.java

  public void load() {
    log.debug("Fetching configuration");
-    Response response = client.get("/api/flag-config/v1/config");
+    String flagConfigurationJsonString = requestBody("/api/flag-config/v1/config");
+    configurationStore.setFlagsFromJsonString(flagConfigurationJsonString);


The ConcurrentHashMap underlying will keep reads and writes happening in an orderly fashion, but there's no lock around the setting of the flags and the subsequent setting of the bandit parameters below, so, there's a brief instant the config store could be accessed while it has incomplete configuration set. If a caller manages to evaluate a bandit between those two instructions, the SDK will have incomplete data and may not have the required bandit. The scenario is probably unlikely, contrived and probably nearly impossible to test but I'm curious how wrong things would go if this happened (user would get returned the bandit ket as the variation with action=null which the devs should already have handling for in their app).

typotter · 2024-08-08T16:53:31Z

src/main/java/cloud/eppo/EppoClient.java

+                flagKey, subjectKey, subjectAttributes, actions, banditParameters.getModelData());
+
+        // Update result to reflect that we've been assigned an action
+        result = new BanditResult(assignedVariation, banditResult.getActionKey());


awesomesauce

typotter · 2024-08-08T16:55:55Z

src/main/java/cloud/eppo/logging/Assignment.java

@@ -79,11 +55,11 @@ public String getSubject() {
    return subject;
  }

-  public String getTimestamp() {
+  public Date getTimestamp() {


Since you're changing the data type of the date value in Assignment, would this be considered a breaking change in the SDKs which consume this class? Devs will need to make a code change in order to manage the new data type. I am not sure what the best way to signal downstream that there's an underlying API change that'll require appropriate version bumps.

typotter · 2024-08-08T16:57:34Z

src/main/java/cloud/eppo/logging/BanditAssignment.java

+import java.util.Date;
+import java.util.Map;
+
+public class BanditAssignment {


Works for me. In the case of the SDKs, my preference is going to always be to "do it like the community does it" regardless of how neat and tidy I might find it.

typotter · 2024-08-08T16:58:28Z

src/test/java/cloud/eppo/EppoClientBanditTest.java

+  }
+
+  @Test
+  public void testBanditLogErrorNonFatal() {


setting the bar higher and higher. 💪

typotter · 2024-08-08T17:01:31Z

src/test/resources/bandits-parameters-1.json

@@ -0,0 +1,101 @@
+{


just my 0.02 (or 0.0144USD after adjusting for ForEx)
For deserialization, I think using these files is the best way to go. Imo, it doesn't make a lot of sense to pull in dynamic data, parse it, and then assert on hard-coded values. Regardless of whether the underlying data is stable, the mismatch of dynamic vs coded feels awkward and hangs a lampshade on the fragility.

* Eppo Client with shared UFC tests passing (#23) * tests passing for rule evaluator, flag evaluator, and eppo value * work in progress * shared UFC tests passing * don't check in test data * changes from self-review of PR * apply spotless linter * working on tests * better test logging * use make test for tests * Add bandit support (#25) * bandit test harness ready * add bandit result * set up for dropping in bandit evaluation * bandit deserialization wired up * loading bandit parameters * bandit stuff happening * shared bandit tests passing * bandit logger classes * bandit log test passing * more tests for logger * bandit tests for graceful mode * apply spotless formatting autofix * changes from self-review of PR so far * more changes from self-review of PR * more changes from self-review * make test less fragile * bump version; don't sign local maven * bandit logging errors should be non-fatal * use normalized probability floor * update result before even attempting to log bandit * spotless 🙄 * do the rename (#26) * Remove singleton pattern (#28) * work in progress * remove singleton for base client * linter * expose bandit test harnesses * expose test uilities * changes from self-review of PR * make base client constructor protected

* Eppo Client with shared UFC tests passing (#23) * tests passing for rule evaluator, flag evaluator, and eppo value * work in progress * shared UFC tests passing * don't check in test data * changes from self-review of PR * apply spotless linter * working on tests * better test logging * use make test for tests * Add bandit support (#25) * bandit test harness ready * add bandit result * set up for dropping in bandit evaluation * bandit deserialization wired up * loading bandit parameters * bandit stuff happening * shared bandit tests passing * bandit logger classes * bandit log test passing * more tests for logger * bandit tests for graceful mode * apply spotless formatting autofix * changes from self-review of PR so far * more changes from self-review of PR * more changes from self-review * make test less fragile * bump version; don't sign local maven * bandit logging errors should be non-fatal * use normalized probability floor * update result before even attempting to log bandit * spotless 🙄 * do the rename (#26) * work in progress * remove singleton for base client * linter * expose bandit test harnesses * expose test uilities * changes from self-review of PR * make base client constructor protected * add simple logger for tests * basic profiling test * improvement not using bigint * faster getShard() * linter * more helpful failure message * increase CPU time allowance to account for slower machines * bump version

* Eppo Client with shared UFC tests passing (#23) * tests passing for rule evaluator, flag evaluator, and eppo value * work in progress * shared UFC tests passing * don't check in test data * changes from self-review of PR * apply spotless linter * working on tests * better test logging * use make test for tests * Add bandit support (#25) * bandit test harness ready * add bandit result * set up for dropping in bandit evaluation * bandit deserialization wired up * loading bandit parameters * bandit stuff happening * shared bandit tests passing * bandit logger classes * bandit log test passing * more tests for logger * bandit tests for graceful mode * apply spotless formatting autofix * changes from self-review of PR so far * more changes from self-review of PR * more changes from self-review * make test less fragile * bump version; don't sign local maven * bandit logging errors should be non-fatal * use normalized probability floor * update result before even attempting to log bandit * spotless 🙄 * do the rename (#26) * work in progress * remove singleton for base client * linter * expose bandit test harnesses * expose test uilities * changes from self-review of PR * make base client constructor protected * add simple logger for tests * basic profiling test * improvement not using bigint * faster getShard() * linter * more helpful failure message * increase CPU time allowance to account for slower machines * bump version * download test data for tests

aarsilv added 11 commits August 3, 2024 23:05

bandit test harness ready

5ff2f5e

add bandit result

c5497f6

set up for dropping in bandit evaluation

ba25612

bandit deserialization wired up

9a601ee

loading bandit parameters

31e193a

bandit stuff happening

000dd9c

shared bandit tests passing

bb747be

bandit logger classes

3b9adb3

bandit log test passing

52e499a

more tests for logger

2be2085

bandit tests for graceful mode

e952177

aarsilv changed the base branch from main to aaron/ff-2710/ufc August 6, 2024 03:38

aarsilv added 4 commits August 5, 2024 21:42

apply spotless formatting autofix

de7bbea

changes from self-review of PR so far

743fc49

more changes from self-review of PR

51070bc

more changes from self-review

e69d9aa

aarsilv commented Aug 6, 2024

View reviewed changes

make test less fragile

abb2848

aarsilv commented Aug 6, 2024

View reviewed changes

aarsilv marked this pull request as ready for review August 6, 2024 05:05

aarsilv requested review from felipecsl and giorgiomartini0 August 6, 2024 05:05

giorgiomartini0 approved these changes Aug 6, 2024

View reviewed changes

aarsilv requested a review from typotter August 6, 2024 17:11

typotter requested changes Aug 7, 2024

View reviewed changes

aarsilv requested a review from rasendubi August 8, 2024 05:21

aarsilv added 3 commits August 7, 2024 23:54

bump version; don't sign local maven

de1c745

bandit logging errors should be non-fatal

a52d153

use normalized probability floor

14bab7f

update result before even attempting to log bandit

5384bd2

aarsilv commented Aug 8, 2024

View reviewed changes

aarsilv requested a review from typotter August 8, 2024 06:18

typotter approved these changes Aug 8, 2024

View reviewed changes

aarsilv self-assigned this Aug 8, 2024

spotless 🙄

efed6f4

aarsilv merged commit 55c3fce into aaron/ff-2710/ufc Aug 8, 2024
3 checks passed

aarsilv deleted the aaron/ff-2710/ufc-bandits branch August 8, 2024 18:22

aarsilv mentioned this pull request Aug 15, 2024

Bring UFC-based EppoClient into Common #29

Merged

		// The coefficient implementation knows how to score
		double attributeScore = attributeCoefficients.scoreForAttributeValue(contextValue);

		Set<String> neededModelVersions = configurationStore.banditModelVersions();
		boolean needBanditParameters = !loadedBanditModelVersions.containsAll(neededModelVersions);

		for (BanditFlagVariation banditFlagVariation : banditReference.getFlagVariations()) {
		if (banditFlagVariation.getFlagKey().equals(flagKey)

		// Note: In practice this double loop should be quite quick as the number of bandits and bandit
		// variations will be small. Should this ever change, we can optimize things.

Add bandit support #25

Add bandit support #25

Conversation

aarsilv commented Aug 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

giorgiomartini0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aarsilv Aug 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

typotter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

typotter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aarsilv commented Aug 6, 2024 •

edited

Loading

aarsilv Aug 6, 2024 •

edited

Loading